Changeset 156

Timestamp:

02/12/06 19:10:31 (3 years ago)

Author:

dsandler

Message:

Import of Calcium, a script to automatically scrape feeds and send their links to the CoralCDN. By reading feeds through a FeedTreeProxy, Calcium has excellent update resolution (read: new links faster).

Calcium can be pointed at feeds of links that have a high probability of becoming oversubscribed to ensure that Coral has a copy of the page before the server is crushed.

Examples of such feeds include:

• Digg's "diggall" feed, http://digg.com/rss/indexdig.xml
  
• Reddit's "new" feed, http://reddit.com/new.rss
  
• Delicious's "popular" feed, http://del.icio.us/rss/popular/
  
  • (Note that while technically a link must already be popular, and thus potentially oversubscribed, to get on this particular feed, the del.icio.us userbase tends to be somewhat "ahead" of the rest of the Web, so this is still a good place to see links "first".)
    

Files:

• calcium (added)
• calcium/branches (added)
• calcium/tags (added)
• calcium/trunk (added)
• calcium/trunk/calcium.py (added)
• calcium/trunk/feeds.shelf (added)
• calcium/trunk/lib (added)
• calcium/trunk/lib/BeautifulSoup.py (added)
• calcium/trunk/lib/BeautifulSoup.pyc (added)
• calcium/trunk/lib/feedparser.py (added)
• calcium/trunk/lib/feedparser.pyc (added)