ftproxy, the FeedTree HTTP proxy

Quick links: Downloads, Screenshots, ProxySetup, Troubleshooting, mailing list

Overview

Rather than develop a fully-featured feed reading application as a front-end for FeedTree, we chose to build an HTTP proxy which can be used with any desktop RSS reader.

Here's how it works: the proxy application joins the FeedTree network and waits for an RSS reader to request a conventional feed via HTTP. When this occurs, the proxy instead substitutes the most up-to-date FeedTree version of the feed, including all updates received from other peers. This way, every RSS client can take advantage of FeedTree's bandwidth savings and instant updates.

Download

See the download page.

Running the proxy

In order for the proxy to connect to the Pastry peer-to-peer network which supports FeedTree, your host will need one of the following:

• a fully routable Internet address (that is, it's not behind a NAT or firewall), or
• a port-forward set up on your NAT/FW for port 29690, TCP and UDP

If you go the port-forward route, the software will automatically detect this and sniff out your external IP.

(These requirements are essentially equivalent to the demands of any other current peer-to-peer system; if you've ever set up your computer for a p2p application, the FeedTree setup is analogous. See also the FAQ.)

To run the proxy for the first time:

• Download and extract the distribution archive for your platform.
• Configure your news reading software to use the FeedTree proxy; see ProxySetup.
• Start the proxy application.
  • On Windows, open the FeedTreeProxy.exe program.
  • On Mac OS X, open the FeedTree Web proxy.app program.
  • Linux users can open the ftproxygui.jar file from a graphical desktop environment.

You should see the proxy's graphical control window, including feedback as it connects to the FeedTree network:

The proxy will first contact bootstrap.feedtree.net to join the global FeedTree network. Once this connection has been established and verified, the FeedTree HTTP proxy will start on port 8501. Some short status messages will be printed to the terminal; for more detail, use the --verbose flag (repeat it for still more detail).

The proxy application's GUI tells you a little bit about what's going on, but for all the gory details you'll want to use the web monitoring interface (see screenshot) running at the URL http://localhost:8500/. This tool lists the feeds that the proxy is currently tracking, including a graph of recent events on that feed (your own polling, messages from other users in the network, etc).

Problems

Can't get it to work? Check out the Troubleshooting & FAQ page, or send an email to the feedtree-users list or to the developers.

Command-line usage

The ftproxy.jar file (included in the "jar+publisher" download) contains a version of the proxy that operates without a GUI. This version also supports a number of command-line flags that allow detailed tweaking of the proxy's operation; java -jar ftproxy.jar --help will tell you about them. Of particular note are:

• --verbose (add more logging output so you can see what's going on)
• --public and --allow (set up your proxy to permit some or all HTTP connections from hosts other than localhost)

Operational details

The proxy observes all HTTP requests and attempts to intercept the retrieval of RSS feeds. Anything that looks like a feed is returned to the user, while the proxy subscribes to the FeedTree multicast group for that URL; subsequent proxied requests for that URL will return cached feed data from the proxy. This cached data is updated any time a FeedTree event (either a new entry in the feed, or a "nothing new" heartbeat message) is received by the proxy. Clients are therefore free to poll feeds as frequently as they want (for example, every 5 minutes) because these requests are intercepted by the proxy. The more aggressively the user's feed reader software polls through the proxy, the sooner the user will see news that has arrived via FeedTree multicast.

If a user subscribes (through FeedTree) to a feed that isn't being pushed by the original publisher, that user will go ahead and poll (as with conventional RSS) and allow everyone in the FeedTree ring to take advantage of that polled feed data. So even if you're the only FT user in the universe, you will be able to read your news as normal; as soon as a second user joins FeedTree, publishers will start to see bandwidth savings. The current system is optimized for substantially more than two subscribers, however; up to three users will cooperate to poll a single feed on a random schedule, resulting in a threefold improvement in the timeliness of news.

The current FeedTree network consists of a cluster of machines at Rice. The FeedTree software connects to this network by default, using the round-robin DNS name bootstrap.feedtree.net; however, a separate ring can also be created if desired (see the flags below).

Help text

usage: java -jar ftproxy.jar [flags]

flags:

  -g/--public
       allow HTTP connections from any host
       (default: disabled; allow only from localhost)

  -A/--allow <host-pattern>
       allow HTTP connections from some hosts 
       if --allow used, all hosts not explicitly allowed
       are denied; can be used multiply for logical OR
       -- recognized patterns --
          N.N.N.N  -- single IP address, dotted decimal
          N.N.0.0  -- IP address block, zeroes are wildcards
          host.name -- single host name
          *.host.name -- many hosts, * is wildcard

  -b/--bootstrap <host:port>
       host to contact to join FeedTree network
       (default: bootstrap.feedtree.net)

  -N/--new-ring
       use no bootstrap; start a new FeedTree network

  -p/--port <port>
       Pastry connectivity port for FeedTree node
       (forward this port through NATs)
       (default: 29690)

  -w/--web-port <port>
       port on localhost for web-based status page
       (default: 8500)

  -P/--proxy-port <port>
       port on localhost for HTTP proxy
       (default: 8501)

  -O/--overload <N>
       number of FeedTree clients that should cooperate
       to poll a given resource
       (default: 3)

  -t/--ttl <N>
       time-to-live (delay between feed polls, in seconds)
       (default: 1800000)

  -R/--reporting <level>
       specify what statistics to report to the project maintainers
       levels: full, anonymous (= full without IP), none
       default: full

  -v/--verbose
       increase the verbosity of output (may be used multiply)

  -V/--verbose-max
       maximum debugging output (equal to -v -v -v -v)

  -h/--help
       display this text