Process a truncation of big feeds rather than ignore them

Mar 26, 2007 16:11


Title
Process a truncation of big feeds rather than ignore them

Short, concise description of the idea
RSS feeds that are too big should be processed in truncated fashion rather than ignored.

Full description of the idea
For instance, here is a feed that is ignored because of its size:
http://syndicated.livejournal.com/gonegaming/profile

This is a Blogspot feed, so it's a widely used platform. I'm sure this problem occurs for many syndicated feeds, perhaps reading in the most recent items only in the first 64KB of the feed.

If this suggestion doesn't make sense, LJ could compensate by reading the whole thing but only processing it once a day, for instance, if the issue is bandwidth. I'm sure there are technical alternatives that could be considered.
An ordered list of benefits
  • Would be able to subscribe to feeds that currently are never processed

An ordered list of problems/issues involved
  • Cost of developing the feature

An organized list, or a few short paragraphs detailing suggestions for implementation
  • Take the first 64KB of a feed

    Move backwards to the last closed tag and truncate the text past that

    The feed will no longer by XML conformant but will contain items that can be processed

    Process the items as normal

syndication, § implemented differently, data limitations

Previous post Next post
Up