The incredibly painful way of getting an RSS feed of a Twitter list

June 5th, 2011

(why does Twitter make it so hard? What did RSS ever do to it?)

The skinny

The Twitter API appears to support two methods for getting RSS feeds of lists. Both of these methods are pretty buried and their API docs don’t help too much, but nonetheless both of these feeds work:

http://api.twitter.com/1/nathansmith/lists/javascript/statuses.atom

If you have the list ID (requires using some dev tools – see below), you can also use this format:

http://api.twitter.com/1/lists/statuses.atom?list_id=41981352

The explanation

As it turns out, getting an RSS feed of a Twitter list is no easy task, at least from what I’ve found. Twitter has largely replaced RSS for me, in terms of getting the latest and greatest information, and maybe Twitter wants to keep it that way by having me rely on their service instead of an RSS feed. That’s how it seems to me anyway (this is also evidenced by their killing off of making RSS feeds easily discoverable but still making the feed available as long as you can figure out your Twitter user id (which is a task in itself), for instance here’s my RSS feed: https://twitter.com/statuses/user_timeline/16521889.rss). I’d say this is simply a conspiracy against RSS feeds and developers in general, which is the direction Twitter is headed these days.

In any case, I digress! I want to get an RSS feed of a Twitter list. Great! All I should have to do is go to the page of the list and subscribe to the RSS feed. But.. there’s no RSS feed. Hmm.

Google is your friend… most of the time

A search for “twitter list rss” comes up with links to implementing an RSS feed via Yahoo Pipes as well as pages referring to an appspot service that now 404s (twiterlist2rss). In any case, they are both not what I’m looking for. Which means I have to get creative. Which means… my options at this point are screen scraping (made harder by the fact that Twitter content is now served with JavaScript) or.. gasp.. using their API!

Twitter REST API to the rescue

After digging through the API docs, I found just the page I was looking for, detailing the REST service for Twitter lists.

As it turns out, the basic structure for requesting list data is as follows:

http://api.twitter.com/version/lists/statuses.format

As a newbie Twitter developer, there were a couple of things wrong here. First, what’s the version of the API I’m supposed to use? From other examples on the web (outside of the documentation…) I just assumed I’d be using version 1, and that seemed to work. Ok, whatever, we’ll go with 1:

http://api.twitter.com/1/lists/statuses.format

Second step: this API is actually quite nice, and will return data in three different formats: xml, json, or atom (RSS!). Ok, now we’re making progress!

http://api.twitter.com/1/lists/statuses.atom

Where’s this list_id you speak of?

Now to pass in the info – I need to pass in the list ID in a key called “list_id”. The only problem is… where do I find the list ID? For the purposes of this example I’ll use Nathan Smith’s excellent JavaScript developers list. To normal human beings, this list might be called “nathansmith/javascript”, but a Twitter ID is numerical, so that’s not what we’re looking for.

So.. I assume at this point I can use some of the Twitter developer tools and fiddle around with them until I find an API call that will translate the human-readable list name into a numerical ID. But no.. I don’t want to download anything, I don’t want to investigate more of the API or go down any more rabbit holes.

It’s time to cut this rabbit hole short and do a bit of investigating. So I pulled up the list, opened up Web Inspector in Chrome, and took a look at the Network tab in the hopes of seeing an API call with an ID I was looking for…

…And I found it! Somewhere down the line, Twitter was calling statuses.json with this request:

https://api.twitter.com/1/lists/statuses.json?list_id=41981352&include_entities=1&include_available_features=1&contributor_details=true

There’s our list_id (and incidentally, the same API call [with json instead of atom].. which shows that Web Inspector is more useful than API docs sometimes)! After a short copy-paste, all the pieces of my Frankenstein API call have been put together:

http://api.twitter.com/1/lists/statuses.atom?list_id=41981352

And it works, without having to mess around with OAuth or authentication, registering an app, or whatever. Of course now we can refer back to the API docs and mess around with some parameters to get just the data I’m looking for. Maybe most useful to developers is the handy ability to get easily paginated results:

http://api.twitter.com/1/lists/statuses.atom?list_id=41981352&page=2

There’s also what would be a handy “count” variable, but it’s altogether useless as it appears to be ignored. The API always returned 20 results for me.

In any case, there you have it! The incredibly roundabout way to get an RSS feed of your favorite Twitter list.

UPDATE

Thanks to this site, it looks like there’s a more straightforward way of getting the RSS feed. Using the same list we were trying to get above, it looks like this also works:

http://api.twitter.com/1/nathansmith/lists/javascript/statuses.atom

Alex says:

June 8, 2011 at 7:33 am

David – thank you for putting this out there; I’ve been looking for a similar solution for quite some time. It’s a shame that Twitter makes this so difficult!

Dave - Lifekludger says:

June 12, 2011 at 9:26 pm

Thanks so much for the article and responding to my tweet alerting me to it.

I dunno why they insist on making this so hard, it never used to bet this hard with the earlier twitter site.

Thanks again.

Where can I find an RSS feed for a Twitter LIST? - Quora says:

June 19, 2011 at 12:15 am

[...] also wrote up an article here with more details: http://davidbcalhoun.com/2011/th…This answer .Please specify the necessary improvements. Edit Link Text Show answer summary [...]

Pete Carroll says:

July 1, 2011 at 7:41 am

Thanks for this – good to know I’m not the only one with this problem.

Googling around found first this

https://groups.google.com/forum/#!topic/twitter-development-talk/Sm4iUJHkYA4

which includes info on lists from themattharris who seems to be a developer at Twitter which lead to

http://dev.twitter.com/doc/get/lists

from which it seems you can get list_id from queries formatted as for example:

http://api.twitter.com/1/lists.xml?screen_name=scibella
http://api.twitter.com/1/lists.xml?user_id=51496296

tried these and the method seems to work

hope this is useful

Tad Staley says:

July 11, 2011 at 8:17 pm

Thanks for taking the time to write this up. Your explanation was easy to follow, and the configuration appeared to work well.

However, I am trying to get tweets from a twitter list into Drupal’s feed aggregator. When I create the new feed using your technique (the string I used was http://api.twitter.com/1/tstaley/lists/enterprise-2-0/statuses.atom), the tweets from the list all came in fine, but the underlying URL for each tweet was the same – hardcoded to be the URL for the first tweet in the list. In this case, each entry had a URL of http://twitter.com/dhinchcliffe/statuses/90450509646938112.

Any ideas on how Drupal is mishandling the incoming URLs?

Thanks again!

Evgeny Goldin says:

August 6, 2011 at 5:28 am

Thanks a lot, David! I was really missing this.

http://api.twitter.com/1//lists//statuses.atom works beautifully

Sandy Martin says:

August 9, 2011 at 10:35 am

You made my day!

August 9, 2011 at 10:43 am

Any idea how to make a feed out of a hashtag?

Using the Twitter API not Twitter RSS : Interwebs & CSS says:

August 19, 2011 at 7:36 am

[...] RSS feeds alltogether. So in an attempt to find the solution I was looking for I found a blog post written by David Calhoun that explains some of his findings. It was very helpful and I recommend it [...]

David says:

August 21, 2011 at 9:18 am

@Sandy Glad you found it useful! Unfortunately it looks like the search API only returns JSON however. See this: https://dev.twitter.com/docs/api/1/get/search

Ashley Feucht says:

September 5, 2011 at 11:54 am

This is fantastic! I’ve been on the prowl for a way to convert Twitter lists to RSS for a LONG time. Finally, I can stream specific Twitter lists to Google Reader, which allows me to search keywords within a list. Thanks for sharing!

Randy says:

September 29, 2011 at 8:53 am

Incidentally, you can also find a list ID by using Firebug to inspect the drop-down list under the ‘lists’ tab on a user’s page. To do this, open the page in Firefox, then open Firebug (or you can use Chrome dev tools, etc). Click the ‘lists’ tab so that the drop-down list shows… then use the ‘inspect’ tool to click on the element and expand it from there. You have to have the drop-down showing first, as it is loaded via AJAX and will not initially show up in the DOM when viewing it in Firebug.

The ID will be held in an LI with class of “stream-link” and a custom attribute of “data-item-id”, like data-item-id=”xxxxxxx”.

Hope that helps some of you. Cheers!

Andy says:

October 10, 2011 at 9:32 am

Brilliant thanks for this… this is a constant headache for me and I actually resorted to using the Twitter embed code and then fudging the CSS to make it look how I wanted… this is great as the Twitter API makes absolutely no sense to me!

Thanks.. This will be pride of place in my book marks

Easy methods of grabbing a RSS feed of a Twitter list or Twitter user | Madison.com Labs Blog says:

October 18, 2011 at 4:29 am

[...] out, it’s not at all difficult thanks to this post via David B. Calhoun’s blog. To generate an RSS feed of a Twitter List you have to make two alterations to a standardised URL; [...]

nuphero says:

December 4, 2011 at 6:56 pm

Thanks you for useful information. I have a small question, is there any way to have RSS feed for List combine with any other operator such as hashtag or keyword. That’ll be perfect.

Create a Twitter RSS feed for Tweets, Searches, Hashtags, Lists or Favorites says:

December 6, 2011 at 8:02 pm

[...] http://search.twitter.com/search.rss?q=search%23love Here is one way of getting rss from a Twitter list: http://davidbcalhoun.com/2011/the-incredibly-painful-way-of-getting-an-rss-feed-of-a-twitter-list [...]

articleideas by profdebock - Pearltrees says:

December 14, 2011 at 4:04 pm

[...] Where’s this list_id you speak of? Now to pass in the info – I need to pass in the list ID in a key called “list_id”. The only problem is… where do I find the list ID? The incredibly painful way of getting an RSS feed of a Twitter list | David Calhoun's Developer Blog [...]

Arturo Tena says:

January 18, 2012 at 11:29 am

What about getting a PRIVATE list?

Bruno Pierre Gebarski says:

March 19, 2012 at 2:34 am

Greetings from this side of the pond! (Hamburg Germany)
RSS feed for Twitterlists:
http://api.twitter.com/1/AUTHOR/lists/TWITTER-LIST/statuses.atom
(If the list has compound words, do separate them with hyphen: “Twitter-list-word

RSS feed for single Twitter account:
http://twitter.com/statuses/user_timeline/ID-Number.rss
you can find any user’s ID name by using the following site: http://idfromuser.com/
If you want more tips on RSS feeds from Twitter, please visit my blog at: http://www.brunogebarski.com and do not forget to leave a comment! Thank you

Do you know how to import those RSS feeds into an 2007 Excel sheet and keep it updated? Would love to find out how to do it: thank you
@bunogebarski

David Hobbie says:

May 24, 2012 at 12:34 pm

Thanks for this–still works despite whatever changes to Twitter API have happened. One comment–in your first list, you should make clear that “javascript” is the name of your list. (Bruno’s comment finally made me understand that).