Is this why Twitter-IM doesnt scale?

Published: 11:06 PM GMT+12, Thursday, 12 June 2008 under: technology jabber
twitter  jabber  scale  fail 

It's been said that twitters IM gateway has been one of the causes for their scale issues, and I just logged into one my older gmail accounts I only use for IM (and twitter-im in particular):

What I found greeting me was 3-4 messages with "off line IM messages" from twitter - 795 lines in one, 1675 in another, and a few other.

Could it be that a simple scaling technique twitter could use is to follow a users jabber presence and NOT send them thousands of off line messages? I had actually thought they already did, but I guess not. Surely only sending IM messages when the user is online would lower their resource consumption tremendously.

Comments (8)

Well I am not sure, I never got any offliners from Twitter yet. Also I am sure, they will be getting loads of presence per second coz a presence is sent for every busy, online, unavailable, available action. One of the possible solution can be caching as discussed here http://tinyurl.com/829d53

left by Abhinav Singh . Friday, 23 January 2009 3:41 AM

great post

left by url submission . Monday, 23 June 2008 5:12 AM

mibus - my assumption wouldn't be to check the database for my last presence and flood me with messages, it would be to simply DROP any message received if they don't have presence. Obviously I'm not around, so I don't care about any missed messages. Direct Messages still get emailed to me so....

left by Mark Derricutt . Sunday, 15 June 2008 12:41 AM

Wouldn't it be *more* work to then wait for your 'available' presence, check the database for all posts since your last presence update (which could be a lot), then flood you with messages?

Or, they "deliver" them to your server, and consider their job done. Seems the easier choice (and lower-workload) to me! :)

left by mibus . Sunday, 15 June 2008 12:36 AM

Weird, I also never got anything offline from twitter. Actually, it even stops sending me stuff whenever I go offline!

I need to explicitly tell it to send me stuff again or it wont.

left by John Bäckstrand . Friday, 13 June 2008 2:27 AM

weird, i never had that problem here. i definitely never got offline messages from twitter.

left by Dennis . Friday, 13 June 2008 1:12 AM

Yannooo, they already receive your presence now. At least when I added twitter as a budy, I received the subscription request from twitter. Looking at my roster, twitter@twitter.com is flaged as both.

So no change to their current load.

Sure it is not free in terms of scalability, but given the ephemeral nature of presence, you could only store it in memcached. I think its very much worth the effort.

Best regards,

Best regards,

left by Pedro Melo . Friday, 13 June 2008 1:04 AM

If the twitter bot keeps track of users who are online, it also implies receiving a huge amount of incoming presence that will need to be processed. In addition you need some memory to store who is online/offline. Not sure what's the best approach here but keeping track of who is online is definitively not free in terms of scalability.

left by Yannooo . Friday, 13 June 2008 12:35 AM
Add Comment