A few months ago we announced our new chatter functionality. The concept is simple. Conference attendees add their twitter accounts to EventVue. We then find their tweets, aggregate them within EventVue and republish them on twitter. This provides a powerful way for people to follow the conversation that is happening at an event. We’ve gotten great feedback on our chatter functionality and our auto-updating screens have been quite popular as well.
While the idea of aggregating tweets is simple, making it work isn’t quite as easy. Here is a behind-the-scenes look at EventVue and the technology that makes it happen. It’s the story of how a company called Gnip came to our rescue at just the right time.
Our first prototype
For our first chatter prototype we used a simple script that fetched the tweets from each attendee one at a time. The script ran once a minute and gave us a great proof of concept. We only had to demo it working with a couple dozen people. It was enough to get people excited about the feature and prove that we were onto something. It quickly became obvious that the method we were using for our prototype wouldn’t scale. We were querying twitter non-stop since we needed the conversations to be displayed in real time. We quickly grew from dozens of queries a minute to hundreds of queries a minute. With each request taking several seconds to process, it became harder and harder for us to present results in real time.
Our second attempt
Then we remembered that Twitter provides a feed of all your friends tweets. Ah-hah! All we needed to do was follow everyone who added their screenname in EventVue and we could get their tweets with a single request to twitter. Brilliant, right? Not quite. It turned out that the feed from twitter was missing about 50% of the tweets and it progressively got worse as we started following more and more people. People started complaining and we we were forced back to the drawing board once again.
Getting desperate
We went back to our first prototype except this time we used Summize (now a part of twitter). Their API is about 7 times faster, doesn’t have any published rate limiting, and most importantly allows complex queries up to 160 characters long. This meant that we could fetch tweets from a dozen people at a time. We turned on multi-threading, sacrificed a CPU and started pounding Summize as hard as we could! This enabled us to process tweets from several thousand people and display the results within just a few seconds. Not bad. It still wasn’t ideal, but we had exhausted every other option. Everything was going smoothly until I got this (somewhat inevitable) email from Greg Pass at Summize/Twitter:
“We notice you’re using the Twitter Search API — that’s great! Unfortunately, you are so exceeding our rate limits, that all of your requests are being refused. You’re currently at ~34 requests per second. If you slow it down to 1 request every few seconds, you’ll be back in action.”
Bummer!
That’s around the time when I heard that this new service called Gnip had added twitter support. Gnip promised to “make data portability suck less”. More specifically, they promised to take away my never ending battle with twitter. Needless to say, they had my attention. I started playing with their API and immediately feel in love. Instead of having to continually ask if there were any new tweets, Gnip offered to watch for me and to let me know when it came across any tweets that I cared about. It didn’t take much of a decision for us to switch to Gnip. A couple dozen lines of code later and Chatter was back — better and faster than ever! Gnip gave me the push architecture that I desperately needed, freed up our computing resources and ultimately saved us a huge amount of time and money.
For anyone who finds themselves in a similar predicament, I can’t recommend Gnip enough. Gnip has a rock solid service and a team that has been incredible in helping us make the transition.
We’re highly indebted to the guys at Gnip. This post is really just to say thanks.
