The Gillmor Gang - Robert Scoble, Mike Arrington, Dana Gardner, and Robert W. Anderson - ask former Twitter architect Blaine Cook about Twitter internals including the valuable Track feature. Google engineer Bob Lee joins again. Recorded Thursday, May 15, 2008.

Scoble: Hey, Twitter is down.

Steve Gillmor: Again huh?

Scoble: Again. And FriendFeed is up and they are all talking about Twitter being down and they are starting pro-test.

Steve: How is FriendFeed up if Twitter is down?

Scoble: FriendFeed stays up because they designed it with a good architecture.

Steve: Now, that is not what I am asking. Can you see Twitter tweaks on FriendFeed?

Scoble: No, there is no tweaks on FriendFeed right now because Twitter is down.

Steve: Oh I see. What is up though is this backwater conversation that you can’t see anywhere else expect on FriendFeed, right?

Scoble: Exactly.

Steve: OK, oh it is not very sad.

Scoble: And they are all talking about how Twitter is down [laughs].

Steve: Who have we got here?

Bob Lee: Bob Lee.

Steve: Bob, welcome.

Bob: Hey, can you hear me OK this time?

Steve: Yeah, you sound great.

Bob: OK good.

Steve: And we have got Robert Scoble and we have got Dana Gardner now. Who I am looking for is Blaine Cook, who should call in, so why don’t you guys talk about something and let me see who this is.

Robert Scoble: We can talk about Twitter being down while you try to get Blaine on the phone.

Mike Arrington: Twitter!

Steve: Who is that?

Mike: Twitter is down.

Scoble: Who is that?

Mike: It is Mike Arrington.

Steve: Hey Mike. So, Twitter is down, I don’t know whether we are going to get Mr. Cook here. He may have been rehired to go and try and fix it.

Mike: I mean, he hasn’t actually come on the call yet?

Steve: Not so far, let me just see if I can reach him.

Mike: Twitter is down and Facebook is blocking Google Friend.

Steve: Yeah, what is the story there Mike, I know you have been [cross talk].

Mike: Actually, I have been talking to CNET shareholders today. I have been talking to Google and Facebook. Actually, Google is going to call me in a second with their side of the story and I am going to have to jump off and put on hold for a second. So, Facebook says Google is violating their API through their new Friend Connect product.

And there are a couple of reasons, but the main one is sort of apparently there is no dashboard on Google that once you give somebody permission to sort of pull the data back and remove the permission, and so they have banned them. I am going to have a talk with Google about some specifics on their product and get a fuller picture in a couple of minutes and do a post, but…

Scoble: That is the Friend Connect meeting on Monday night. I was talking to Dave Moran right afterward. And he said the one thing that caught his attention was where they showed Facebook’s membership being sucked into Google’s Friend Connect. That was the beginning of this process.

And I know from my previous conversations with Dave Moran that really concern that when you take an email out of Facebook onto a site that doesn’t have a contract with Facebook, but when you decide to delete that email or change that email address, it now is sitting on somebody else’s servers that they can’t delete it off or change it [crosstalk] have them.

Steve: They already are violating that term of service by sending that email in regular SMTP email. I mean, that content is already outside their clause.

Scoble: Well, that is what they told me, I am just passing on.

Steve: Well, I am making a point which is, what are they are talking about.

Mike: What is Facebook talking about you mean?

Steve: It makes no sense.

Mike: Well, I think, it is hard to revoke permission. It appears to be their position on this. So, once you give data, it is hard to get it back and they have their own privacy dashboard and they are going to add in third party app stuff, you can pull it up, just like you can revoke applications for having access to stuff as well. You know, we will see.

To me, it almost is irrelevant because if the third party app wants to harvest data, they are going to ignore terms of service anyway and they are going to just harvest data. And maybe they will even do in the background and never tell anyone and just do it for ever before they actually start to exploit it and I am not really speculating what topic.

[crosstalk]

Steve: I think, what they are doing, what they said that they were doing when they showed us on Monday is that they are using Facebook’s interface to talk to Facebook.

Mike: Hey, I am going to go on hold because this is them calling and I might come back.

Steve: All right, well come back, I think, Blaine is here now, right Blaine?

Blaine Cook: Yeah, I am here, yeah.

Steve: OK. So, I will let Mike Arrington come and try and beat you up in a minute, but first.

Blaine: [laughs].

Steve: I know that is what we are all waiting for, but we have some serious business at hand here. So, can you just tell us a little bit about the history of your involvement with the whole Twitter concept, and I’d like to get you to describe how you came up with the Track function?

Blaine: Sure. I was at Odeo previous six years, and went over to Obvious and have been serving around for a long time there. And as Odeo got spun off and sold, Twitter became my main focus. And the first thing I did there was, I have never been a big fan of SMS and so I built the IM interface. I sit in my computer all day, I want to get those updates, but I don’t want to have them flood my phone. So, that was sort of the first main thing that I worked on there.

So, as far as the Track stuff, we were having a discussion one day about search, and Jack was saying, ‘oh we want search’ and I was like, ‘Ahh,’ we have got all these issues to deal with, for search, we are going to have to put out a cluster and all that bit of thing. He is like, but it would be really cool if people knew what was going on and if we could show people what other people were saying.

And I am like, oh you want real time search, no problem. And we actually had a little bet and I built it in 12 hours, and it is the simplest thing that could possibly work, which is how you should always approach these things. So, it is not a complicated thing. There is plenty of companies that do much more complicated things like John Colicke, who is now a Twitter built SQL stream. I think, it is the company name; I might be bunking on it right now.

And then, there is Correlate, I think, there is some discussion on one of his blogs a while back about Correlate and what not. And so, those are like stream-based processing like real-time analytical engines that are sort of like the mother load as far as Track thing. Track is an hand-only query interface. So, you can only do… I could track for example Blaine and Cook, but I can’t track Blaine and not David for example. So, I know everything that David Blaine is doing all the time and it is like, ah I don’t actually care. So, I mean…

Steve: So, how complicated would it be to add some sort of filtering capability, which would essentially say, I want to hear about this keyword but if somebody spams me basically with too much information about something that I’m not interested in around that on a personal level, not on a global level, but on a personal level, could I basically block that?

Blaine: So, the advantage, I mean, the reason that I have been advocating messaging systems and I mean I am sure we will get into a little bit, so that the whole Jabber stuff and Federation and all that kind of stuff applies more genuinely than just Twitter. And part of the reason for that is if you have an event-driven system, you don’t have to do the queries against large datasets. You are just looking at an individual piece of data in memory.

You have all the context around that object and so you can do certain things like to block someone on a real-time feed is really really easy. To say, here is all of the updates that I have received, and then, oh yeah filter these 4000 people from those updates is a little bit more complicated, because now you have to analyze this data, but all of those things in their history view, right, so that if you can…

Steve: Right, but I mean, I didn’t say that that would be a bad thing. I mean that is basically a client approach to a filter and I think, that is obvious that something can be done.

Blaine: Sure, but you can build systems that way like internally, clearly this is all event driven and doing filtering on a message-by-message basis is much easier than trying to do the filter as sort of a post facto, just put this custom view of stuff, like for example, Google, only in the past couple of years that they have actually introduced custom search results.

And the flexibility that you have is still fairly limited like they don’t let you do massive amounts of filtering across their sets of data. You can do a little bit, but I mean they are Google right, like they can do that kind of thing, but it took them a long time to get there. So, it is that sort of rich filtering ability that you can do if you are doing a delivery-based system.

I mean, that is why Track is only available over SMS in Jabber because to build it to the real time delivery to systems is frankly pretty trivial. But, to do it so that you can present like a historical view for all of the uses on Twitter for every move that they want to track, whether or not they are looking at it is the bigger question, right?

Steve: Yeah. And my question here is what if we don’t want that. I mean, that sounds interesting, but what we are asking here about is something that, if I understand you correctly, is something that could be done downstream albeit what the economic model of being able to provide that service is not understood or something that you came to build in the ocean in order to be able to do it with the current toolset.

Blaine: Sure. I mean like the current toolset, all of the stuff that I am interested in is a couple of years out or maybe six months and I don’t know on the show few days ago next week or something, but it is not going to be right now. I am just saying like if you want that sort of that really rich functionality that frankly hints at you not looking at the same kind of…

The interesting thing about Track is that it tells you what is going on right now and you can apply whatever filtering you want. You can handle thousands and thousands of messages a second and build whatever kind of service you want on top of that. I think, that is well-understood and that is not necessarily down to Twitter. I mean, I think that they have obviously a unique opportunity to do that better than anyone else, but I mean, as far as making that a reason to not do the centralized Twitter, not do these sorts of things, it doesn’t quite add up. I am not sure what you guys were talking about before I was on the call so.

Steve: No, no, that is fine. I mean, we had a couple of minutes talking about the whole Plaxo and Friend Connect and what Facebook is doing to try and stop it. So, there wasn’t much that was talked about, but the reason that I had Bob Lee come on the show again is to basically function in the same way that he did on the previous show.

And I just want to make one comment here so that we don’t get it off the rails about issues that I think, are not specifically germane here. The question of decentralizing Twitter…

Mike: I’m back.

Steve: Who is that?

Mike: It is Mike.

Steve: OK great.

Mike: Did Blaine join?

Steve: Yes, we are talking with him.

Blaine: Hi Mike.

Steve: Just trying to ask him a question here. So, the…

Mike: Hey Blaine.

Blaine: Hey Mike.

Steve: Mike, I am trying to ask him a question here.

Mike: Steve, I get to go next?

Steve: Yes. The question of decentralizing Twitter is a political question in my view, not a technology one.

Blaine: Sure.

Steve: You made several comments on the Gillmor Gang side and also on Cliff Gerrish’s side that we are really interesting and suggested that you felt that there were some scalability issues about doing what we were talking about. And I wanted you to sort of address that, so that I can understand why you think that is the case without getting into some sort of political discussion about whether or not decentralization is a useful strategy or not, because I am not sure that you and I or anybody is in much disagreement about that.

Blaine: Right, for sure.

Mike: What is our agreement that we think decentralization is good?

Steve: I don’t know. I don’t care personally about whether it is good or bad, I just want to know if it is going to happen or not. And my opinion is it is not going to happen because the political strengths that Twitter enjoys don’t mandate any reason to do it, but others might disagree with that, but I still want to know about the scalability issue. You seemed to imply that there was something about accessing the Twitter core as I would call it that it would require a big player like a Google to be able to lift that number of bits?

Blaine: Oh, I am sorry, let’s see here if I can find that comment. I didn’t mean to imply that it would require a Google, I am just saying that Google or those sorts of organizations, if they were a decentralized network of posts flying around to leverage that and to provide a real time search across the Internet, someone like Google is going to be better positioned to really take advantage of that.

I think, obviously Twitter has a gigantic opportunity there because I don’t think that Google has paid much attention to sort of real time messaging and what not. I think that it has been thrown out there that no one else but Twitter could do this, well there are some “I’s” right and there is other people that are consuming the real-time feed and really they have the ability to see that.

Obviously, they don’t have the ability to do private Twitters yet, but at least as far as public information is that that ability is I think, it is proven that other people can do it. And so, I think, the Internet is much more interesting and people compete on features and building interesting things rather than competing on walled garden user bases, right.

There is an interesting demo that I wanted to point out that some friends at Stamen built called Firehose.Stamen.com. You can go there now and it is a flash visualization of basically all the Twitters in real time and you can just go watch it. It is pretty cool.

Mike: It is not working right now.

Blaine: I think, it is fairly obvious that if you have that data, well you can add a lot of value to that.

Mike: It is Firehose.Stamen.com?

Blaine: Yeah, that is correct. And I don’t think that this anything that Twitter is opposed to. I mean, they can obviously disagree with me.

Mike: How do you pronounce Stamen?

Steve: Stamen.

Blaine: Stamen, yeah.

Mike: It is not showing as even a legitimate URL, it is just firehose.stamen.com?

Blaine: Yeah, that is correct.

Mike: OK.

Blaine: Yeah, works for me.

Steve: OK, so Mike go ahead.

Mike: Well, did he answer your question. I mean, he should place that out.

Steve: Yeah, I got exactly what I wanted, so it is your turn.

Mike: OK. So, why is Twitter down today?

Blaine: So, that is a good question. I have no idea.

Bob: This is Bob. If it has anything to do with what happened yesterday, they actually posted about to the API, looks apparently like all their caches…

[crosstalk]

Mike: What happened yesterday; did somebody else sign up for the service?

Bob: All of their caching got invalidated at once for some reason. They didn’t give any details, but basically they had to reload all the data I assume from the databases into the caches and that takes probably quite a while.

Mike: OK, so here is my question.

Blaine: You know very well that I wouldn’t be able to tell you why. I mean I am not at liberty to say those things.

Mike: So I have two questions, one which you won’t answer but I will ask anyway.

Blaine: OK.

Mike: Why is it that the day before you guys bailed off of Joyent that you guys wrote the blog post called “Happy Happy…” I think, it was “Happy happy Joyent.” Is it true that they threatened to basically take you guys down if you didn’t write that blog post?

Blaine: You were right in guessing that, I don’t think I can answer that.

Man1: Ah-huh, so you are confirming that if it is like that?

Mike: So, here is my real question, so if you were starting Twitter, if Twitter didn’t exist and you were starting it today, what would you do differently architecturally to prepare for the growth that happened, that you know will happen?

Blaine: Right, right. So, there are two ways to answer that. If I was starting Twitter again today with the same knowledge, Twitter was a stupid idea two years ago. I mean, we joked about calling it “Friend Stalker.” It was one of these things where like really do you think anyone is going to like try and meet up for coffee based on SMSs. I mean, it was a fun idea, but it was like, there were a lot of like SMS - no one used SMS. No one had unlimited SMS plans.

Mike: I have always guessed that by the way that you guys didn’t even really have any clue as to how cool what you built really was when you launched.

Blaine: I mean, Jack has been obsessed with this idea for years. I think, he fully understands how cool it was, but as a product it is sort of one of those things where like you are going to build something and maybe it is cool, maybe it gets adoption. There is plenty of cool stuff that gets completely ignored constantly, but the question is, is it going to be successful? And the only way you are going to find out is by building it.

And so at that time, the decisions that we made were perfectly reasonable and I will defend them. I mean, I don’t think that everything went right. I don’t think that we did everything right, I don’t think that I did everything right, you know I mean, I am not going to stand here and say, oh we did everything right. It was just the forces of the universe forced our hand and we couldn’t do anything right, I am not going to say that.

But, as far as taking a pragmatic approach to building a system, I mean, you are not building to scale Twitter on the first day to millions and millions of users. And you shouldn’t, because you are going to spend the better part of six months or a year doing that if you are building it from scratch. And then, you are not going to build Twitter, you are going to build the system that no one cares about and you are going to throw away all of that effort.

Mike: But, you are not actually answering the question though. I’d actually love to know like if you were building it from scratch today knowing that the growth was going to happen, what would you do to make it bulletproof?

Blaine: Sure. Sure. So, if I were to build it from scratch to make it bulletproof, the messaging system, you know, I would take a lot of time looking at that. And I mean frankly, I don’t think it is a secret that the hard part of scaling Twitter is not the messaging part of it. That’s a fairly well understood thing. The hard part is well SMS and the second part is displaying archives; so, displaying your friends’ timeline.

There are some complicated things in there, I mean, I am not going to get into them on the phone call, but building to optimize that and make that scalable. You know I’d love it if there were - if [indecipherable] or Hypertable or any of these other alternative data storage were usable right now for large scale systems, they are not, they are moving very quickly and so hopefully they will be.

But, right now, I am not sure Google or Yahoo!, I mean, it is really hard to say whether or not the data structures like these new systems, these new database systems that are being built, are going to work for what you are building, like MySQL, it is pretty awesome. I mean, it does some really amazing things. It is not perfect. If I were starting from scratch, I’d probably just use EC2 or Google app engine and say [indecipherable] it, like I want to build something cool quickly, I am going to use that and…

Steve: Hey Blaine? So, if you could do that, why not start right now and start building out redundant services on EC2 or app engine and then cut over to them?

Blaine: Because it is not that simple. I mean…

[crosstalk]

Steve: But you said that’s what you would do.

Blaine: Right, but I mean when you have a system that you are running 24×7 and trying to deal with all of the issues there and you got a small team, obviously you don’t have the luxury to sit down and start rewriting a parallel system. I mean that is what Friendster did and it didn’t get them very far, did it? So, Blogger did it as Eric Case who is sitting right here mentions and it took them two years and they were down most of the time, right?

Scoble: Blaine, this is Robert Scoble, how would you respond [background music].

Mike: Somebody’s phone is playing music.

Steve: Hey Robert, turn the music off.

Scoble: It is not me.

Mike: Somebody put us on hold.

Scoble: Not me; I can still get around it.

Steve: Everybody can just…

Scoble: Steve, can you kill the phone number.

Steve: I don’t know what that is…

Scoble: Somebody put us on hold.

Mike: They told me to.

Scoble: Somebody is going to get a talking to when we come back off from hold.

Mike: Wow, that really sucks.

Steve: All right, here it goes.

Scoble: What?

Man: Hey guys, I want to drop off, call back…

[cross talk]

Steve: OK. I know [cross line] line.

Mike: Whoever this is, they should be banned permanently just for doing this.

Steve: Just kind of dropped the out dial.

Scoble: I mean it is… there it is [cross talk/music] they put us on hold.

Steve: OK, there we go.

Mike: Who was it Steve?

Steve: What?

Mike: Who was it?

Steve: It was Mars calling, what do you mean who was it?

Mike: There was somebody on the call who put on us hold.

Steve: It was the phone service, I was… can we move on?

Mike: It was Steve OK, let us move on.

Steve: It was me.

Scoble: So, Blaine, this is Robert Scoble, how would you respond to the claims that some of the architectural decisions were really poorly made. For instance, I keep hearing that all of the little thumbnail pictures were all in one folder and not in separate folders where they would be more efficiently served out.

Blaine: The profile pictures?

Scoble: That is what I hear, I don’t know enough about the architectural details, but I keep hearing this rumor when on that parties has got that, a lot of the architectural details were just not well thought out for scaling up a system like this. And now, it is really tough to re-rip out that code and change that. So, just like you said a few minutes ago that it is really rough to rip out the code and change it once you have made a decision like that, how is that…?

Blaine: The profile images have been on that site for a long time. That wasn’t a huge deal to change.

Yes, I did not write that code in my defense. But, it was a pretty silly decision to plug all the profile images in one directory. But, I don’t think, I don’t recall that ever being a major problem as far as scaling the site. Displaying text images is not difficult.

I didn’t really come on the call today to talk about scaling Twitter because I can’t say very much. It’s one of those things where… I don’t know. If you don’t like it, build something else, I guess.

Man: Well, there is certainly…

Blaine: Obviously, Twitter needs to address these things. My sole focus for several months was making sure that the site didn’t go down. So, they are either going to address the issues or not.

[crosstalk]

Steve: All right, but I still think - Blaine, without getting into the details of what could have been done, I still don’t understand your answer to my question, which was why not build a black box outside of Twitter and then import the data over time so that you’ve got two machines essentially running simultaneously, one of which is not visible. And then, cut the power on the other one and move on.

Bob: Because you have tens of thousands of lines of code. You can’t just write a new system and say, “Oh, that works,” because it’s complicated.

Mike: Steve didn’t say that would be an easy thing to do.

Bob: Right. Right.

Mike: But, why isn’t there a migration strategy? I think, that’s the question.

[crosstalk]

Blaine: OK. OK. OK. OK. Right. That is a fantastic point. There is a migration strategy. Of course there is. You’re not going to run the same system forever and ever.

I’m just saying, when you said, “Why don’t you just move over?” … up until October, maybe it was September, we had three engineers total, and one operations guy. So, it’s not that easy.

If you are trying to keep up a site and you’re trying to do all the maintenance and the management…

Scoble: Until October you had three engineers, including yourself?

Blaine: Yes. Including myself. Three engineers. One operations guy.

Mike: How much? Three plus one ops?

Blaine: Yes.

Scoble: Mike, how much funding does Twitter have offhand?

Mike: Well, I think, they announced at five. They closed their next round but I don’t have the details.

Scoble: They have three developers?

Steve: All right. So, we haven’t got a lot of time with Blaine.

Mike: Blaine, how many total employees did you have in October?

Blaine: I think, about 10.

Mike: What were the other seven doing?

Blaine: Product and support and various other things - whatever other people in the company do.

Steve: All right. So, I’d like to ask a question before you have to leave, which is about the Track. If you look at the Follow mechanism, which is essentially a social graph and then you add Track, which is essentially a discoverable social graph…

Blaine: Sure.

Steve: … And you were to add the capability of being able to take the signals that you get from repeated hits in Track to build out algorithmically or at least automated sign-ups of Follows and automated un-signs of Follows, wouldn’t you have a much more dynamic kind of social graph possible that pretty much no one with a social platform has?

Blaine: Just to make sure I understand, you mean if Twitter were to broadcast notifications of so-and-so followed this other person or… like that sort of broadcasting?

Steve: Well, you can talk to it and by typing into the stream you can say ‘follow’ and ‘unfollow.’

Blaine: Right. Oh, and so you just broadcast those to everyone?

Steve: Yeah, exactly.

Blaine: Yeah, right.

Steve: A perfect example would be, you’ve got somebody who keeps talking to you and you want to follow them for some period of time. But, you might not want to follow them in general, because it’s going to bloat out your cloud. So, what you would do is have the software ready so you say, “OK. I’m interested in you. I’d like to not only get what you are saying moving forward, but I’d also like to see the last 10 Tweets of this guy.”

This is something I do typically all the time as a result of Track.

Blaine: Right.

Steve: Is that something that can be built with the current infrastructure? Yes or no.

Blaine: Yes. David [indecipherable] posted a while - I think, October - he posted an application that takes apart some, that gives you a relationship update stream. I certainly, wanted to build that and just never had the tools.

Steve: But, you are not there anymore.

Blaine: Right.

Steve: So, the question is, what do we have to wait on in order to get that functionality?

Blaine: Email them. I’m not sure. That’s up to them. So, obviously I can’t do that. Tracking that data - Google is doing a little bit. I guess, they don’t have the streaming version of it so you can’t see the events happen. So, you would have to build some kind of tif engine or something against Google’s social graph API.

Mike: Well, somebody could do it automatically outside of Twitter right now.

Blaine: Right.

Mike: If they get [indecipherable], like the MySpace and then they use that functionality and then it’s just a matter of them using the API to automatically follow and automatically lead people on your behalf.

Blaine: Right, if the Follow and Lead were broadcast, which they are not right now. So, that’s the blocking issue. Yeah, yeah.

Mike: Oh, I misunderstood. I think, I misunderstood the problem. This is what I gathered from it.

I thought Steve was saying, “Hey, I want to track this keyword. I don’t only want to see that one message. I want to see the messages from that person for the next 10 minutes or the next 10 messages or whatever.”

Blaine: Right. Right.

Mike: Is that what you mean?

Steve: Well, you can do it by just following them and then following them after 10 minutes.

Blaine: Yeah, yeah.

[crosstalk]

Steve: So, that’s doable. The question is where you can go back. What I heard you say was there was something you couldn’t do. I couldn’t tell whether that was Bob or you saying it currently. Can you say what that was?

Robert Anderson: This is Robert Anderson. I think, what Blaine said is that you can’t get notification of people’s following and unfollowing. Right?

Blaine: Right.

Robert: You get notifications of tracked - that is the tracked keyword.

Blaine: Right.

Robert: You can’t get those either.

Blaine: Right.

Robert: You can’t ask what is being tracked.

Bob: You can. You can send a message to Twitter saying, “Track.” It will tell you what you are tracking but not the overall who it is tracked by.

Robert: API’s don’t support it though, right?

Bob: Right. Right.

Robert: You would have to go through the XMPP gateway.

Blaine: So, everyone should think of the XMPP step as an API. That is the thing in which I am most interested coming out of Twitter. XMPP as an API is awesome. There is tons of stuff that you can do with it.

And to say that the Web APIs don’t support it… well, it doesn’t really matter because you can just use the XMPP API.

Robert: Well, it does matter if you have to support both APIs to get what you need out of the system. That’s inelegant to say the least.

Steve: Yeah, but we are not talking about elegant here. We are talking about doable or not doable.

So, when you say you think of it as an API, can you describe how a developer would actually approach that?

Blaine: So, most of the stuff that is being done right now with XMPP API is like for total stream. So, if you want to track the full public timeline, you can email for that and they will decide or not to give you the full feed. And then, you get every update as it comes in and you get a full atom, like the atom stanza right there in the XMPP body.

So, if you are using like ADM to track Twitter, you also get the atom feed, you just don’t see it. And so you can use that whole data to then do anything that you would in a normal Twitter API. You can follow people, you can unfollow people; you can track things, you can untrack things. It is more of an event-driven thing, rather than like calling some web URL, you do the action just like you would using Twitter from like either SMS or the Jabber interface.

As far as like thinking about Jabber APIs in general, I mean you can do the same thing like you can… I mean, think about all of the services that use email as an API. So, I take some photos and I email them to Flickr and then Flickr posts them. That is the same model. It is just that Jabber gives you presence and it gives you a richer sort of framework to think about the dataset that you are transferring.

So, I mean, Jabber is just email-improved basically and usable as an API. They have been describing this stuff for ages and they have descriptions of every possible thing you could imagine doing with Jabber. It is a little bit attempt to read and kind of get into, but I have been working with a number of people who are trying to change that and trying to make it more accessible to the average developer and what not.

Steve: All right. So, now we are back to the decentralization question, why do you think or do you think that there is any possible way that Twitter can be decentralized or do you think it is completely under the control of Jack and Ev, and it is not up to the community?

Blaine: It is a political decision, but I mean if applications in the community build… if people build applications that do decentralized Twitter and there are example applications that are not well designed necessarily, but they have the core functionality, then it is just a matter of time in my opinion like I see social network in general.

Facebook, MySpace, Flickr, Twitter, all of these things, like CompuServe and AOL in 1995, they had their own email systems, they had their own TAP servers, they had everything internally. And they are like, oh we are not going to play with anyone else and then people realized, oh I can just set up an email server. It is open source, I will just go and install it and then I don’t have to deal with CompuServe and it was over. I mean CompuServe and AOL had to “separate.” They had to play with everyone else.

And if I can go into Twitter and say, I want to follow lia@pownce.com or ralph@jaiku.com, I mean, that is awesome because now I don’t have to move my contact list over to Jaiku. I don’t have to use Jaiku and Twitter. I don’t have to use Jaiku, Twitter and Pownce and the next one that comes out in order to get the full experience. I can just say, you know I like the features.

Steve: I understand the pipe dream thing, what I am asking about is, what is the gating factor right now. I mean you said something about vendor. Another vendor has to go and send them an email and get permission to be able to access the full XMPP stream, is that what you are saying?

Blaine: Sure, but right now, right. But, I mean, this stuff is so new. I mean, I am talking about right now. I think, from a… hopefully.

Steve: So, a solution for scalability or reliability that isn’t about right now is meaningless?

Blaine: Perhaps, but I mean it is I mean I am not… honestly the Federation stuff doesn’t handle like it doesn’t fix the scalability problem, don’t confuse the two, they are different things.

Steve: I am not confusing the two, others have.

Blaine: Right. Right. Like if you have a number of large federated services like if there is, let’s say there is 500 million or a billion micro blogging users, you still have to show them the history of every message that they have received; you still have the scalability problem. I mean, there is solutions for it, but you still have to solve that and you still have to build out large systems to be able to deal with traffic and all that kind of thing.

I mean, I am just looking at it from all of these open social and et cetera, et cetera, some announcements recently. Really, they are just saying you can use our walled garden to kind of improve your site, but it is still a walled garden right, it is not actually changing thing. So, that’s… I mean, it is a idealistic, it is a pipe dream at this point. But, I know a lot of very smart people that are working on these systems and looking at how they can build these sorts of solutions. So, whether it is a year out or six months or two years…

Steve: I understand. I know you haven’t got a lot of time here, so I am just trying to compress this. So, given that fact, what is the fundamental gating factor right now, is it some permission to be able to access the full stream?

Blaine: Probably.

Steve: How do you query the core of the Twitter information base outside of going to the Twitter core?

Blaine: Outside what? Right, if you want to do it by polling, you can do that, but that doesn’t solve the permission thing. Right now, you can’t do it and there is some… I mean, I can get into all the technical details. I’d say it is half technical partially because no one implemented a system that does this yet.

I mean, Twitter has something internally that is fairly close, but they need to do some more work on it. And then as you said, it is a political thing, does Twitter want to open up the network and that is a business decision for them to make to decide whether or not it is going to make their product more valuable.

Steve: Right. But, the way to encourage them to make that business decision would be to come up with an alternative that basically did the same thing, harvest a month or two of data so that people can go back and understand what the threats are going back as a tool and then basically say, either do this or we are all going to leave.

Blaine: Sure. I mean, if you can just point to the Jabber spec, the Jabber XEP-060, the pubsub spec and then there is couple of other ones that Peter and Ralph and Joe Hildebrand have written. They have described how to do all of this stuff. I mean if you want to present this compelling argument to say this is what we should be doing, that is easy. I mean the technical side of it is too complicated to describe in a phone call, but it is well understood how to do that. There is no need to reinvent the wheel or anything. But, if somebody wanted to put forth some kind of consortium, well I mean, you can do that right.

[background noise]

Steve: Is that Scoble?

Scoble: Yes, sorry am I still on.

Steve: You have to get some place.

Scoble: Are you hearing me. I thought I am.

Mike: No, Robert, we don’t hear you.

Steve: Go ahead.

Blaine: Yeah. So, I mean, if you want to make that push, that is up to whoever decides to do that to make the political argument and you know honestly gathering up strength to be able to make it happen and that is as much a political move as anything and like you have the full community.

Steve: Right. You will be better to encourage them to do that rather than to go to all of the extreme work in order to be able to accomplish something that they should be doing anyway, which is I think, your point in the first place.

Blaine: Right.

Steve: OK. So, Bob Lee, do you have any questions?

Bob: No, sounds good.

Steve: All right, so.

[crosstalk]

Bob: Well, you know, it’s… I think, you guys just kind of touched on it, but it’s like, does anybody want to do that right now? I mean, is anybody asking Twitter to do that? I’m sure if somebody was out there kind of building, I guess, Twitter’s sister and asked them to do it, from what I’ve seen of them, it looks like they’d probably say yes and work with them, right?

Steve: Are you asking me? Because I have no idea.

Bob: I’m asking you guys, yeah, what you think. I mean, you keep asking whether you think Twitter would work with these people, but I’m asking: who are these people? Is there even anybody out there yet, or…?

Steve: Well, I mean, Robert was talking about brand feed. You know, there are a bunch of vendors who are living off of the backwash of Twitter right now that could conceivably start to pipe that data stream to a more persistent store and then start building that in infrastructure to be able to harvest it.

[crosstalk]

Mike: Is this Bob Lee from Google who is on the phone?

Steve: Uh-huh.

Mike: Oh. I didn’t know he was on. Hi Bob!

Bob: Hey! [laughs]

Mike: I’m a fan of your work.

Bob: Oh, thanks!

Mike: You weren’t at Campfire One the other day, were you?

Bob: No, I was sick, still.

Steve: Blaine, are you still there?

Blaine: Yep, I’m still here.

Steve: OK, so did you have an answer to Bob’s question?

Blaine: Well, yes and no, I mean, there are so many people working in this field, and I think, I mentioned, many of them on… Cantor had a post that I commented on, you know, there is Mediamatic in The Netherlands that’s working on not specifically microblogging, but the same kind of federated structure.

So, I went to a thing that they did in Amsterdam back in December, and they do consulting for nonprofits and stuff, and so every time they set up a new site for a new organization, they have to go and say: “OK, everyone that was on the last site, move over to this new site, we’re doing a new campaign,” or whatever.

And so they’re really interested in this data portability thing, but what they have found is that people don’t want to set up new accounts on every single site, and so they are looking at ways to sort of say, you just have one canonical identity, and you then you can exchange data between the sites as appropriate, right?

And so this is where OpenID, and Olos and all of these different things come in, but you can also use…

[crosstalk]

Bob: Passport [laughs]

Blaine: [laughing] … Potentially. So, you know, you’re doing all these things, and you can build in the federated stuff. And I mean, the Twitter stuff is just Atom, so you could apply that to photoblogging, or regular blogging, or location services, or whatever you want to do. So, there’s people, NoseRub is an organization in Germany that’s working on this stuff, Dolfa is doing some experiments with this kind of data exchange and whatnot.

I mean, there’re lots of people doing things. I’m not sure that any of them are really interested in like, up-seating Twitter. So, you know… who knows?

Steve: Right. And then finally, have you followed what Dave Winer has discussed along the lines of moving some of this data to RSS as sort of a back-up model? I think, that’s what he’s talking about.

Blaine: Yeah, I mean that…

[crosstalk]

Mike: Isn’t that the least interesting aspect of all this though? This backing everything up?

Steve: I think, it is the least interesting, but I think, we’ve covered everything else, so I just want to be complete on this call, and have Blaine talking about what he thinks Dave’s talking about.

Blaine: Honestly, I feel like I have a back-up already. My IM client logs all of the updates that I receive. I have a back-up, so I’m not really sure what the goal is there, because if Twitter goes away, the data isn’t the hard part.

I mean, the data is federated and all; people have the data all over the place, because of the RSS feeds and because of the Jabber stream and all of that sort of thing. I mean, how many search engines are there that have complete back-ups of basically every query that has ever been sent?

It’s the network and the community and the infrastructure that is the hard part. And so, having a back-up, well, it sort of goes down, yeah you get to go and look at your archives, but I don’t see what else.

Steve: OK, your hit list if you were trying to enhance the Track functionality, what would the top three items, or even the top one item that you would want to do that only Twitter can do at this point?

Blaine: Right. Well, that only Twitter could do, there’s probably not a lot. I mean, Twitter has obviously the optimal interface for it. So, I would want to be able to specify like, just tracking “@ replies,” rather than tracking the screen name.

So, right now, the Track stuff just gets rid of all the punctuation, and so having some concept of punctuation in there would be helpful.

Bob: Well, Track does, Untrack doesn’t [laughs].

Blaine: [laughs] Uhmm… they’re the same thing. Anyway, so [laughter]…

[crosstalk]

Steve: I don’t see that as a bug, I see that as a feature. Because I don’t debate any reaction [indecipherable] for the @ sign other than to create opportunities for people who are really noisy.

Blaine: Yeah. I mean, that’s clearly a product debate, and there’re arguments on both sides.

Steve: So, what is the argument in favor of the @ sign?

Blaine: Because people don’t expect the behavior. I mean, you saw his post about bad replies recently…

[crosstalk]

Steve: Yeah, I haven’t read it, what does it say?

Blaine: Basically, it just says that it’s a really confusing feature. It is. It is a really confusing feature; people don’t really understand how it works and why it works that way.

We had a lot of discussions about changing it, so that for example, if you reply to someone and you had a third friend who only followed you but didn’t follow the person that you were replying to, that your friend would see that update, and essentially see only half of the conversation.

And then, we ran some numbers and it turns out that you would get a significantly larger number of updates as a result of that. And so it would kind of pollute the stream for a lot of people, and before we added the feature, a lot of people were like: “Why is all this stuff in my stream, it’s too much information!”

Some people liked it, but you know, these are people that follow 5000 people, so… [sighs] what are you going to do, right? I think, the ideal following number is somewhere around 40 or 60 or something.

So, I could go either way on it, but that’s neither here nor there; I’d like to have some kind of punctuation so that you could track specific things.

The other thing is being able to filter stuff, so being able to say, like: Track Blaine and not David, is an example. So, I can track my name, but not have to see everything about David Blaine. And I think that that’s… I mean, Track isn’t very widely used, I don’t think, as far as the whole Twitter user base is concerned, but I suspect that that’s the problem that people run into when they’re tracking things.

Steve: Totally, so…

Blaine: Yeah.

Steve: Yeah, and the idea that something is not used much is obviously one of the reasons that you guys got in trouble in the first place, because the service was used for purposes that were not anticipated.

Robert: Hey Steve, I’m sorry, I’ve got to drop off the call. I just wanted to thank everybody…

[crosstalk]

Steve: OK, thanks Robert!

Robert: … and I’ll see you soon. Talk to you. Bye.

Steve: OK. You know, the unintended consequences of the service suggest that Track might be used by a lot of people soon.

Blaine: Right. I mean, I think, Track is a fantastic… it’s definitely one of the things that I was most excited about.

Steve: And it allows you to keep down the number of follows, and still have a larger slab.

Blaine: Right. I think, the thing that is interesting about Track is that it really pins it like for so long there has been this discussion about like intelligent agents or the services that give you information that you are interested in without forcing you to step through reams of information that you are not interested in.

And I think, Track is just like the first inkling of that actually happening. For me, that’s really exciting, because if you can start applying that to, like I always hoped that Del.icio.us would do something along those lines, where you can sort of say: here is all of the people that I follow on my Del.icio.us feed, here are all the links that I link to. Give me something that I am not seeing and let me track like pegs and stuff, but filter it based on these sort of metrics and what not. And it never really took off that way, but that kind of thing is really exciting to me.

Bob: Track could also be kind of a differentiator for Twitter too, because I mean if you imagine like AOL or somebody going in and kind of doing the same thing as Twitter on top of their existing instant messenger update, they are not going to be able to do something like Track, because none of their users expect that their status updates are going to be broadcast and approachable by everybody on AOL.

Blaine: Right. I should pack-up and go do that.

Bob: [laughs].

Steve: Yeah, yeah, OK.

Scoble: Blaine, you mentioned that the perfect number of followers or people you are going to follow is about 40 to 50 and I agree with that if you look at it from one angle, but from another angle I totally enjoy being on and watching a large number of people. I would never have known about the Chinese earthquake if I had only watched 40 of my chat buddies or even worse 40 of my close friends. I would never have seen a message from somebody in Shanghai that I have never met.

And so, I really enjoy watching a large number of people in some context, but in other context I’d really like to have a small number of really handpicked people that I am following. And about a year ago, when I came and visited Twitter, you guys were talking about adding grouping features and we never saw those features make the light of day.

Do you think we are ever going to see that kind of feature where I can have one group, so that if I only want to watch a small number of people, I can watch that and then another group where I can watch everybody, you know, 20, 000 people; another group where I can watch 1, 000 people.

Blaine: Right, I hope so; I am not really sure.

Bob: I can say that is the number one request of people.

Blaine: Yeah, I mean groups are really complicated, mostly because of the mobile interaction models. If you can imagine switching groups on the phone, it is not an easy thing you can anticipate for. So, I mean, in my opinion that’s… and obviously there is architectural and scalability issues that prevents us from doing [crosstalk] have a site that goes down and then say: oh we introduced groups which makes it harder to keep the site up. We are not going to do that.

Steve: Yeah, but I mean, even forget the scalability issue. I mean you got a Track mechanism right now which will do exactly what Robert is talking about.

Blaine: Right. I mean, people have implemented group. It is not quite that simple. When you start factoring in social stuff into the Track mechanism, it becomes more complicated. I mean, you have to deal with a lot more issues once you - like right now Track doesn’t care about, it only looks at public Twitters which makes it really simple. As you start to have interface and sort of like really find controls and you know, you are going to complicate things quite a bit.

Steve: Why do we care about anything that isn’t public?

Blaine: Because you saw all sorts of people that aren’t public.

Steve: Oh, so in other words you could follow somebody who is not public?

Blaine: Right.

Steve: So, why can’t you, since you have already seen what they are saying because of the contract between you and them, what has that got to do, you are already seeing them.

Blaine: It just seems that you…

Steve: You don’t recover that, what you look for are the people that are talking publicly who are in the same ball game. And once you do that, then you can apply the follow handshake and then you are where you need to go.

Blaine: Sure. As far as the interaction model and as far as like there are some subtle things with the implementation that make it more complicated. You are increasing the complexity of the sort of request that you are handling.

Steve: I don’t buy what you are saying there.

Blaine: I mean, you can buy it or not, but [laughs]…

Steve: I will put it to you in a different way. I will ask you the questions sequentially and if you say yes to all of them, then I am right.

Blaine: [laughs] OK.

Steve: OK. So, what prevents you from having a mechanism that basically uses public discovery via Track to identify people of interest. At that point, you communicate with them directly to establish some sort of a follow mechanism and nail that up. That can be concealed by the interface with a software, so that it appears to be intuitive.

Blaine: Yeah… I mean, that wasn’t a yes/no question. That’s a complicated problem. I mean, I don’t actually have time to get into it right now, but I need to…

Steve: We will follow up on it the next time we meet. I will write it all down and then we will wait for you.

Blaine: For what it is worth, people have implemented group functionality outside of Twitter, so you can do direct like private groups outside of Twitter. I mean you could conceivably implement public groups as well. So, I mean, just to throw that out there, the one that I know of was written by a guy named Kellan at Flickr who wrote Twitterpated. I think, it is at Google Code. The project name is just Twitterpated.

So, he built it to sort of support thinking he was trying to build an internal group at Flickr. So, that is out there and you can just install it on your server and then you got to find a group. I mean, that is one of the fantastic things about Twitter is it is completely open. You can build stuff on top of it and just run with it, right?

Scoble: That is really cool. It seems like every idea that I can come up with, there is already some way to work with the Twitter people.

Blaine: Right [laughs].

Steve: All right. So, what are you working on right now?

Blaine: I can’t say it, but I will be able to say soon.

Gillmore: What’s soon?

Blaine: Soon, yes.

Steve: Is it related to Twitter?

Blaine: Ahh no.

Steve: OK. All right, well I very much appreciate you coming on the show. Hope we can meet again in the future?

Blaine: Thanks for having me. Sounds good.

Steve: OK.

Blaine: Talk to the guys there.

Steve: Take care.

Blaine: Thanks.

Steve: So who is still on the call?

Scoble: Robert Scoble.

Bob: Bob Lee.

Steve: And that’s it, OK.

Bob: Is that it?

Steve: I think, so. I mean you asked all your questions.

Bob: Yeah.

Steve: Yeah. This is Steve Gillmor. This has been Gillmor Gang for Thursday May 15th. I want to thank everybody who showed up and especially those who didn’t, we will see you again tomorrow, bye-bye.

This entry was posted on Thursday, May 15th, 2008 at 5:41 pm and is filed under Gillmor Gang. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.