I'd like to forward this to the Crisis Mapping / Disaster Response mailing lists I'm on. They're been very active with Twitter. Is that OK?
On 04/16/2010 10:54 AM, Marcel Molina wrote: > Hey everyone. One of the things we talked about at Chirp is the new > Annotations feature we're working on. In short, it allows you to annotate a > tweet with structured metadata. We're still working on Annotations, but I > wanted to share with a wider audience beyond those I was able to talk to in > person at Chirp about how we're thinking of doing Annotations. > > * What is an annotation more exactly exactly? > > First off let's be clearer about what an annotation is. An annotation is a > namespace, key, value triple. A tweet can have one or more annotations. > Namespaces can have one or more key/value pairs. > > * How do I specify what annotations a tweet should have? > > Annotations are specified for a tweet when the tweet is created. When > submitting a POST to /statuses/update, you'll include an "annotations" > parameter with your annotations. We're thinking we'll provide two mechanisms > for specifying what a tweet's annotations are: > > 1. JSON > 2. form encoded parameters > > * How big can an annotation be and how many annotations can I attach to a > tweet? > > There is no limit on the size of any given namespace, key or value but the > entire set of all annotations for a given tweet can not exceed some fixed > byte size. That size isn't set in stone yet. We will be starting small > (probably 512 bytes) and growing it gradually as we incrementally roll out > the feature so we can gauge its scalability at various sizes. We'd like to > (no promises) have it end up around 2K. How you use that 2K is up to you. > You can attach one honking annotation, or a thousand+ tiny ones. You can > attach one namespace with hundreds of key/value pairs, or hundreds of > namespaces with just one key/value pair. We want to keep things as flexible > and open ended as possible. > > * What kind of data can go into an annotation? > > We'd like to allow for any arbitrary data to be stored in an annotation. > Arbitrary Unicode? Sure. MIDI? Go for it. Emoji? Yes please! There might be > some tricky edge cases though. Skip the rest of this paragraph if you don't > care about the details of edge cases... For one, since these annotations > will be serialized to, among other formats, XML, and we'd like to keep the > XML succinct, the namespace and key components of an annotation triple would > likely be an XML tag with its value as, well, its value. If that's the case > then the data of the key must be a valid XML tag. This greatly limits what > it can contain (not even spaces for example). If allowing all three elements > of the triple to contain any arbitrary data is more important than a > succinct XML payload then we'll design a more verbose XML payload. Up to you > all really. I've included examples of both options below. Make a case for > another proposal if you have strong opinions. > > * What constitutes a valid annotation? > > Aside from the size and data type restrictions listed above, another > requirement is that namespaces and keys be non-empty values. Values, on the > other hand, may be empty. In this way the namespace/key pair can be treated > like a flag of sorts. It should be noted: I'd encourage everyone to always > think of a namespace as a namespace, to think of a key as a key and to think > of a value as a value. Don't take the fact that a value can be empty to mean > that you can skip out on the whole namespace think and morph the namespace > into a key and the key into a value. While open endedness and flexibility is > a quality of the Annotations feature that I'm most excited about for the > developer community, this kind of approach seems prone to causing confusion > by undermining namespaces. > > * What namespaces can I write to? What namespaces can I read from? > > Anyone can write to or read from any namespace. We aren't planning on > enforcing any policy that restricts someone else from adding an annotation > with "your" namespace or seeing annotations only if they are logged in with > a certain account. In the absence of some really compelling reason to do > that, we want to err on the side of making this feature as flexible and open > ended as possible. Namespaces aren't intended as a way for people to claim > their little slice of the tweet space. Rather they are intended to > dramatically increase the possible significance of a given key/value pair. > If you want a given key to mean one thing and someone else wants that same > key to mean something else, and someone else still wants another meaning, > consumers of your annotations are put in a tricky spot trying to figure out > how to interpret a given annotation without the disambiguation of a > namespace. > > * How do we consume annotations? > > For convenience, we plan on including annotations for a tweet directly > embedded into that tweet's payload. The XML payload of a tweet I just > inspected at random came out to about 2K in size. The "worst case" > annotation would a little more than double that payload to probably about > 5k. We're erring on the side of thinking that the moderate increase in > payload size for tweets with annotations, even on slow connections, is both > more convenient and faster than the latency and inconvenience incurred by > adding another HTTP round trip. Though we'd like to provide an embedded and > non embedded option, the maintenance cost and fragment cache space increase > makes supporting both likely unrealistic so we're going with what we think > satisfies the 80% case. Push back as appropriate. > > * What will the payloads look like? > > This isn't final. The payloads could end up wildly different after we noodle > around in things like RDF and the semantic web's literature and all that > kind of stuff. You can't see me but my hands are waving vigorously. > > Given a hypothetical tweet, "Just got 'Although Of Course You End Up > Becoming Yourself' in the mail. Hopeful. Heart broken." > > JSON > > 'annotations': > { > 'iso': > { > 'isbn': '030759243X' > }, > 'amazon': > { > 'url': ' > http://www.amazon.com/Although-Course-You-Becoming-Yourself/dp/030759243X' > } > } > > XML option #1 which is succinct but restricts the possible values of > namespaces and keys > > <annotations> > <iso> > <isbn>030759243X</isbn> > </iso> > <amazon> > <url> > http://www.amazon.com/Although-Course-You-Becoming-Yourself/dp/030759243X > </url> > </amazon> > </annotations> > > XML option #2 which is more verbose but allows for namespaces and keys to > contain arbitrary data > > <annotations> > <annotation> > <namespace>iso</namespace> > <key>isbn</key> > <value>030759243X</value> > </annotation> > <annotation> > <namespace>amazon</namespace> > <key>url</key> > <value> > http://www.amazon.com/Although-Course-You-Becoming-Yourself/dp/030759243X > </value> > </annotation> > </annotations> > > If we went with XML option #2 it may or may not be a problem that it isn't > "symmetrical" with the JSON representation. On the other hand, JSON and XML > tend to be culturally at opposite sides of the Pithiness Spectrum. > > * Can I add annotations to a tweet after the tweet has been created? > > No. Like the text of a tweet, its annotations are also immutable. They can > only be specified when the tweet they are being attached to is created. For > talking purposes, though, if you want to add annotations to a tweet after > the fact, you could retweet the original tweet and attach annotations to the > retweet. > > * Ok, great. What should I use annotations for though? > > We don't know! That's the cool thing. Annotations are a blank slate that > lend themselves to myriad divergent use cases. We want to provide open-ended > utility for all the developers to innovate on top of. Some of us have > initial ideas of cool potential uses cases that I'm sure we'll start to > share just to seed the conversation as we get closer to launch. Developers > will experiment with annotations. Certain ideas and approaches will catch > on. Certain annotations will become standards democratically because > everyone agrees. Some might have diverging opinions. It's something that we > hope will grow organically and be driven by sociological and cultural > forces. > > * Ok, great. How are we going to figure out what Joe Random's annotations > actually mean? > > That's something we need to figure out as a community. But here is an early > idea: People could add some agreed upon "meta-annotation" that points to > something which *describes* the annotation or annotations that person is > using. Think something sort of like XML DTD, though not necessarily machine > readable. This meta annotation could point to a URL that simply has an HTML > document that gives a description with some examples of the various > annotations you're experimenting with or standardizing on. > > * Will it be in search? Streaming? Mobile? My toaster? > > We hope so! When we launch you will at minimum be able to attach annotations > to a tweet and consume annotations from a tweet's payload via the REST API. > Of course it would be awesome to be able to say to search or the streaming > API, "give me all tweets with this namespace", or "give me all tweets with > this namespace and key", or etc. We're working with the Search, Streaming > and other teams to make all this happen. We can't promise it'll be ready by > launch but we know it's killer and a must have and are trying to get it > ready soon. > > * When is it going to launch? > > This is, pretty much, the only thing a couple of us are going to be working > on until it's launched. We really can't wait to get it in your hands to see > all the cool things you'll do with it, so we're cranking to get it out as > soon as possible. If I had to provide a guestimate, I'd wave my hands in the > direction of 2 months for a early, incremental roll out. We not only need to > implement all the functionality, but we also need to productionize it in a > measured and responsible way to ensure its quality of service is high. > > In closing: > > We're really excited about Annotations. Annotations mark one of our first of > many departures from keeping in lock step with features on the web site. To > truly be a platform, we want to expose high-leverage general purpose utility > for the developer community to innovate on top of. Annotations is just the > first of several high-leverage-general-purpose-utlity features we're hoping > to get to after Annotations. > > Think big. Blow our minds. > -- M. Edward (Ed) Borasky borasky-research.net/m-edward-ed-borasky "A mathematician is a device for turning coffee into theorems." ~ Paul Erdős -- Subscription settings: http://groups.google.com/group/twitter-development-talk/subscribe?hl=en