Walter Rumsby's Web Site

Tom Coates: Designing For A Web Of Data

2008-09-21

You can never have too much sweet, sweet data

At Webstock 2006 Flickr was the darling of the conference. It seemed to be mentioned by almost every speaker so I was curious to see which site(s) would be fêted in 2008. Flickr still got more mentions than any other site (Twitter came in at number 2 by my reckoning), but this time round there was more of a focus on the Flickr platform and not simply on Flickr's design innovations (AJAX, tagging, vowel reduction, etc.).

Tom Coates covered this most effectively in what I personally saw as the best/most important/key speech of Webstock 2008. For me the theme of the whole conference shifted from the 06's focus on web design & usability to the idea of the web as a platform and Coates addressed this topic directly.

What follows is more or less a rough transcription of the presentation.

Introduction

Flickr, Twitter, Google, Upcoming, Facebook, delicious, last.fm, Dopplr and Amazon are all special sites and services. They are more than just web sites. They've managed to break out of the frame of the browser and manifest themselves onto desktops, devices and other sites. They provide a platform on which other things can be built by anyone and as a result those sites have benefited - they've generated more revenue, more ethusiasm and had a bigger impact than they would have had if they did not provide those platform services.

Five Things

Your Web Site Is Not Your Product
You Must Play Well With Others
You Can Never Have Too Much Data
Hierarchies Can't Take Weight
Collaboration Is Important

What Is The Web Of Data?

The web of "pages" is changing to the web of "data". Web pages are now simply one way of accessing the data.

In this context "data" means "stuff that a computer can access" (e.g. blog posts, pictures, videos, book reviews? resources).

So what we have is

A web of data sources [and] services for exploring and manipulating data and ways people can plug them together.

Your Web Site Is Not Your Product

Your territory is anywhere your network touches. For example people can interact with Twitter using the web site, IM clients, SMS, anything that can broadcast information to Twitter. 90% of the traffic to Twitter is via their API (i.e. only 10% via the web site). This means that the Twitter web site is not the product - the product is a way for people to keep in touch via short messages and the site is merely one instantiation of that.

Similarly there are other ways to interact with Flickr: eStarling, Moo cards, widgets, web site badges, desktop clients, mashups, etc.

In the case of last.fm many users interact with last.fm through a desktop player app, and not through the web site at all. OS X users can also play customised radio streams in iTunes and not have to spend any time looking at the last.fm web site.

This is all stuff you can do today! As the network touches more "things" computing becomes ubiquitous and while you could say that many of the applications today are niche, toy, "play" apps there are other possibilities for a wide range of applications.

When the web of data bleeds into the real world:

Physical objects respond to and visualise data from a network (e.g. Nabaztag; Ambient Orb)
Interactions with physical objects allows people to change data stored on the network
Physical objects act as a sensor that writes to the web of data (e.g. Wattson)

You Must Play Well With Others

If a web site is just one manifestation of your data you need to design for recombination because there are opportunities when your data can be mashed up with other datasources:

You can drive people to your site (e.g. Amazon)
People might pay for your data
You put yourself (your data) in an ecosystem that contains similar data
It makes your service more attractive to users
It reduces amount of central development (Flickr couldn't build everything that people wanted from the product so they let others build on top of Flickr)

Other examples: Oakland Crimespotting combines two datasets which can be understood in the context of each other (crime by geography, geography by crime); Fire Eagle (provides a geolocation service that other services can use to provide another layer of data on top of an existing service).

You Can Never Have Too Much Data

2,264,930,075

Photo by wasabicube

2,264,930,075. That's the number of photos on Flickr as of February 2008. That's 1 photo for every 3 people on the planet and that number increased by more than 900,00 in the 12 hours before Coates' talk.

How do you make that scale of data communicable, explorable, navagable to users? By finding more data!

Capture meta data during the production of the object (e.g. data about when a photo was taken; the mode used; etc)
Analyse the object
Crowd-source the creation of meta data (e.g. tagging)
Perform behavioral analysis (e.g. Google - PageRank is based on how other pages link to a site)

For this photo of Piccadilly Circus, there are a number of related data axes:

Photos by Tom Coates
Photos by Tom Coates taken on 15 August 2007
Photos by Tom Coates uploaded on 16 August 2007
People who call this photo a favourite
Etc. (by set, tag, location, camera info, license)

It's even possible to explore photos by apeture if you know what you're doing. The only reason this feature isn't more obvious is that the Flickr team haven't worked out a good user interaction to do this via the web site.

Hierarchies Can't Take Weight

Information Architecture circa 1999 was about organising your data into hierarchies (taxonomies). IA circa 2005 was about organising your data into folksonomies (e.g. tagging).

You can have both! Amazon.com provides multiple ways to explore data, using both traditional taxonomies and folksonomies.

Hierarchies don't scale. Peter Morville who presented earlier also made this point and it's not something that is unique to the web. "Weblike" navigation is a much better way of exploring large data sets - "top navigation is just the jumping off point".

Collaboration Is Important

Coates works at Yahoo! Brickhouse which is set up to run like a startup inside a large organisation. The teams at Brickhouse are small, inter-disciplinary teams (e.g. BravoNation, Fire Eagle).

A key idea here is that idea generation is not the responsibility of one role (e.g. programmers, graphic designers), but is shared across roles. Coates strongly believes this is the best (only?) way to create new products.

Conclusion

Not a lot to disagree with in what was covered. It's interesting to me how web-based applications have evolved and have to deal with tremendous scale and complexity. Back in 1999 I can remember a friend bemoaning the fact he was working on web sites and not "real" applications. As we grow a richer appreciation of how we can develop applications on the web the kinds of solutions being developed are far more interesting than the client/server applications of the 90s.

If Google Maps was the catalyst for CEOs/CIOs demanding AJAX-based solutions it stands to reason that once they understand the concept that you can create a web-based platform their view of what's possible will broaden in a similar way (then again maybe there was an element of "oh this graphic/web designery stuff is easy, we could do that too!"; it seems likely that if "normal companies" were to develop platforms that they'd turn to Google, Amazon, etc. to provide [some of the] infrastucture).

And finally, while I agree that "weblike navigation" works better for large datasets, it's not a big deal if you don't stumble across a particular photo of your favourite MC when using Flickr, but I can't help thinking that for certain classes of applications there is a large set of information that users have to see. Hierarchies obviously don't work in those situations, but I'm not sure folksonomies completely solve that problem either.

Lots of stuff to think about. Since it took me 7 months to write this post I can now add that Coates' is returning for Webstock 09!

Comments

On DOM Ready

2008-06-25

YUI provides the YAHOO.util.Event.onDOMReady method which "lets you define a function that will execute as soon as the DOM is in a usable state."

I've heard reports (both internally and on the YUI mailing list) about this method going a bit haywire - specifically firing before the DOM is usable - in Internet Explorer, but these seem to be in weird situations and it has been (seemingly) impossible to create a test case to demonstrate when/where/why this problem happens. This makes it pretty difficult to raise a bug that the YUI team can look at, so the problem still (apparently) exists.

I was looking at an old post by Dean Edwards that suggests using the script element's defer attribute to work out when the DOM is "usable" in IE. Edward's solution suggests using conditional comments to remove the defer attribute for non-IE browsers, but I wonder if that's even necessary.

This leads me to think, assuming your external script is entirely wrapped up in an onDOMReady function, why not throw a defer attribute onto the script element? My hunch is:

IE will defer the script's execution until the DOM is usable.
Other browsers will ignore the defer attribute, but since onDOMReady works for those browsers (assuming they are A Grade browsers) they will also wait until the DOM is usable.
By the time IE trys to work out if it should fire onDOMReady the defer attribute will mean that it's okay for it to fire, so you should sidestep the "onDOMReady fires in IE before the DOM is usable" issue.
All of this will happen before the window's load event fires regardless of browser.

I've set up an example page which appears to confirm this, but of course this page doesn't suffer from the incredibly tricky to replicate problem so this is very much a guess at a solution. The other thing is this assumes that all the code in your script runs inside an onDOMReady function, which might not be what you want.

I've followed this up with a message on the YUI mailing list to see if anyone can spot any problems with this approach and plan to update this post with anything that comes out of that.

Update: there is now a bug with test cases. Hopefully this is resolved in the next YUI release, but in the interim, or if you are using an older YUI release, it appears that adding the defer attribute to script elements does avoid these errors.

Comments

Next Previous Navigation

2008-06-22

FFFFOUND! is a social bookmarking site for images (only).

I quite like it. It feels like flicking through a book or magazine full of interesting images and I think one of the reasons that they're able to achieve this is thanks to the incredibly useful, and simple, navigation options they provide.

In the top, right corner of the browser window are some navigation links which also come with clearly indicated keyboard shortcuts.

I find it really useful to be able to hit the j key and navigate through the images in a way that's a whole lot nicer than hitting Page Down Page Down doesn't know where images begin and pretty quickly you'll have to fiddle with the scroll bars to fix things.

I've hacked together my own ultra-basic version of this for the posts page here in the hope that it is useful to some people.

(This is currently not so great in IE6).

Comments

A Few Weeks Ago

2008-06-01

Photo by Charis

Comments

Test

2008-04-10

Klute is playing in Queenstown on May 13.

You need Flash player 8+ and JavaScript enabled to view this video.

Comments

Webstock: The Final Countdown

2008-02-18

Well, here we go. Assuming my addition is correct here is the final memescore for Webstock 08:

Popular memes at Webstock 08
Meme	Mentions
Twitter	12
Flickr	14
iPhone	10
LOLcats	8

I'm sure some people will be surprised that the LOLcats count is so low.

There were a few other topics that got frequent mention. History was quite a common thread - Nat Torkington started it all off with the Crimean War, Liz Danzico talked about Miles Davis' Kind of Blue, Scott Berkun talked about Guttenburg and Newton, Jason Santa Maria talked talked about graphic design during the Renaissance and I'm sure if I scour my notes I can find more examples. Last.fm also got a few mentions as did Facebook. Russell Brown talked about SuicideGirls (I think he has a wee crush on Posh - but given that she is an attractive, creative, Mac fangirl who spends her spare time playing WoW and dates the owner of an independant media site maybe that's not so surprising).

So how did Webstock 08 rate? My opinion has shifted a bit over the last few days. Initially I thought the event wasn't as strong as 06, but thinking about it more that's probably because I attended two full-day workshops in 06, and only one half-day workshop this time. When it came to the actual conference I think I actually got more from the general conference speakers this time round (I've already made a series of redesigns to this site based on Luke Wroblewski's workshop and presentations by Dan Cederholm and Simon Willison).

To be honest my big mistake this time was probably forgetting to go to the Matterhorn (technically speaking I did pop my head in the door trying to find some Webstockers who ended up being next door at Mighty Mighty).

Over the next few days (or weeks) I'll post some more detailed entries on my favourite presentations - Liz Danzico, Tom Coates, Dan Cederholm and maybe a few others - plus the Luke W workshop, but as a general review of the whole event I'll have to say well done again Webstock.

Just to emphasise that point, on Friday evening I was chatting to Tom Coates and along came Nat Torkington who said hullo (to Tom). They both agreed that this was one of the best conferences they had ever been to. So I wouldn't be surprised if O'Reilly conferences borrow one or two ideas.

Meanwhile, I'm already looking forward to the next Webstock.

Comments