Star icon to indicate updates At a glance

Writings and musings on the latest web trends and life, advertising, design, projects, and news from an avid and prolific web designer.
Up and about since 2003.

Click here for the RSS 2.0 feed Follow me on Twitter Pictures from my Flickr account

website »
Is your web design hurting your search engine rankings? Teknicks is a professional web design agency and SEO agency.

Instagram feed updated yesterday.


Mat Kearney

Most commented posts Most commented

Interesting Defunct United States Airlines
What I learned from the Boston Career Forum
Install Android 2.2 Froyo on iPhone 3G on Windows with pics and video
Getting FancyUpload to Work
Google Plus bar color changer extension
Home Sweet American Culture
An Engineer's Nightmare - Ocean Tower, South Padre Island
Python on the iPad
6 sensational unsolved missing persons cases
The Kite Runner and Atonement Compared




Jan 19 | In search of serendipity and soulful data

Every Wednesday I get the privilege to attend HCI lunches here at Stanford where a mishmash of scholars, professors, and students meet over lunch to watch someone's presentation on research (and proceed to tear it apart). It does result in some interesting discussions - yesterday being no exception - and I thought I'd share some thoughts on this elusive idea of "serendipity".

Terri Winograd, one of the luminaries in the field, put it succinctly when he said that

serendipity is to data as intuitive is to a user interface

What is intuitive for you may not be to me, just as serendipity cannot be quantified per se as a quality of some data. In this particular context, serendipity remains one of Monica Lam's holy grails - how might one sift through the masses of data we manufacture each day and reveal the ones that are particularly "serendipitous"? Jesse brought up the valid point that intuitiveness will likely not change as quickly as serendipity, which may make the latter even more elusive. Let us see. Merriam-Webster defines it as:

the faculty or phenomenon of finding valuable or agreeable things not sought for

Other sources inevitably include terms like "happy accident" or "pleasant surprise". Hence the keywords there are

a) "something that is useful or creates some sense of pleasure or happiness"
b) "something that is unexpected"

We can definitely cater to b) with what we have: tf-idf, in its roughest sense, provides some measure of information quality relative to rarity. Algorithms such as LDA do a pretty neat job surfacing themes and topics that are somewhat unexpected and which may provide some serendipity, but it is hardly predictable. What may be the right approach, then, is to consider technologies that surface information that is an aggregate, but not a composition, of existing data. Where we currently see a garden full of different colored apples and we rejoice at an algorithm's capacity to say "there are many different apples, and this one may be of interest", or "we can group your 100 apples into 7 different logical clusters", we may benefit from the approach that says "this garden has too many apples".

In that sense I feel the approach of creating serendipity by surfacing past data is inherently problematic. I do acknowledge the fact that getting better access to our past can be interesting (and serendipitous), but it hardly a worthwhile challenge. We may choose to omit certain aspects of our past history specifically because we don't want to be reminded of them - how would a system be able to discern that fact? Is Facebook's "People You May Know" feature one that uncovers "serendipity"? Are other people better arbiters of "soulful data" than algorithms will ever be?

Let me enumerate things that make me happy especially if this is something that can be provided to me online (without my explicit, active participation):

a) finding a long-lost friend
b) hearing good news about a close person's success
c) reading about a news article that is related to a current research topic or question
d) seeing videos of cats
e) finding a really sweet online deal that is within my budget and what I had been looking for

It's clearer now why websites like Reddit and Facebook succeed, because the narrow circles of individuals we associate are likely to provide us with those happy discoveries (this, as opposed to the worries of the echo chamber - this later finding is worthy of a whole different discussion).

Ultimately I think any system that attempts to pursue the notion of surfacing soulful data needs to be reminded of the fact that there needs to be a clear and transparent method for communicating how that data is derived and ultimately chosen for display. Also, there is a very fine line between letting people "search" for this data and providing it at the opportune moment. Maybe that is the key question: what is the opportune moment to provide someone with information? The most basic answer to that would be: "when they are searching for it". We can definitely do better.

In short:

a) surface information that a user cannot or would rather not spend the effort to search for
b) choose carefully the moments you expose that information
c) be transparent about how that data is obtained


More recent posts


Jan 21 | A reflection on Stanford’s AI-class, DB-class

Last quarter I had the opportunity to participate in one of Stanford's newest education initiatives, db-class and ai-class.

Jan 19 | In search of serendipity and soulful data

Every Wednesday I get the privilege to attend HCI lunches here at Stanford where a mishmash of scholars, professors, and students meet over lunch to watch someone's presentation on research (and proceed to tear it apart).

Jan 6 | The new urban landscape

I had the opportunity to visit the Design With The Other 90% exhibit at the United Nations lobby today and it was a stark reminder about how so much is happening around the world that I must not to lose touch with, and I think one of the more powerful insights I gained was how so many cities around the world are making successful transitions to more efficient people transportation in ways that have so much positive ripple effects.

Jan 4 | Help me test out Haptico!

Haptico is a little app that I made over Christmas break that lets your phone tell you if you're headed in the right direction without having to take out the phone.

Jan 2 | Predictions for 2012

It's taken me a week to get my thoughts together about what 2012 may bring us, but here's to a ritual that I've kept up for the better part of a decade.

Dec 30 | Goodbye GoDaddy

On January 13, 2005, I purchased chronicled.org from Globat.com, and it entered registration 3 days later.

Recently read

Updated 10 days ago.

» Are expensive batteries worth it? Maybe not

» 10 things to do with Hostess Twinkies

» He’s eaten at 362 pizza joints in New York City and has reviewed each one on an 8-slice scale.

» If you’re going to destroy your reputation as a PR person, better do it in an epic way

» Beijing pollution meter goes off the charts, literally

» Photoshop orgasm takes the shape of deblurring blurry images.

» The psychology and profile of the average Angry Birds addict

» Don’t worry, this column was written by a human.

» A ripoff Apple Store in Kunming looks just like the real deal

» Visualizing how the US would be if we colored it by the likelihood of calling within that area

» One book, many readings. Choose your own adventure books, visualized.

» The eternal shame of your first online handle

» A great piece of Bhutan’s Gross National Happiness from PBS.

» Photo fakeouts - comparing your dream holiday destinations to real pictures

» The beautiful things you can do with photographs and animated GIFs

Subscribe »

Search Categories