SXSWi: Metric Driven design

Session: Saturday 12th March
Speaker: Joshua Porter
Hashtag: #metricsdrivendesign
Get the slides at: http://bokardo.com/talks/metrics-driven-design/


I hadn’t initially intended on going to this session, but found myself nearby at 5pm and I ducked in. The name of the speaker looked familiar (Joshua Porter), and I soon realised it was because I read his blog (bokardo.com), and have read his book “Designing for the Social Web”.


The talk today focused on how to make design decisions based on metrics.  Going into the session, I had some initial pre-conceived notions about what this talk would be about, and I was pleasantly surprised to find the session very engaging and thought provoking.  Here are some notes on the session:


Why Doug Bowman left google and 41 shades of blue

Porter began the discussion with why Doug Bowman left Google – he simply was unhappy about the way the company was being run – “when a company is filled with engineers, it turns to engineering to solve its problems – turning everything into a simple logic problem, removing all subjectivity, and just looking at the data”. He felt that it “paralyses the company from making any daring design decisions“.


This was an interesting way to start the discussion, as I’m sure many of the audience members were designers who may have been in the same predicament. (Luckily, I’m not one of them ;) ).


As an aside, Porter takes us through an example of extreme optimisation undertaken by Google, where they tested “41 shades of blue” to find the optimal link colour – and there was a difference in clickthroughs, based on the slightly different shades of blue.


Intuition design vs. Data driven design

Porter takes us through the spectrum of design, with  ”Intuition design” on one extreme, and “Data driven design” on the other extreme.  Doug Bowman’s view point is on the “Intuition design” end of the spectrum, where the designer works on their “best guess”, relies on previous experiences, benchmarks similar designs, uses best practices, principles and practices.  It’s “instinctive, subjective, daring”.


On the other extreme, data driven design is where we rely on data for decision making – every design choice is tested, and design is treated as a logic problem. It’s “deliberate, objective, safe”. Porter mentions that data driven design typically iterates by small things at a time, and as a result is a very slow process.


The mountains

Porter uses two mountains to illustrate an example to great effect.  Imagine two mountains, one with a lower peak, and another with a higher peak.

Source: Joshua Porter http://www.bokardo.com

If we take the first shorter mountain as our current design model, then engineering design optimisation will only ever take you to the peak of that first mountain – there’s a limit to how far a design can be optimised within the current model. You can’t get any further without changing some of the fundamentals of the design.


Porter mentions that you actually want to be on the other, higher mountain – which represents the better design.  But you cannot get there through optimisation alone.


Porter talks through two phases of design – typically design begins with the process of ideation – innovation thinking (how can this thing can be better?).   And then you do optimization for awhile until you hit the maximum level that can be reached. The second phase is an assessment to redesign, and make the big leap to the other mountain.  According to Porter:

Optimization asks what works best with the current model?
Design innovation asks what is the best possible model?



On metrics

Metrics are numbers that measure the effectiveness of your business or your clients business.

1. Metrics reduce arguments based on opinion. They reduce, don’t take it away.
2. Metrics give you answers about what really works. (or lead you down a rabbit hole)
3. Metrics show you where you are strong as a designer (and where you are weak as a designer)
4. Metrics allow you to test anything you want (empowers you to try things, go all out and test it. Allows you to push the boundaries)
5. Your metrics will be as unique as your business


The usage lifecycle
Porter mentions the need to consider the usage lifecycle of the product.  Users move between:


Interested –> Trial/beta user –> Customer –> Passionate customer

Between these stages are hurdles:
Acquisition –> Conversion –> Engagement –> Satisfaction


Acquisition vs Referral –  Case study – Dropbox
Porter mentions Dropbox as an example – with this company, they initially used Adwords to drive traffic – which needed four steps to become a customer (search on a keyword, visit the site, sign up for service, become a customer). It cost them $233-$388 per person, for a $99 product.  The metrics told them it simply was not working.


They then moved to referral program – a two sided incentive. It was so successful that it increased signup by 60% permanently.


Conversion case study: Twitter

The original sign up flow for Twitter would invite you to follow some random people (typically celebrities) – but their conversion wasn’t that good. What Twitter did was to gather some metrics about what people were doing on Twitter – they looked at the most passionate users and reverse engineered their experience.
The people who were really engaged were the people that were really engaged into specific topics.


What’s interesting is that Twitter actually added a screen to the flow, which improved their sign up conversion.  This second page contained a LH vertical menu of categories down the side, that allowed people to select what they were interested in, and then listed people that were popular to follow in those specific areas.


Facebook’s deactivation page

Facebook redesigned their deactivation page adding a “These people are going to miss you” message, with pictures of your friends, and this accounted for 1 million members not leaving the service.


On Satisfaction metrics

It’s easy to think of metrics as hard data. But you can turn qualitative data into quantitative data.



Finding the sweet spot

Porter uses some examples on how services find that magic number or critical factor and mentions that when looking at a service or software, you need to look at these things to improve.


Friendfeed: The magic number for friend feed was 5 – once a user found 5 friends, they became active users.
Once they found this out, they designed a friend finder that helps users find contacts, right in the stream.
This led to a sudden influx of friend feed friending – the results of which was visible on twitter.


Blogger: At blogger, their most critical number was the number of posts. The more posts, the more engaged the user was, and the more potential for others to consume that content and be exposed to the service.


In summary

  • Optimize in small steps, innovate with daring leaps
  • No design survives contact with the user
  • Small improvements taken together yield amazing results
  • Testing is empowering, reversion is cleansing
  • Metrics are not creative human beings are
  • All team members are responsible for the user experience


Seth godin on Netflix
The three biggest assets of the company could not be tested.
Real victory comes when you have the guts to develop the untestable.



Overall I thought this a well balanced discussion on how metric driven design is merely one facet – one tool we as designers can use to help inform design decisions. And that we need to balance guts, intuition and daring, with data driven design decisions.

  • http://twitter.com/nilanp Nilan

    Great post on metric driven design – I missed this session. Looks like it was good

  • Pingback: sxsWINNING* | nerd alert!

  • Natacha

    Just found your post and thank you for this clear and enlightning resumé ! The question that it leaves me with is : how does the designer fare in a metrics driven company that has no business/design intuition ?
    Does this mean that you have to take micro design leaps on an everyday basis and hope this fits into a grander intuitive view ? I’d love to hear what’s your take on this