When data makes us Product Cowards

I love data as much as the next robot.

Like everything — for data, there is a time and a place. Data is great in that it allows us to test our hypotheses. Less so for providing hypotheses to test.

More generously, it rarely gives us hypotheses to test — the data is infrequently so clean-cut as to point out where the big opportunities lie. It is almost certainly too messy to suggest improved details of optimisation and design and hand them to us on a plate.

We are storytelling hominids with a recently-acquired ability for working with numbers. SQL databases, and their more intelligible younger cousin, Google Analytics, are a long way from the savannah, from the farm, from the village.

Feedback might just be one of the truly great forces in the universe — but it needs a starting point. And lots of points of inspiration along the way. We are not yet equipped to rely on data for this starting point.

Despite our ambitions of quantitative rigour, data remains largely illegible — and where we can claim data literacy, it centres around evaluative questioning. In data education, generative questioning is ignored — assumed to be covered by qualitative exploration.

With improving tools and an education more grounded in statistics, we can increase this legibility. Making us better at reading the results of our tested hypothesis. And perhaps, in time, using the data to trawl for hypotheses worth testing.

Further caution

Even in evaluating hypotheses, data is not infallible and threatens to lead us astray. The Quantitative Fallacy (also know as the McNamara Fallacy) lurks behind our metrics, giving us false confidence in our results. Making a decision based solely on quantitative observations and ignoring all other inputs can be a recipe for disaster.

‘It is better to be roughly right than precisely wrong.’

— John Maynard Keynes

Pragmatically of course, it is better to be precisely wrong than roughly wrong — at least when your reputation is on the line. It is a brave man who bets their credibility on an imprecise guess.

And this is where I fear that data tempts us into cowardice. The increasing focus on data-driven decision-making (of which I am a true-believer) can quickly descend towards data-bureaucracy. Ideas not obviously suggested by the existing data are ignored or postponed. Hypotheses most readily measured take precedence.

Qualitative research and plain intuition still have a role to play in product decision-making. If we’re careful to explicitly recognise that this is what we’re doing. For the rest, there’s data.

Let’s make sure that we’re using data for the right reasons.

All this, from a recovering data scientist.

— — —

This article originally appeared on UX Collective.