Teaching to the test: weak signals from the emissions scandal


Who’d have thought it?

Since the late 18th Century, moral panics have centred on the propensity for industrialisation and financialisation to turn people into machines.

‘You must either make a tool of the creature, or a man of him. You cannot make both. If you will have that precision out of them, and make their fingers measure degrees like cog-wheels, and their arms strike curves like compasses, you must unhumanise them.’ — John Ruskin

But now the Volkswagen emissions scandal lays bare that we stand on the cusp of the opposite peril.

We have trusted machines to perform repeat operations without fear or favour. We could count on them to do the same thing over and over and always render the same results.

Volkswagen’s cars didn’t have a fault in their diesel motors — they were designed to lie to regulators, and that matters, because regulation is based on the idea that people lie, but things tell the truth. — Cory Doctorow

Likewise, corporate brands have been defined by their superhuman consistency and unnatural longevity. Large teams of people with weighty manuals have been devoted to the maintenance of corporate “values”, “tone of voice” and “brand personality”.

That must be why it comes as such a shock to discover that both can exhibit the hitherto exclusively human characteristics of inconsistency, fallibility and mendacity. They think nothing of behaving one way on the emissions test treadmill and another on the open road.

At the corporate level, some of the debate about Volkswagen has focused on governance, on the relationship between innovation and regulation. Can those acting in the public interest ever enforce moral behaviour on private enterprise with systems of numerical targets and controlled-environment tests? No. Given big enough financial incentive and small enough risk of detection, corporations will game those systems. Every last one of them, all of the time.

The novel feature exposed by Volkswagen is that corporations are increasingly playing the game through the medium of software.

VW’s “defeat device” is not a physical device but a programme in the engine software that lets the car perceive if is being driven under test conditions – and only then pull out all the anti-pollution stops. — The Guardian

How will we respond when such potentially malign ingenuity is embedded in every vehicle and household object? An arms race of more targets, tests and regulations will be futile.

Opening the data will get us somewhat further. The wonder of connecting all the things to the internet is that we can see how they are performing day in, day out, in the wild, not just under test conditions. How long before a continuous stream of emissions and mechanical safety data replaces the annual MOT test?

But that will not be enough. The quantitative data will only ever capture an incomplete picture of the many externalities that such machines belch out.

Publishing the code will allow sufficiently skilled and committed auditors to understand behaviour at an atomic level. Surely we must legislate all black box software out of safety critical systems.

But can software really encode the most difficult trade-offs that people make in the moment? Trolley problems have been a stock in trade for philosophers since the 1960s. They won’t be settled by a C++ function.

My Honda's a consequentialist

The parallel social constructs of corporation and computer have long sheltered behind a convenient fiction: of being neutral technologies, forces of nature, a domain apart from the messy, contested world of the human politics. Volkswagen reveals this to be a false division.

Meanwhile we humans have been trained for two centuries in machine-age educational institutions for careers in command-and-control organisations. But we urgently need to make a break for it, back to the exposed flank of behaving like people at work. And we need to get there before our unaccountable creations form an unholy alliance to pull it off better than we do.

We’ll need new kinds of transparency and accountability, ones that recognise the responsibilities of directors, designers and developers as indivisible from those of the companies and code they create.

To the people who run big organisations: we expect you to be explicit about your business models and service blueprints. We need to know whether you see us as your customer, or as just another product in a multi-sided marketplace. We need to be sure that you treat the society in which you operate with respect, not contempt.

To the people exploring new opportunities and refining service through customer development or lean start-up methodologies: a reminder not merely to optimise for the easy metrics. Your success depends on your ability to design for the full diversity of people in this most complicated of worlds.

Think about that Spotify “how to build a Minimum Viable Product” graphic that did the rounds on Twitter a while back…

Basically it’s lean saying screw the externalities. How else could you end up with a road-hogging, obesity-making, fossil fuel guzzling car as superior to a bicycle?

And to the people who develop the software: you bear a special responsibility. With each line of code you commit, you pour a little of your soul inside the machine. You cannot do this anonymously; you can never wholly shuffle off this responsibility, even when someone else writes the specification or pays the bill.

Increasing storage capacity will soon make it feasible for even the most deeply embedded system to carry a copy of its entire version history. Maybe the Volkswagen engineers would have paused for thought if they’d known that their names would be openly entwined for eternity in the defeat device’s mitochondrial DNA.

Updated 3/10/2015 to add that brilliant Cory Doctorow quote on regulation

Not All Mammals! In defence of designing for “people”

I’ve been thinking about this exchange with Roberta…

@mattedgar Lots of people _talk_ about getting users in the room. This weekend @mHealthLeeds is actually doing it. #mhleeds

@RobertaWedge @mattedgar Users of what? In a health-care context, the term covers layers of euphemism.

@mattedgar @RobertaWedge fair point. Alternatives to the word ‘user’ gratefully received. (Often but not always “people” works just fine)

@RobertaWedge @mattedgar I am anti “people”. Citizen, student, resident, account-holder, patient, passenger, woman, employee – precision aids discourse.

@mattedgar @RobertaWedge indeed. Though may also reinforce rigid role definitions and allocations. People wear many hats, sometimes simultaneously.

Also this post by Russell…

I’m old enough that I’ve seen the same debates go round and round a few times.

One is the (always well-intentioned) cry – let’s stop saying customers/consumers/users, let’s remember they’re people! This always snags an emotional latch but I think it’s worth resisting.

Firstly, let’s remember that they’re also mammals – does that help? No. Moving up to the next biggest category isn’t especially useful.(*1)

Secondly, if you need reminding that your customers/consumers/users are people you have bigger problems. Changing what you write on your briefs/stories isn’t going to help.

I know where they’re coming from. I get the need for precision. I think we all agree that whatever you call them, we make stuff for and with messy, multi-faceted actors. But, as a people-centred service designer, I reckon the P word is worth defending.

While there is a place for “user” and other words of precision, it should not be at the expense of open-ended human-centred inquiry. A more interesting question might be, what are the right words for where we are now, in our digital culture as a whole? I for one think “people’s” time has come.

cat and mouse

For starters, the point about designing for humans as opposed to animals is not as facile as Russell makes out. At risk of being hauled before a gavel-toting, wig-wearing dolphin in the post-speciesist court of the future, Not All Mammals! My cats have evolved to simulate affection, but I’m certain that if I lay dead at the foot of the stairs for a couple of days they would eat me.

We are not cats. We can and should accord our human users a human level of tolerance and curiosity. Overuse of reductionist language is a tell-tale sign when we forget to do so. “Customer” or “claimant”, “passenger” or “potentate”, “servant” or “CEO” – they’re all different flavours of the same remouillage.

Moving up to the next biggest category – at least for a while – is what designers do. An iterative process zooms from the big picture to the tiny details and back again. The words we use as we zoom signal where we are in the focal range. Worrying about whether the next link is obvious, or the service accessible with a screen reader? “User” may be the best word to deploy. Helping someone unpick complex medical and social factors that impact their mental health? They probably need you to see the whole person. Over the course of any design process, it pays to mix it up, to vary the vocabulary.

Shoes - Some rights reserved paul-w-locke

We set ourselves too easy a task if all we do is satisfy the needs that present themselves at face value; often things that matter are hiding in plain sight. In my work I’ve found myself pointing out…

  • to a footwear brand that teenagers’ feet are still growing
  • to a retailer that shop floor workers turn to family members for help with the intranet
  • to a utility company that couples argue about who spends too long in the shower.

Banal insights like these make a direct difference to the service we offer. They can only be had by breaking free from blinkered caricatures of “runner”, “employee” or “resident”.

Every time we boldly launch our little boat by asking “what is the user need?” two further questions lurk implicitly upstream: which users, which needs? Ignore these and we will be forever tethered to our preconceptions about the nature of service we aim to deliver. Maybe some people call a contact centre to rapidly resolve a service problem. Maybe some call for reassurance that there are real people behind a digital service. Maybe others just call because they’re lonely. People-centricity reveals dimensions here that focusing only on the caller as user would miss.

We should also consider the number of actors. Service dominant logic dictates that service is always co-created by multiple parties – as a minimum, the one demanding it, and the one delivering it. Service design and innovation processes look at how those parties work together. Sometimes the best way to unlock greater value for end users is to set free those who serve them to do their best work. Want to improve the experience of online news? You’ll need to change the way news is gathered and edited as much as the way it is accessed and explored.

When we follow all the actors and understand their capabilities, we find that the boundary between “consumer” and “producer” is more malleable than the reductionists assume. Mobile, social media turns public transport “passengers” into providers of powerful real-time information service. US supermarket “employees” donate food so their colleagues in need can enjoy a Thanksgiving dinner. The complex and variable geometry of service only emerges when we accept the people involved for everything they are.

Texting in the park - some rights reserved - duncanh1

One of the delights of the new GOV.UK (in which – Disclosure! – I play a bit part, but for whom I certainly do not claim to speak here) is the way it demonstrates that many of the debates of Noughties web design are now pretty much solved problems. Form follows function. Text and hypertext dominate the flashy, non-standard carapaces on which millions were wasted by private and public sectors alike. High levels of accessibility, responsive design, open source frameworks, web-native APIs – they’re all just manifest good sense things that make user experiences better.

This good news is not yet evenly distributed. Many organisations would do well to take their lead from the Government Service Design Manual. Like learner drivers they might need to go through the consciously competent stage of focusing on their users. But when they’ve internalised that then what?

The settlement of those user-level questions should free them up to direct their attention to more positive visions of digital service, and to people’s higher-order, higher-value issues. They can focus on making explicit those questions that so often go begging: which users, which needs? They can create systems with continuous improvement built in. They can ensure there is empathy and the possibility of change every time service is delivered.

Users may well be the place to start. But people must surely be the end-game.

Erm, excuse me, but I think Everybody was here all along

It’s taken me a while (and 83 more pages of Here Comes Everybody) to understand my unease with the “technology changes everything” discourse around social media, and now to reach an alternative hypothesis. In my last post I questioned whether the advent of the internet in the place of television could, as Clay Shirky suggests, awaken some kind of latent creativity and collaboration. Could the web really turn the tables on the mass media, humble big corporations and bring about revolutions?

Here Comes Everybody contains a number of such vignettes to back up the case for the technology-led societal shift: the phenomenal accumulation of quality volunteer-contributed content in Wikipedia, British students’ Facebook revolt against changes to their HSBC bank charges, Belarus “flash mob” protests, and so on. Nothing like these things could happen, the story goes, without new tools built on top of mobile phones and the internet.

Except that they could, and did. Because for every story of 21st Century people getting together to achieve something amazing using new technology, there’s a story from history of people who did much the same without the benefit of the world wide web. One of these even gets into Shirky’s book: the 1989 fall of the Berlin Wall and all that it stood for. But to that we might add any number of 20th Century educational movements such as the Workers’ Education Association, student boycotts of Barclays and Nestle in the 1980s, the demonstrations of May 1968 (the same year, by the way, that a contract was awarded to build something called the Arpanet).

These big things, of course, are just the tip of the iceberg. To these we must add countless more localised acts of collaboration and creativity: the village antiques society of which my grandmother was treasurer, the baby-sitting circle where my mum and dad traded nights out with other parents using curtain-rings as currency, countless fanzines photocopied and posted. Sure, it was a little harder to shift ideas around the world, but from what I can recall we mostly managed OK. After all, making and sharing stuff are two of the most defining characteristics of being human.

So how come it still feels like the internet is changing everything? I have a suggestion.

When Clay Shirky talks in his blog post about a massive television-related bender spanning the whole second half of the 20th Century, he’s half right. But it wasn’t the mass of the population that was rendered senseless by the broadcast media – no they kept on creating and collaborating much as people always have. Rather, the intoxication induced by television was mainly in the minds of big business and mass media. Broadcasters and brands became so drunk with the power of pushing content one-way into people’s living rooms that they forgot that their “audience” might be busy doing other things.

It was a wise executive who admitted “I know half my advertising doesn’t work, I just don’t know which half” because the mythical housewife never was waiting patiently for the television to tell her which brand of soap powder to buy. She was too busy chatting to her next-door neighbour while they scrubbed their doorsteps, or making bunting to string along the street on carnival day. But business, the media and government didn’t get that. It was their tragedy that there was no return path. Information flowed in only one direction – away from them – leaving them to revel in their own self-importance.

It’s my contention that the amount of collaboration and creativity in the world is not changing greatly as a result of new communications technologies. There may be a little incremental creation, but mostly it’s substitutional of other activities that have gone on in some shape or other for thousands of years. What has changed is that new technologies make those old activities more visible. All those conversations used to happen in drafty village halls, through the post and over the phone. Now they are on the web for all to search and to see. It’s no longer possible for the mass media and big businesses, or even governments, to imagine that they have it all their own way, because the curtain has been drawn back to reveal just how irrelevant some of them have become.

It’s not so much a case of “Here Comes Everybody”, as of “Everybody Was Here All Along”. People aren’t late to this party, technology and business are. Only by understanding that can traditional organisations have a chance of being welcomed into the conversation. If they come at this change from a technology point of view – thinking that they’re going to instantly enable incremental communications for an amazed and grateful populace – then they’ll likely fail to make the grade. But if they understand that it’s mainly substitutional then they’ll see why their customers set the bar so high.

People have been communicating and interacting for thousands of years without the help of mobile phones and computers. They have developed sophisticated ways of doing so. Social niceties and nuances make their collaborations highly efficient. If you or your business want to be a part of that you’d better first watch and learn. See how natural are the conversations, and how easily people negotiate complex issues of coordination and collaboration. Then try to design tools and talk in a language that matches that quality. Or to put it another way, Here Comes Technology, Late As Usual (but if you sit quietly at the back for a bit Everybody might let you join in).

Update 2 October 2008: David Cushman interviewed Clay Shirky in London and is posting a series of videos at Faster Future, including an answer to my question. Worth a look.

Update 1 November 2008: Simon Collister is not alone. I still haven’t finished my copy either.

Update 11 October 2011: John Dodds on the (re)discovery of the second screen.