AI, black boxes, and designerly machines

On my holiday, I started reading into some topics I ought to know more about: artificial intelligence, genomics, healthcare, and the fast approaching intersection of the above. Here follow some half-baked reckons for your critical appraisal. Please tell me what’s worth digging into more. Also where I’m wrong and what I might be missing.

1. Opening the black box

large ribosomal subunit (50S) of Haloarcula marismortui, facing the 30S subunit. The ribosomal proteins are shown in blue, the rRNA in ochre, the active site (A 2486) in red. Data were taken from PDB: 3CC2​, redered with PyMOL.
By Yikrazuul CC BY-SA 3.0, from Wikimedia Commons

Reading Siddhartha Mukherjee’s ‘The Gene: An Intimate History’, I discovered the amazing trajectory of human understanding of DNA, RNA, enzymes, proteins, the genome, and the mechanisms by which they interact. There’s no doubt that this stuff will transform – is already transforming – our relationships with medicine. Crucially this generation of scientists are looking inside a black box, where their predecessors could observe its effects but not its inner workings.

At the same time, fuelled by petabytes of readily available data to digest, computer science risks going the other way in the framing of artificial intelligences: moving from explicable, simple systems to ones where it’s allowed to say, “this stuff is so complex that we don’t know how it works. You have to take it on trust.”

When we apply artificial intelligence (AI) to healthcare, transparency is essential; black boxes must be considered harmful.

It’s not just me saying this. Here are the words of the Institute of Electrical and Electronics Engineers (IEEE):

“Software engineers should employ black-box software services or components only with extraordinary caution and ethical care, as they tend to produce results that cannot be fully inspected, validated or justified by ordinary means, and thus increase the risk of undetected or unforeseen errors, biases and harms.” — Ethics of Autonomous & Intelligent Systems [PDF]

Transparency must be the order of the day. It comes in (at least) two flavours: the first is clear intent; the second, understandable operation. Both are under threat, and designers have a vital role to play in saving them.

2. The opacity of intent

It’s a commonplace to say that technology is not neutral. I won’t labour that point here because Sara Wachter-BoettcherEllen Broad and others do a good job of highlighting how bias becomes embedded, “AI-washed” into seemingly impartial algorithms. As the title of Ellen’s wonderful book has it, AI is ‘Made By Humans’.

That doesn’t seem to stop stock definitions from attempting to wall off AI beyond the purview human control:

“In computer science, AI research is defined as the study of ‘intelligent agents’: any device that perceives its environment and takes actions that maximise its chance of successfully achieving its goals.” — Wikipedia

But what goals exactly? And how did the AI get them? The Wikipedia definition is silent about how goals are set, because, in the words of Professor Margaret Boden“the computer couldn’t care less.”

“…computers don’t have goals of their own. The fact that a computer is following any goals at all can always be explained with reference to the goals of some human agent. (That’s why responsibility for the actions of AI systems lies with their users, manufacturers and/or retailers – not with the systems themselves.)” — Robot says: Whatever

When any technology moves from pure to applied science, intent must be centre stage. If we fixate too much on the computer science of AI, and not enough on the context of its application, intent will always be unintentionally obscured.

Many discussions about the “ethics” of AI or genomics are really, I think, discussions about the opacity of intent. If we don’t know who’s setting the goals for the machine, or how those goals are derived, how can we know if the intent is good or bad?

Moreover, true human intent may be difficult to encode. In a domain as complex as health and care, intent is rarely straightforward. It can be changing, conflicting and challenging to untangle:

  • a boy was triaged on first contact as in less urgent need, but has suddenly taken a turn for the worse
  • an elderly woman wants to get home from hospital, but her doctors need first to be sure she’ll be safe there
  • the parents want to help their children lose weight, but know that pester power always leads them back to the burger chain.

In these situations, even Moore’s Law is no match for empathy, and actual human care.

3. Designers to the rescue

Design, in Jared Spool’s wonderfully economical definition, is “the rendering of intent.” Intent without rendering gives us a strategy but cannot make it real. Rendering without intent may be fun – may even be fine art – but is, by definition, ineffective.

It’s time for designers to double down on intent, and – let’s be honest – this is not an area where design has always covered itself in glory.

We know what design without intent looks like, right? It’s an endless scroll of screenshots presented without context – the Dribbblisation of design.  If you think that was bad, just wait for the Dribbblisation of AI. Or the Dribbblisation of genomics. (“Check out my cool CRISPR hacks gallery, LOL!”)

Thoughtful designers on the other hand can bust their way out of any black box. Even if they’re only called in to work on a small part of a process, they make it their business to understand the situation holistically, from the user’s point of view, and that of the organisation.

Design comes in many specialisms, but experienced designers are confident moving up and down the stack – through graphic design, interaction design and service design problem spaces. Should we point an AI agent at optimising the colour of the “book now” buttons? Or address the capacity bottlenecks in our systems that make appointments hard to find?

One of my team recently talked me through a massive service map they had on their wall. We discussed the complexity in the back-end processes, the push and pull of factors that affected the system. Then, pointing at a particular step of the process: “That’s the point where we could use machine learning, to help clinicians be confident they’re making a good recommendation.” Only by framing the whole service, could they narrow in on a goal that had value to users and could be usefully delegated to AI.

4. How do you know? Show your thinking.

School exam paper. Question:

Crucially, designers are well placed to show the workings of their own (and others’) processes, in a way that proponents of black box AI never will.

This is my second flavour of transparency, clarity of operation.

How might we:

  • communicate probabilities and uncertainties to help someone decide what to do about their disposition to a form of cancer?
  •  show someone exactly how their personal data can be used in research to develop a new treatment?
  • involve people waiting for treatment in the co-design of a fair process for prioritisation?

In a world of risks and probabilities, not black and white answers, we should look for design patterns and affordances that support people’s understanding and help them take real, fully informed, control of the technologies on offer.

This is not an optional extra. It’s a vital part of the bond of trust on which our public service depends.

5. Designerly machines

Applying fifty iterations of DeepDream, the network having been trained to perceive dogs CC0 MartinThoma
Applying fifty iterations of DeepDream, the network having been trained to perceive dogs – CC0 MartinThoma

The cultural ascendancy of AI poses both a threat and an opportunity to human-centred design. It moves computers into territory where designers should already be strong: exploration and iteration.

I’m critically optimistic because many features of AI processes look uncannily like a repackaging of classic design technique. These are designerly machines.

Dabbers ready, eyes down…

  • Finding patterns in a mass of messy data? Check!
  • Learning from experiments over many iterations? Check!
  • Sifting competing options according to emerging heuristics? House!

Some diagrams explaining AI processes even resemble mangled re-imaginings of the divergent/convergent pattern in the Design Council’s famous double diamond.

Diagram showing how design moves from problem to solution in four stages, shown as one diamond after another. There are two pairs of divergence and convergence: Discover and Define, Develop and Deliver
© Design Council 2014 – https://www.designcouncil.org.uk/news-opinion/design-process-what-double-diamond

A diagram outlining a forward pass though three 3D generative systems, data is divergent and then convergent
“A diagram outlining a forward pass though our three 3D generative systems.” – Improved Adversarial Systems for 3D Object Generation and Reconstruction [PDF]
The threat is that black box AI methods are seen as a substitute for intentional design processes. I’ve heard it suggested that AI could be used to help people navigate a complex website. But if the site’s underlying information architecture is broken, then an intelligent agent will surely just learn the experience of being lost. (Repeat after me: “No AI until we’ve fixed the IA!”)

The opportunity is to pair the machines with designers in the service of better, faster, clearer, more human-centred exploration and iteration.

Increased chatter about AI will bring new more design-like metaphors of rendering that designers should embrace. We should talk more about our processes for discovering and framing problems, generating possible solutions and whittling them down with prototypes and iteration. As a profession, we have a great story to tell.

A resurgent interest in biology, evolution and inheritance might also open up space for conversations about how design solutions evolve in context. Genetic organism, intelligent software agent, or complex public service – we’re all entangled in sociotechnical systems now.

Advertisements

And Science — we have loved her well

And Science — we have loved her well, and followed her diligently, what will she do? I fear she is so much in the pay of the counting-house, the counting-house and the drill-sergeant, that she is too busy, and will for the present do nothing. Yet there are matters which I should have thought easy for her; say for example teaching Manchester how to consume its own smoke, or Leeds how to get rid of its superfluous black dye without turning it into the river, which would be as much worth her attention as the production of the heaviest of heavy black silks, or the biggest of useless guns.

William Morris, ‘Hopes and Fears for Art’, 1882

On the way to dConstruct: a social constructionist thought for the day

A desire to put some theoretical acro props under my vague unease with the determinist narrative of so much of our technology discourse has led me to the writing of the French anthropologist Bruno Latour. His work on the social construction of science, an ethnography of the R&D lab, has a special resonance for me, a humanities graduate who finds himself colleague to a legion of French engineers.

I’m stumbling intermittently through Catherine Porter’s translation of Latour’s 1991 work “We have never been modern“, as a prelude to David Edgerton’s “The Shock of the Old“. At times it feels a bit like eating up the broccoli before allowing myself desert, but the rich, buttery morsels like the following make it all worthwhile.

The story so far.

Latour argues that modernity, from Civil War England onwards, managed its contradictions by placing boundaries between nature and society. Thomas Hobbes, writer of the Leviathan, was taken up as a founder of political philosophy while Robert Boyle, he of the air pumps, was channelled as a natural philosopher and pioneer of scientific method. In truth both men speculated on both politics and science, but this inconsistency was whitewashed by their modern successors seeking only the pure narrative of one or the other.

And so we are today in a world still riven by CP Snow’s two cultures, where right-wing bloggers can grab acres of media coverage against climate scientists by finding just the tiniest trace of political “contamination” on the lab’s email servers.

But I wonder if the disconnection and reconnection of nature and society is also a useful way to understand some of the ideas I’m expecting to hear today at dConstruct, a conference at the cutting edge of technology and media convergence.

The 19 years since Latour published “Nous n’avons jamais été moderne” roughly spans my working life so far. I’ve witnessed the amazing things that can happen when you expose the humanities-soaked world of newspapers, books and TV to the attentions of software engineers and computer scientists. The results have been delightful and depressing, often both at the same time. Who knew back then that floaty copywriters would have to cohabit – for better or for worse – with the number-crunchers of search engine optimisation?

This fusing of the worlds of media and technology is only just beginning, and the next step is evident in the hand-held touch-sensitive, context-aware marvel of creation that is the latest smartphone.

Hitherto we have seen the the world of human-created information, the texts of the ancients and the tussles of our own times, through the pure window of the newspaper, the book, the TV, the PC screen. But the smartphone is a game-changer, like Robert Boyle’s air pump. With its bundle of sensors, of location, of proximity, and in the future no doubt heat, light, pressure and humidity it becomes a mini-lab through which we measure our world as we interact with it.

All manner of things could be possible once these facts of nature start to mix with the artifacts of society. My Foursquare checkins form a pattern of places created by me, joined with those of my friends to co-create something bigger and more valuable. My view of reality through the camera of the phone can be augmented with information. We will all be the scientists, as well as the political commentators, of our own lives. This is the role of naturalism in my “Mobile Gothic” meander.

To recycle Latour on Robert Boyle’s account of his air pump experiments:

“Here in Boyle’s text we witness the intervention of a new actor recognised by the new [modern] Constitution: inert bodies, incapable of will and bias but capable of showing, signing, writing and scribbling on laboratory instuments before trustworthy witnesses. These nonhumans, lacking souls but endowed with meaning, are even more reliable than ordinary mortals, to whom will is attributed but who lack the capacity to indicate phenomena in a reliable way. According to the Constitution, in case of doubt, humans are better off appealing to nonhumans. Endowed with their new semiotic powers, the latter contribute to a new form of text, the experimental science article, a hybrid of the age-old style of biblical exegesis – which has previously been applied only to the Scriptures and classical texts – and the new instrument that produces new inscriptions. From this point on, witnesses will pursue their discussions in its enclosed space, discussions about the meaningful behavious or nonhumans. The old hermeneutics will persist, but it will add to its parchments the shaky signature of scientific instruments.”

I don’t yet know where I stand in this picture. Am I the experimenter, his audience, or the chick in the jar?

An Experiment on a Bird in an Air Pump by Joseph Wright of Derby, 1768
A desire to put some theoretical acroprops under my vague unease with the determinist narrative of so much of our technologydiscourse has led me to the work of the French anthropologist Bruno Latour. His work on the social construction of science, anethnography of the R&D lab, has a special resonance for me, a humanities graduate who finds himself colleague to a legion of 

French engineers.

I’m stumbling intermittently through Catherine Porter’s translation of Latour’s 1991 work “We have never been modern”, as a

prelude to David Edgerton’s “The Shock of the Old”. At times it feels a bit like eating up the broccoli before allowing myself

desert, but the rich, buttery morsels like the following make it all worthwhile.

The story so far.

Latour argues that modernity, from Civil War England onwards, managed its contradictions by placing boundaries between

naure and society. Thomas Hobbes, writer of the Leviathan, was taken up as a founder of political philosophy while Robert

Boyle, he of the chicks in air pumps, was channelled as a natural philosopher and pioneer of scientific method. In truth both

men speculated on both politics and science, but this inconsintency was whitewashed by their modern successors seeking only

the pure narrative of one or the other.

And so we are today in a world still riven by CP Snow’s two cultures, where right-wing bloggers can grab acres of media

coverage against climate scientists by finding just the tiniest trace of political “contamination” on the lab’s email servers.

But I wonder if the disconnection and reconnection of nature and society is also a useful way to understand some of the ideas

I’m expecting to hear today at dConstruct, a conference at the cutting edge of technology and media convergence.

The 19 years since Latour published “Nous n’avons jamais été moderne” roughly spans a working life in which I’ve witnessed

the amazing things that can happen when you expose the humanities-soaked world of newspapers, books and TV to the

attentions of software engineers and computer scientists. The results have been delightful and depressing, often both at the

same time. Who knew back then that floaty copywriters would have to cohabit – for better or for worse – with the

number-crunchers of search engine optimisation?

This fusing of the worlds of technology and media is only just beginning, and the next step is evident in the hand-held

touch-sensitive, context-aware marvel of creation that is the latest smartphone.

Hitherto we have seen the the world of human-created information, the texts of the ancients and the tussles of our own times,

through the pure window of the newspaper, the book, the TV, the PC screen. But the smartphone is a game-changer, like

Robert Boyle’s air pump. With its bundle of sensors, of location, of proximity, and in the future no doubt heat, light, pressure

and humidity it becomes a mini-lab through which we measure our world as we interact with it.

All manner of things could be possible once these facts of nature start to mix with the artifacts of society. My Foursquare

checkins form a pattern of places created by me, joined with those of my friends to co-create something bigger and more

valuable. My view of reality through the camera of the phone can be augmented with information. We will all be the scientists,

as well as the poticial commentators, of our own lives. This is the role of naturalism in my “Mobile Gothic” meander.

To recycle Latour on Robert Boyle’s account of his air pump experiments:
“Here in Boyle text we witness the intervention of a new actor recognised by the new [modern] Constitution: inert bodies,

incapable of will and bias but capable of showing, signing, writing and scribbling on laboratory instuments before trustworthy

witnesses. These nonhumans, lacking souls but endowed with meaning, are even more reliable than ordinary mortals, to whom

will is attrributed but who lack the capacity to indicate phenomena in a reliable way. According to the Constitution, in case of

doubt, humans are better off appealing to nonhumans. Endowed with their new semiotic powers, the latter contribute to a new

form of text, the experimental science article, a hybrid of the age-old style of biblical exegesis – which has previously been

applied only to the Scriptures and classical texts – and the new instrument that produces new inscriptions. From this point on,

witnesses will pursue their discussions in its enclosed space, discussions about the meaningful behavious or nonhumans. The

old hermeneutics will persist, but it will add to its parchments the shaky signature of scientific instruments.”

I don’t yet know where I stand in this picture. Am I the man in the white coat or the chick in the belljar?

Reflections on Reading of Mr Joseph Priestley and M Antoine Lavoisier While Travelling by Air Plane Between Leeds and Paris

Steven Johnson’s The Invention of Air sparks a delightful reverie on the pivotal role of 18th Century scientist, non-conformist minister and poltical thinker Joseph Priestley.

Living in Leeds, I was vaguely aware of Priestley from local museums and the blue plaque at Mill Hill Unitarian Church on City Square. What schoolchild could fail to be impressed by the tale of Priestley inventing fizzy pop after studying the bubbles in a brewers’ vat on Meadow Lane? He open-sourced the method, leaving one Johann Schweppe to make a fortune.

But until I picked up Johnson’s book I hadn’t grasped that Priestley’s years in our Northern English city included experiments that shaped scientists’ understanding of gases, plant and animal life, and ultimately our planetary ecosystem.

Johnson tells how, after various gruesome experiments resulting in the suffocation of spiders and mice by placing them in sealed containers, Priestley wondered how long it would take a sprig of mint to succumb to the same fate. (Mint grows like a weed in gardens round us!) To his surprise, the mint lived, thrived even. What’s more, a flame could be lit in the sealed container, something that had not been possible in the containers where animals had expired.

Priestley wrote of his discovery to his friend Benjamin Franklin who almost at once made the further leap that, “I hope this will give some check to the rage of destroying trees…”

Serendipitously, I read this section of the Invention of Air on one of my increasingly regular flights from Leeds to Paris. Across southern England and the Channel, I was engrossed in Steven Johnson’s account of how Priestley made his experimental breakthrough, yet got the explanation wrong. He believed that the animals and flames emitted a noxious substance known as “phlogiston” and identified the gas “mended” by the plants as “dephlogisticated air”.

Then, literally as my plane broke through the clouds on the descent to Charles de Gaulle Airport, the action switched to Paris where the English hacker Joseph Priestley shared his discoveries with French aristocrat Antoine Lavoisier. It was Lavoisier who, after absorbing the implications of Priestley’s discovery, proposed a theoretical framework, correctly identified that a gas was used up in burning and respiration, and named that gas oxygen.

The English hacker, the French theorist, the combination of the two in innovation. The thought made my day, so apologies to the various colleagues upon whom I inflicted this convoluted story.

Sadly neither country was eternally grateful: years later Priestley was forced to flee to the United States after a Church and King mob burned down his Birmingham home and laboratory, while Lavoisier was beheaded in the French Revolution.

I can’t recommend this book enough. If there’s one criticism it’s that Johnson sometimes seems a little too pleased with himself to have hit upon a “long view” narrative linking Priestley with Northern England’s Industrial Revolution preeminance and atmospheric oxygen levels in the Carboniferous Era. But I guess I would be too, if I’d thought of that. It’s engaging, readable, and packed with thought-provoking ideas.

A final thought provoked: many people read while travelling, yet “airport” has become a perjorative term in relation to books. Can someone create a service that helps match reading to travel and create more srendipitous moments like mine? I’m looking at you, Dopplr bookcampers.