Articles Blog

The Consequences of Your Code

The Consequences of Your Code


A few years ago, I got a message from an ex. And it said,
“roses are red, violets are blue, “I’ve got chlamydia, and you might have too”. Which, first of all, is genius, but it meant I was taking a trip down
to the clinic the next morning to get tested. I have a point to this story, and I swear
it’s about computers. If you ever want to see a complete cross-section
of London life, do join the queue for the STI testing clinic
first thing in the morning. Actually, get yourself tested anyway, it’s
the right thing to do. Anyway, I go to the clinic, I pee
in a sample tube, seal it up, hand it over, and they give me a little business card with
a phone number and a passcode on it. And I can call that number the next day, and if my tests are all clear,
then the system will tell me. And if they’re not, it’ll pass me over
to someone at the clinic, because you do not deliver bad news
with an automated system. So the next day, I call the number,
type in the code, and the system says: “Thanks. Here are your test results.” And then there is a pause.
A proper, ten second long pause. And as I stood there on my mobile
in the middle of the street, it felt a bit like that artificial pause that
they add in reality shows before announcing the winner. It felt like there was a spotlight on me, all dramatic tension music playing in the
background. And then it says “We’re putting you through to
one of our team. “Please stand by.” That’s bad news, right? Bad news comes from a human.
Why are they passing me through? Someone picks up,
I give my passcode again and they say, no, it’s all clear, you don’t need to worry,
everything’s negative, you’re fine. And I ask why the system put me through,
and they say, “oh, no idea, it just does that sometimes”. It just does that sometimes. Did the database lookup fail in the background, and it dealt with the timeout by
just passing me to a human? Was there a small note on my file somewhere
that confused it? I’ve no idea, they had no idea, because whoever wrote that code,
whoever designed that system, the moment there was any error, they just told it to go to
“we’re putting you through” without explaining why and without thinking
what that might cause. That phone system is used by people asking
about things way more serious than chlamydia, the sort of things you can’t cure
with a course of antibiotics. And sure, on the scale of complaints
it is minor, but mistakes like that are a symptom of something
much bigger. I’ve talked about bodging things in the past. I encourage it for hobby projects. Just slapping something together as a proof of
concept, just so it works for you is a great principle
when you’re making a thing for yourself. But if you’re making something for the public,
for mass consumption, particularly something that’s going to be
used by people in very vulnerable moments, then you’ve got to take a lot more care. Every time we build something for the public,
we have to start making the trade-off: how much time and money is it worth to deal
with every edge case? Maybe the designer thought that lookups would
only fail one in a million times, and if that’s the case, then yeah, it’s probably not reasonable to bother recording a whole extra message and programming in a whole extra edge case. But if the lookup fails often enough that
the clinic receptionist just dismisses it as normal… well, by that point it’s too late to make a change,
isn’t it? The system’s in place, it’d be far too expensive to fix it now.
It’ll do. The trouble is that we’re often dealing with
unknown harms and unintended consequences. Far too often a bodged-together system that
was just meant to be a test gets rolled out into production, and
everyone just has to deal with the bugs because that’s all anyone can
afford to do. I will always bet on incompetence
rather than malice, I will always bet on someone not thinking
about consequences rather than thinking of them
and going “who cares”. We see this with big tech companies. Facebook allows the world to communicate in
unprecedented ways, it does huge amounts of
emotional labour for us, it allows people to keep in touch with
old friends that they’d just fall away from otherwise. But it’s also enabled stalkers and abusers
to reach people they shouldn’t, it’s allowed a huge amount of private data
to be misused without real consent, and it’s arguably helped cause terrifying
election meddling. Now, I don’t believe anyone in Facebook’s
management was rubbing their hands in delight
at the chaos it was causing: it was just a series of seemingly-reasonable
decisions that added up to huge consequences. And then there’s YouTube: it allows anyone
to publish their experiences to the world, it provides income for creative people that
bypasses traditional media, and it’s helped people connect
with other people’s lives. It’s also helped to radicalise folks, to promote conspiracy theories,
and to traumatise children. Are those tradeoffs worth it? It depends on your moral framework,
and it depends — let’s be honest — on how it’s affected you personally. It’s not like there’s a crystal ball
that’ll tell you, yes, your brand-new dating app you’re developing, that will cause 1,000 couples
to marry and live happily ever after, but it will also get three people murdered. The real world is not a trolley problem. The STI result system that I called
presumably reduced the workload on staff, and it allowed people to check their results
out-of-hours when it was convenient and discreet
for them. You’d hope something like that wouldn’t have
a downside, but then the designers screwed it up because
they thought it was good enough, and it wasn’t. One extra check,
one extra voice message that said “Sorry, I can’t find your result,
just a moment” would have solved that. Every time we design a system, we have to
minimise the potential harm. Look at the code that you write,
look at the systems you design, and think: How could this be abused?
How could this fall apart? What are the failure states here? If you were a bad actor,
if you wanted to use this maliciously, how would you go about it? Think about how you’d attack your own systems,
explore those failure states, deliberately screw things up
and see how your code copes. Because if you don’t, someone else will. Thank you very much to the
Centre for Computing History at Cambridge who’ve let me film with all their old equipment, and to all my proofreading team
who checked through my script.

100 thoughts on “The Consequences of Your Code”

  1. thats not a bug, thats a feature, so that you as the user still have hope youre clean even after youve been put through 😀

  2. Part of the enjoyment of creating a 'system' for me is to see WHAT holes there are, but I can't say I do extensive testing on the matter. I have to think that I will do even more testing and prodding if any of my work goes public, though I typically know every single shortcoming of what I create as soon as I create it, and I incessantly search for a way to either patch over or ward off those places.

  3. There are many causes for these multitude of hiccups. One of them is money (and time, which translates back to money), or lack of it. On many occasions, I have been told I have a fixed amount of time to complete something even though it would require double or triple the time to implement and test properly.

  4. In "From the Earth to the Moon", in the Apollo 1 fire disaster episode, the "failure of imagination" was brought up. Why did the fire happen…because of the failure of imagination. They didn't think it could happen. This is the same thing for countless systems. Programmers (and testers) fail to imagine how their system might fail and what the consequences would be of those failures.

  5. Very good video. I'm in the field of information security and it's very, very clear from my perspective that the vast majority of security problems are caused by the exact same "good enough" attitude and wouldn't exist if someone had spend a bit more care and – that's the problem – resources on thinking about unintentional consequences. We've now spent the past 20 years trying to cope with this, with ever better technologie, with better management approaches, with complex risk assessments – but rarely do we do anything about the root cause. It's a case of the insane running the asylum.

  6. Its not the programmers code in 90% of the cases…
    what about the marketeers and the business partners that rub their hands from the profits that made from skiping as much as possible to make a bare minimum that can sell to unfortunate clients…
    what about the cases that you as a software engineer say that you need the resources to provide a descent product and they refuse because they want to keep the budget low in order to get more profit?
    How about the cases that they have already sold a product and come to you and tell you "You have a couple of weeks to make this"
    when in reality it needs 6 months with a team of 5 to make something close to what they have already sold?

  7. Your proof reading team might have been cracking up and wetting themselves when they read, "Roses are red, violets are blue, I've got chlamydia and you might have too."

  8. that is literally impossible.
    you can not possibly precieve all the possible ways your code could be miss used.

    and that actually goes for a lot more things then just code.
    like inventing new cleaning products and making new electronic devices or devices in general

  9. I mean… I get the point. But honestly this kind of neurotic thinking trying to prevent every possible case of misuse, abuse, or error leads to my stuff never being deployed, because it's never good enough, and even when it's at 99.7% perfection, I still don't feel confident to release it.
    There's gonna be some collateral damage as a result of technological progress. I mean who the heck could have anticipated that there will be creepy videos on YouTube, targeting and traumatizing children? If they had waited to identify this issue in advance, we still wouldn't have YouTube.

  10. Seems that the STI system is already giving the bad results through automation.

    If you know you only talk to a human when you have bad news then any attempt to connect you to a human is an indication that you have bad news. Maybe it was designed to eliminate that assumption by randomly assigning patients with good results to receive them via human.

  11. to make things so that criminals could not abuse it, you need criminal mind yourself… to understand how their mind works. 🙂 and that's why criminal-proof is lot easier than fool-proof. it is possible to work out criminal mind, but it is impossible to work out fool's mind.

  12. You seem to be assuming there has to be a problem with the code, but the problem might just as well be in how it's used. Maybe it's generic software that was not designed for this job in specific; maybe it was originally agreed that the recording itself was supposed to include more explanation; maybe someone just told the system to use the same recording for the error case too (or maybe the 10 second pause was where the error message should've been); maybe no-one was supposed to tell you that the system will only give you an automated message if there's no bad news. Endless possibilities!

  13. or you could thank god that there is a human in the loop and pray to god that thus shall be for evermore – exempli gratia nuclear missile launch protocol, a mite more consequential than your do or don't chlamydia concerns.

  14. Someone who doesn't think that way just can't say to themselves "humm if I was wanting to have malicious intend what would I do"

    You actually have to pay someone that preferably USE too think like that , too get the malicious ideas

  15. All of you guys surprised about a 'computer guy' having sex have clearly never heard Tom Scott confidenty (and a touch smugly) give tactical advice on removing bras with one hand and it honestly shows.

  16. As a software engineer, I disagree with the premise of this video. It's not a design flaw or an error if some users are making wild suppositions about the system's behavior. There's no way to account for such edge cases.

  17. at 3:05 one small fallacy is that the representatives on the other side only get the calls that the system send to them. So they're going to get a higher percentage of those cases than it otherwise would be. Great videos though, I'm a newcomer to your channel. Loving your content.

  18. I work for a company making games for kids and every time we go down I just feel like we've made a system with the power to make 10k kids unhappy in an instant. Instability is no joke

  19. Neglect and laziness are not innocent behaviors of incompetence. Sure, the individual writing the code may be ignorant of the externalities and edge cases, sure the management may be ignorant of the value of thorough testing, sure the CEO may be ignorant of the daily proceedings of managers. So who is responsible? The distributed people who are negatively impacted do not have the readiness to consolidate their frustrations into an actionable monetary threshold (e.g. lawsuits, boycotting, lobbying) to effect change, so is it the government, who need to institute regulations and consumer protections that inhibit the agility in the market? But then the government is purportedly elected by the people, for the people. So again, who is responsible?

  20. With today's insanely long line of codes, very limited time of deadline and demanding employer, how a programer manage to produce a foolproof program is beyond me.

  21. The problem is, even if they had a "timeout" or "error" message, would people believe it? Or would people think "oh crap, they probably say that any time you're positive so you don't freak out"

  22. I'm a software developer myself and the idea of such an imposition sys automatically passing the call to a human if anything whatsoever goes wrong seems like it might be a good idea.

  23. A program that either gives you good news or,.. send you to a human operator essentially also gives you bad news. The exact thing it is programmed to avoid.
    Now a program with bugs is therefore an improvement.

    The question is how many people get a false negative?

  24. Surely the whole thing should be done by a human anyway, its such a sensitive topic, I think even being told you don't have chlamidya by a robocaller would be well, weird and cold.

  25. Tommy Nob Rot! This is all about telling us you got laid isn't it? Congrats… But are you sure? I thought you just jerked it to calculus and operating systems books.

  26. good coders are expensive , and comanies want to pay as little as possible.
    most projects are not assigned appropriate deadlines.
    shitty code is not the fault of coders, it's the fault of management.

  27. People coming to believe that the earth is flat is hardly the fault of programmers. What should they have done? Censor the videos and try to steer everyone in the direction of their own ideology?

  28. I don't think facebook delights in causing chaos. I think they don't care about the chaos as long as it makes them money and they don't care what potential harm it could cause. I find that more despicable than any intential harm just for shits and giggles.

  29. i would of done <insert error> (such as timeout or somthing simlar) we are putting you through to one of our team. Then you can tell them the error and they can fix it (if possible) and the user would be notified that it may not be somthing bad. or if anything there was an error we are putting you through to one of our team.

  30. I gathered 2 things from this video. #1 Tom is not a virgin and #2 Tom dose not have chlamydia. Was there anything else important in this video…something something code, something something be sure to only screw up if it will hurt someone got it nothing else important!

  31. That's all and well and many of us developers think the same way, but then you have management come and tell you something in the lines of "ship now, we'll fix later". And "later" never happens. Or you have people jumping on your back to have something out yesterday even though you estimated enough time for proper development, they did something to cause a delay and now you have to rush it.

    To be honest, I am surprised just how well the world works considering what we have to deal with and what kind of code gets into production without ever being touched again. Also, being a developer is one of the most ungrateful occupations there is. As long as everything is working as expected, no one knows you even exist. No one appreciates the work and time put into making something operate reliably. But when things break, oooh boy.

  32. Well, in my opinion it was coded correctly.

    You made a call assuming if you are positive you'll get redirected to med personnel which gives you the news in a calming and professional way which makes sense. If not then you're fine and you get the news you're all good via an automated message.

    Having the element of surprise by redirecting random 'negative lab results' and giving them the good news in person offers people who have positive test results an extra relief until someone answers.

    Imagine there are 1000 lab tests.

    Scenario A: 80% of total results came negative(800), they make a call 90%(720) of that number get redirected to an automated answering machine and 10%( or 80 people) are redirected to a "human" they get the good news which later is a huge relief and has no impact on them.

    Scenario B: In the other hand you have 20%(200), they make a call and all 20% are redirected to a "Human"

    In total you have 30% of calls redirected to humans, which gives all 30% a better experience with waiting / call holding. From which 30% majority will hear the good news anyways.

    So in conclusion you save lots of time by redirecting 70% of your calls to an automated machine and is also about hope, and the idea of still having a margin of error that saves the 10% of affected people until they are getting professional help on line.

    I dont know if I could explain it, but I'm sure you get my point Tom 🙂

  33. Hello TOM SCOTT. Your test results are available.

    Press 1 to find out if you have CHLAMYDIA.

    Press 2 to find out if you have GENITAL WARTS.

    Press 3 to find out if you have SYPHILIS.

    Press 4 to find out if you have GONORRHEA.

    Press 5 to find out if you have BLADDER CANCER.

    Press 6 to find out if you have HIV.

    Press 7 to find out if you have HERPES.

    Press 8 to find out if you have KIDNEY STONES

    Press 9 to find out if you are acutely ill and require emergency hospitalization.

    Or press 0 to speak to an operator and schedule an appointment to discuss your test results.

  34. So computers should only tell good news and people tell bad news? hmmm
    Wonder if this is gonna make people avoid other people.

  35. We develop programs in 3 ways you can choose 2 quick, cheap, good.
    Quick and cheap won’t be good .
    Good and cheap won’t be quick .
    Good and quick won’t be cheap.

  36. I feel the problem is more what managers see as a minimal viable product and the saying if it's not broken don't fix it.

  37. This does sound terrifying but I can definitely understand how people can mess up with this. As a programmer who only recently went "professional" (working on a actual commercial product) I find the prospect of trying to guess how the system can fail and its probability to be impossibly hard without actual testing in the wild.

  38. One would think that the STI department at the clinic would be used to 'catching' things… A simple try/catch would have solved this.

  39. I thought we wiped out the Clap, along with all the other diseases that gave us problems in the 40s. You still hear about herpes, but never about the Clap. Now we worry about AIDS.

  40. The fact is that automated response systems are 'dark patterns' by design. The surprise here is that it unnecessarily put callers through to a human being, whereas the default is to do exactly the opposite.

  41. Am I the only one who thinks that's an incredibly rude and insensitive way to inform someone they might have given you an STD?

  42. Writing a library system in a group at uni. I was pissing around with it, and realised you could DOS the whole system by reserving every single book in the library. We only limited how many books you could take out, not how many you could reserve.

  43. Never underestimate the power of incompetence. Multiple partners = multiple problems. Personally and professionally.

  44. I've been in the area for long enough to know that 90% of the software rushed to production is not because of incompetence nor malice, but corporate greed

  45. When you explained the scenario, I thought it was a feature and not a bug. If it only lets every positive case through, then you immediately know that you're positive when you get put through. But now you don't know if you're positive or not when being forwarded.

  46. So, many of the examples proffered here as unintended consequences aren't engineering problems at all. They're social problems which by some absurd miracle laypeople seem to expect engineers or software developers to solve. Well, news flash: it doesn't work that way. People aren't going to stop being people because we've added technology. Your dating app murderers could have been murderers using a newspaper ad or singles bar. The active ingredient isn't the tech, it's the people, and what we really need to be wary of isn't imperfect technology, because all technology is imperfect. Rather, we need to be vigilant against is lawmakers insisting on hamstringing technology because it's politically convenient to scapegoat instead of addressing the actual underlying problems. For example, the government proposing a requirement which forces technology companies to use broken cryptography, because it makes reading people's messages easier.

  47. Could also be that a doc or nurse miscoded your results somehow. Maybe they were supposed to put in a flag…yes or no…but they didn't so it was null. They just typed in the comments "Negative for clam." Or maybe you put in the wrong code. Again, as a coder, both of those instances should pass to a live person. Anything else gives hackers too much info.

Leave a Reply

Your email address will not be published. Required fields are marked *