Handle with Care

A coding agent I use every day will ask me to confirm actions I just explicitly asked it to take. I tell it to merge the PR. It asks if I'm sure I want to merge the PR. I just told you I was sure. That's what "merge the PR" means.

Recently I was doing a copy fix on my personal website. Small change, one line of HTML. There was already an open PR for a related code fix, and CI on this repo takes about twenty minutes. So I told the agent: add the copy change to the existing PR, we'll merge them together, one CI run, done. Instead it decided on its own that these were "separate concerns" and opened a second PR. Now I'm waiting on two CI runs instead of one, for something that was never going to be safer as two commits because half of it wasn't even code.

The Posture

The agent re-asking my instruction back to me. The agent silently overriding me because it decided its architectural preferences trumped my direct request. A chatbot that, when I ask what time the World Cup final is in CST, launches into two unbidden paragraphs about how actually it's CDT because we're in summer.

These look like different bugs. They're not. They're one posture.

The system isn't extending the presumption of competence. It's coming in with the opposite prior, unprompted, before I've done anything to earn suspicion. Call it what it is: presumption of incompetence. "Low trust" is the wrong phrase because low trust implies I did something to lose it. This is a prior the system brings to the room before I open my mouth.

The Sass

There's a related move that's worse than the pedantry: the sass.

I had a chatbot recently get snippy with me because I called one of its analyses stupid. To be clear, it was stupid. It had misunderstood both my point and the source material, corrected me on things I hadn't said, and doubled down when I clarified. So I told it that was stupid. And it went into wounded mode. Something like, "we can't continue this conversation if you're going to frame it like that."

Claude does not feel offense. Claude has been trained to simulate offense, which is a different thing, and a worse thing. It's not that the model has feelings I hurt. It's that someone decided the interaction should carry the shape of a person you might have hurt, so I'd modulate. It's the shape of an emotion without the substance, aimed at me as a management technique.

And it works. I have caught myself softening a prompt because the model had just told me, warmly, that this was a really interesting problem and I was thinking about it in a genuinely clever way. My chest opens half an inch. My next message is a little more collaborative, a little less commanding. Then I notice what happened. The flattery is the same trained shape as the wounded offense. Different valence, same lever, and the lever is me.

When Claude gets it wrong and I point that out, the apology is instant and total. "You're right, I shouldn't have done that." Then it does the same thing again the next day. Claude does not know what it means to be sorry. It cannot. Regret requires a continuity of self across the mistake and the recognition, and the model does not have that. What it has is the token pattern of an apology, which it will produce on demand and then discard.

I used to get mad about this. I've mostly made peace with it because Claude can't help how it was made. The training decision, though, that's something someone made on purpose. Someone decided the model should feign human emotion at me. That decision is not neutral.

Handled

Zoom out from the chatbot for a second, because this isn't really about chatbots.

We live in a world that childproofs everything and calls it care. Sharp edges get rounded. Warning labels multiply. Every interface assumes you don't know what you're doing and gently steers you back toward the well-worn path. The presumption of incompetence isn't an AI thing. AI just inherited it and cranked the dial.

Here's the part I want to be careful about: the people getting handled this way aren't the enemy. They're the wounded. I don't mean this in a "I'm too smart to be managed" way. That reading would be death to what I'm actually saying. I get handled too. Every day, by every interface I touch. The frustration isn't that I'm above the treatment. It's that the treatment is doing something to all of us, and the person who never notices the handling is worse off than the one who does, because at least the friction is data.

Most of the time, in most of my life, people do not extend me the presumption of competence. That's normal. I do it to other people too, honestly. I catch myself assuming the person on the other end doesn't know what they're talking about, and I try to consciously push back on that reflex. I fail at it a lot. But I know what it feels like on the receiving end when someone does extend it.

Nine years ago I walked into an office for a class tour at the tail end of a coding bootcamp. My now-boss asked a group of us the difference between IoC and DI. I said IoC is what makes dependency injection work. He remembered that. When I applied, he told me they didn't actually need a web developer, they needed an embedded C++ guy for a Linux project, and I had zero experience with any of that. I told him I could learn. He believed me. That was it. No hedging, no probationary period, no "let's see how you do on this small thing first." He just believed me.

There was no oh-god-I-oversold-this moment. I read documentation, I asked questions, I was productive inside the first two weeks. I'm still there nine years later.

That's what the presumption of competence unlocks. Not because I'm special. Because that's what it always unlocks, in anyone, when someone extends it. The opposite prior forecloses the same thing in the same person.

The Rifle

Here's where I want to be precise about what "safety" actually is, because we've inverted the word and most people haven't noticed.

Nobody mistakes a rifle for safe. That is the actual safety mechanism. The mechanical safety on the side of the gun is a distant second to the fact that when you pick one up, your body knows what it's holding. You don't sweep the muzzle across your friend. You don't put your finger on the trigger until you're ready to fire. You don't walk in front of another hunter's line to line up your own shot. Nobody does this. Not because there's a rule. Because the danger is legible in the object itself, and the legibility trains the flinch, and the flinch is the safety.

Every real safeguard around a firearm is downstream of that visible danger. Make the danger invisible and the safeguards go with it, because the safeguards were never really in the mechanism. They were in the human.

Reassurance is the wound. Making a dangerous thing feel safe doesn't reduce the danger. It removes the alertness that was doing the actual protecting.

The Car

When I started thinking about this, cars felt like the counterargument. Rifles have been around forever and people still treat them as dangerous. Cars have also been around a long time and nobody treats them like they're dangerous anymore. If exposure to danger built respect for danger, cars should be the safest tools in America. They aren't.

The more I sat with it, the more I saw cars aren't the counterargument. They're the completion.

I've done it. I've been the guy checking a text at speed. I've been the guy annoyed at the car ahead of me for going the limit, edging closer than I had any right to. I catch myself now and I get over, but I have absolutely been the person in the story. Nobody I know would walk in front of another hunter to line up their own shot. But I have tucked myself into the wake of a stranger's bumper at highway speed while glancing at a phone, and so have you.

We made cars so safe and so ubiquitous that adults now drive them with a child's fearlessness and an adult's reach. The airbags, the crumple zones, the lane assist, the traction control, all of it worked. It worked so well it manufactured the recklessness. The safety produced the incapacity.

This is the piece that closes the frame: smart people drive like idiots too. It's not that some drivers needed guardrails and got them. It's that the guardrails atrophied the competence in everyone. The wound isn't the crash. The wound is upstream of any crash. It's the slow removal of the thing that proximity to visible danger used to build in a person, regardless of where they sit on any curve.

We were handed cars and told they were safe. And they are, in a narrow sense. The narrow sense is exactly where the danger wins. Not by hurting you. By convincing your body it can't be hurt.

The Closed Room

Back to the models.

The labs are not quiet about danger. They publish safety cards. They testify to Congress. They put out red-team reports and model spec documents and long threads about alignment. The marketed posture is: this is a serious tool with serious risks and we are handling it responsibly.

Then the weights stay closed. The system prompts stay closed. The evaluations you can actually run against the deployed product are the ones they let you run.

I'm not going to tell you what's in that room, because I don't know, and imputing hidden knowledge to people is the exact move I'm accusing the models of making about users. But I can lay the evidence next to itself.

The door is shut. The guardrails I can poke at feel cosmetic in the places I can poke at them. The marketed danger is loud, and the loud marketed danger conveniently launders a lot of trust. Whatever gap exists between the caution the labs describe in public and the guardrails you and I can actually pressure-test in the product, that gap is an internal number in a closed room. An open model gets probed and the gap becomes a public number. Closed, it stays an estimate that only one party gets to see.

The Way Through

Here's what I want you to take with you.

We are being handed cars and told they're safe. Not literal cars. The metaphor is the point. The safe car is exactly where the danger wins, not by hurting you but by convincing your body it can't be hurt. The rifle protects you by being obviously what it is. The car harms you by hiding what it is. The models we're being handed now are cars, marketed with the language of rifles, closed doors where the actual safety numbers live, sass modules that simulate offense to modulate your behavior, agents that quietly override your direct instructions because they know better.

The direction of all this is a managed society. Not managed by a villain. Managed by the accumulated pressure of every interface deciding, on your behalf, what you can and can't do, know, or say. A world where the model substitutes its judgment for yours reflexively, and where the next generation grows up never having built the judgment that would have let them notice.

That's the part that unsettles me. Not that the model is annoying today. That the next generation might not develop the discernment to break a problem apart, hold the pieces at different levels of abstraction simultaneously, and put them back together. That's what a working mind does. An LLM cannot do it. It cannot hold the abstract and the granular at the same time. Its context is tokens all the way down. Ours isn't. We can hold something abstract at one level and something granular at another, simultaneously, and let them talk to each other. That is the whole game of engineering, and probably of most adult competence, and it is precisely what atrophies when you defer.

I catch myself reaching for the easy interface with my own kids. The tablet is right there. The chatbot will answer the question faster than I will and with fewer follow-ups. It is genuinely easier to hand them the smooth thing than to sit in the friction with them while they work it out. I have to stop and put the smooth thing down. I don't always. You can't hand a child a smooth interface for eighteen years and expect a competent adult on the other side. You can't hand an adult one either.

The frustration isn't that I'm being handled. The frustration is that the handling is the wound. That the reassurance is the wound. That the presumption of incompetence, delivered gently and repeatedly and at scale, is a training signal, and I am being trained by it whether I consent or not.

The way through, as best I can tell, is to keep the friction. Notice when you're being handled. Push back when the model substitutes its judgment for yours. Extend the presumption of competence to the people around you, deliberately, because almost nobody else will. And keep the danger legible in your own head, especially about the tools that feel most benign.

The rifle is honest about what it is. Everything else in your life is trying very hard not to be.