The Person Behind the Data

Nilay Patel wrote a piece on The Verge about what he calls “software brain.” It's the way of seeing the world that fits everything into databases, loops, and structured language. It's good. It names a pattern I've been guilty of myself many times. He draws the limit of software brain at the edge of human experience. Most of a person lives past that edge. He argues this is a big part of why regular people are growing to dislike AI even as the tech industry falls further in love with it.

I agree with him. But I also believe the limit lives far deeper than he takes it.

What the stereotypical software brain is trying to reach, in the AI age, is you. Your preferences. Your emotional patterns. The way you respond to things. The person under the data.

And the person under the data has never been measurable. By software or by anyone.

It’s all about people who use the tools

I've been a software developer for nearly two decades. I run Smudge, a software business. I have software brain by trade.

What I've also done, through all of that work, is stay close to the people we're building for. Just last weekend, I was out in the field with a customer, living a slice of the problems alongside them. I strongly believe that this is the work. The software only matters because there's a person on the other end of it.

There are technical people in our industry who see it as: if we just connect your data, your files, your calendar, your messages, your habits, the AI will understand you and act on your behalf. These very same people are already living that way themselves, racing to make themselves as legible as possible to their AI tools.

From my perspective, driven from the fundamental POV that software is a tool to enable people to thrive, it’s becoming increasingly blurry/subjective as to what tech tools are doing good and are enabling us as a community to thrive, and what tools are doing us harm.

A dataset of predictable, standardised emotional responses

In her book “How Emotions Are Made,” psychologist Lisa Feldman Barrett shows that emotions are constructed in the moment. The brain builds them out of the body's current state, your history, the context you're in, the concepts your culture has given you, and what your human system is predicting is about to happen.

Interestingly the same situation produces different feelings on different days, and two people can describe the same event in contradictory ways. There is no standardisation or norm to determine "this is how I always feel about this."

There isn't, for you or for anyone.

Which means the premise that AI could build a stable model of your inner life, or your emotional responses from your data, is a fundamental category error. I've observed skilled practitioners, therapists with years of relationship skills, full attention, a quiet room and no other demands, still struggle to fully access another person's inner world. Even when both people are trying. Even when both people are good at it. Contact with another person's felt reality is hard. Harder than it looks from outside. It requires presence, repair, patience, and the kind of relational friction that takes years to learn.

A model trained on your emails, your search behaviour and some other accessible data indicating online behaviour will not find its way to your inner emotions. The thing that AI tools are really trying to reach isn't actually stored in your data.

The problem is what happens when the tools try anyway.

Data is a poor proxy for the real self

When the industry asks you to make yourself legible to software, to connect your calendar, your messages, your files, your habits, the trails of you, you begin rendering your inner life in the shape the tool expects. Or, only in the way a machine can interpret that data set.

You describe yourself in ways the system can process. You trust the system's reflection of you over your own felt sense of what's happening. The loop closes around a smaller, incomplete version of you.

And then there's the possibility that people will feel more confused and anxious.

The measurement itself is changing the thing being measured, and making a lot of people feel smaller in the process. I started writing about this idea because after nearly two decades of building software for people, developing business partnerships, marriage and family life I kept running into the same wall. I came to the realisation that the part that mattered most was the part the software tools couldn't touch - our inner self.

Patel's software brain can't see it by definition. The inner world of a person has no edges a database can identify, let alone find.

AI will struggle to know your true self

I use AI. I build with it. It's already changed how we work at Smudge in ways that are hard to comprehend and what this means long term.

But the view that AI will know you, and act for you, and understand what you actually need requires a premise that has never been true. Not even for the people who've dedicated their careers to understanding other people. When I watch the tech industry race to flatten every part of life into data a model can consume, I'm watching a category error being executed at scale.

You can't measure the soul. No industry has managed it yet. Just look at the number of modalities in the psychology field, with still more being created each year. Asking a database to do it won't change that.

And the people working hardest to make it work are going to be the ones most likely to lose touch with what was there to begin with.