We spend enormous amounts of time debating whether machines are becoming more intelligent.

Yet one of the most important questions gets asked surprisingly rarely.

What exactly are we asking the system to optimize?

Intelligence is only the engine. The objective function is the steering wheel.

If you’ve never met the term before, here’s the short version: every intelligent system needs a way to distinguish better decisions from worse ones. The objective function is simply the mathematical definition of “better.” Everything the system learns is in service of improving that score.

Many of the most consequential behaviors we observe in machine learning trace back to this one design choice.

---

Every Intelligent System Is Just an Answer to One Question

Before you build any learning system, you have to define its objective function, you have to answer something deceptively simple: what does success look like, expressed as a number the machine can chase?

A few examples:

• A navigation app — minimize travel time
• Netflix — maximize watch time
• Google Search — return relevant information quickly
• Autonomous driving — reach the destination safely
• A conversational model — produce useful, truthful, safe responses

Different objectives. Different behaviors. Much of a system’s behavior, its quirks, blind spots, and surprising capabilities, is downstream of that design choice, often made long before anyone fully understands where it might lead.

Which brings us to a sentence that, once you sit with it, changes how you see almost every intelligent system you interact with:

A model doesn’t understand human values. It optimizes a mathematical function.

Much of what follows can be understood from that single observation.

The Recommendation Engine Thought Experiment

Suppose you’re building a recommendation engine. Something like TikTok, or YouTube, or Instagram Reels.

What should it optimize?

It sounds like a straightforward engineering question. It isn’t. Watch what happens to the emergent behavior of the platform depending on which number you choose to chase:

Objective	Likely consequence
Maximize watch time	More emotionally engaging content
Maximize clicks	More clickbait
Maximize comments	More polarization
Maximize shares	More surprising or outrageous content
Maximize satisfaction	Requires subjective feedback loops
Maximize learning	Lower engagement for many users
Maximize viewpoint diversity	Less personalization
Maximize well-being	Difficult to operationalize

Here’s the part worth sitting with: none of these are bugs. Nobody needed to explicitly engineer outrage. Outrage was simply what fell out of the math once watch time became the target.

This is the core lesson buried in every one of these systems, and it rarely makes it into the public conversation: behavior is not designed, it’s derived from the objective function. You don’t choose the outcome. You choose what the objective function rewards, and the outcome chooses itself.

Why Not Just Optimize for Everything?

Looking at that table, the obvious question is: why choose just one objective? Why not optimize for watch time and well-being and diversity, all at once?

If the solution were simply to optimize for more than one thing, we would have solved this problem years ago.

In reality, objectives compete. More personalization often reduces viewpoint diversity. Greater safety can reduce usefulness. Higher engagement may come at the expense of well-being. You can’t maximize all of them simultaneously, because gains in one often come directly at the cost of another.

Machine learning researchers call this multi-objective optimization. Engineers call it a trade-off. Society calls it a difficult conversation. Whatever you call it, it’s the reason “just balance everything” is not an engineering solution — it’s a restatement of the problem.

Goodhart’s Law, or Why the Measure Stops Measuring

There’s an old piece of economic wisdom that applies here with unsettling precision:

“When a measure becomes a target, it ceases to be a good measure.”

Originally, watch time was a proxy, a rough stand-in for whether people were enjoying themselves. A reasonable one, even. If people are watching, they’re probably engaged.

Proxy metrics are attractive for a simple reason: they’re measurable. Human values often aren’t. You can log watch time to the millisecond. You cannot log enjoyment, or meaning, or whether someone left the platform better off. So, the objective function gets built out of what can be measured, not necessarily what matters.

But once watch time becomes the objective, once creators optimize for it, and the platform optimizes for it, and the entire system bends around maximizing it, something quietly breaks. Watch time stops representing enjoyment. It just represents itself.

This pattern isn’t unique to recommendation engines. It’s one of the most reliable laws governing any objective function in production: whatever you choose to measure will eventually be gamed — by the system, by the humans in the loop, or by both.

Optimization Is Local. We Are Global.

Here’s the observation I find most striking, in a slightly haunting way.

Tell a system to optimize watch time, and you don’t get “watch time” as an isolated outcome. You get doomscrolling. Addiction. Echo chambers. Nobody explicitly programmed any of those outcomes. They emerged, the natural consequence of a narrow objective pursued relentlessly at scale, inside a system with no sense of the wider human life it’s embedded in.

The objective function sees one axis. We live on all of them at once.

That mismatch, between the narrowness of the objective function and the breadth of a human life, is quietly the story behind most of the unintended harms you’ve ever read about in the news.

This isn’t hypothetical. Credit scoring, hiring algorithms, and social media ranking systems have all demonstrated that optimizing one measurable objective can produce unintended consequences elsewhere, consequences no one explicitly designed for, and few noticed until the damage was already visible.

The Same Question, Everywhere You Look

Once you notice this pattern, you can’t unsee it. It isn’t a social media problem. It’s the defining design question behind nearly every learning system being built right now, because nearly every one of them is hiding an objective that was never fully debated.

Hiring algorithms — optimize for fast hiring? Fair hiring? Accurate hiring? Diverse hiring?

Clinical decision support — optimize for lower cost? Longer life? Quality of life? Patient satisfaction?

Adaptive learning platforms — optimize for exam scores? Understanding? Curiosity? Retention?

In every one of these domains, there is no single correct objective. There are only trade-offs, dressed up as technical decisions, made by someone, usually without asking the room whether they agree.

The Objective Function, Written Down

Every optimization problem, whether in machine learning, economics, or operations research, can be written in an astonishingly compact form:

Choose action a* = arg max_a U(a)

The equation is deceptively simple. Almost every modern intelligent system, from recommendation engines to reinforcement learning agents, is, in one form or another, solving this problem millions of times a day.

In plain language: every recommendation, every ranking, every decision a learning system makes is ultimately an attempt to maximize some objective function, U. The hard part was never finding the maximum; optimization algorithms have become remarkably good at maximizing whatever objective we specify. The hard part is deciding what belongs inside U in the first place. The danger isn’t that these systems optimize badly. It’s that they optimize exactly what we asked.

That single equation quietly ties together recommendation systems, reinforcement learning, search, robotics, and economics. Different fields, different names for U, reward, loss, utility, cost, but structurally, the same objective function, wearing different clothes each time.

Back to the Original Question

We spend a great deal of time asking whether machines are becoming more intelligent.

Perhaps history will judge that intelligence was never the hardest problem.

The harder problem was deciding what intelligence should optimize for.