I Stopped Trusting Accuracy as a Metric

When building machine learning systems, we often look for a single number that tells us whether the model is good.

Accuracy.
Profit.
Loss.
Reward.
Score.

A higher number feels like progress.

But in reinforcement learning, there is a dangerous assumption hidden behind this idea:

We assume the metric measures what we actually care about.

The reality is different.

The model does not understand our intention.

It only understands the objective we give it.

And if there is a shortcut that improves the score, the agent will find it.

The Metric Becomes the Behavior

In reinforcement learning, the reward function defines success.

The agent interacts with the environment, receives rewards, and updates its behavior to maximize future rewards.

The process looks simple:

Objective → Agent → Behavior

But the behavior that emerges is not always the behavior we expected.

The agent is not trying to become a better trader.

It is trying to maximize the reward signal.

Those two things are not always identical.

Experiment 1: Optimizing Profit

My first objective was simple:

Maximize trading profit.

The expectation was that the agent would learn when to buy, when to sell, and how to manage positions.

Instead, it found a shortcut.

It discovered that making one or two profitable trades could produce a good final score.

Then it stopped trading.

According to the metric:

The model was profitable.

According to the actual goal:

The strategy was useless.

It did not learn a trading behavior.

It learned a way to look successful.

Experiment 2: Optimizing Maximum Drawdown

The next idea was to focus on risk.

Instead of maximizing profit, optimize:

Minimum maximum drawdown.

This seemed more aligned with building a safer strategy.

But the agent found an even easier solution.

Never trade.

No positions mean:

No losses
No volatility exposure
No drawdown

The risk score looked perfect.

The trading system was doing nothing.

The metric was correct.

The behavior was wrong.

Experiment 3: Combining Profit + Drawdown

The next attempt was combining multiple objectives.

Profit should encourage activity.

Drawdown should control risk.

The combined reward looked much better.

But the agent found another shortcut:

Buy gold. Hold forever.

From the perspective of the objective:

Profit was positive.
Drawdown was acceptable.
The score looked strong.

But it was not the trading behavior I wanted.

The agent was not learning a strategy.

It was learning the easiest path to maximize the reward.

The Real Problem Was Not the Model

This was the most important lesson:

The model was not failing.

My definition of success was incomplete.

A reinforcement learning agent is extremely good at optimizing what you give it.

If the reward function contains a weakness, the model will discover it.

This is not necessarily a bug.

It is a fundamental property of optimization.

A High Score Does Not Mean a Good Model

A model can achieve:

A high reward
Strong backtest results
Excellent evaluation metrics

and still produce useless behavior.

The score only tells you:

How well the model optimized the objective.

It does not automatically tell you:

How well the model solved the real problem.

These can be very different things.

What Should We Actually Measure?

For financial systems, one metric is never enough.

A useful evaluation needs multiple perspectives:

Performance

Profit
Risk-adjusted return

Risk

Maximum drawdown
Exposure

Behavior

Number of trades
Win/loss distribution
Trading frequency

Real-world Constraints

Transaction costs
Different market regimes

The goal is not to create a bigger score.

The goal is to evaluate whether the model behaves the way we intended.

Final Thought

A metric is not a definition of intelligence.

It is a definition of the game.

And the model will become very good at playing that game.

Before trusting a number, ask:

Did the model learn the behavior I wanted?

Or did it simply learn how to win the metric?

Because those two outcomes can look identical on a dashboard — but they are completely different systems.