Inside the Reinforcement Learning Agent

by Prathmesh Salunkhe

In the world of Reinforcement Learning (RL), we often start with a simple picture: an agent interacts with an environment. The agent takes an action, the environment responds with a new observation and a reward, and this loop repeats. The agent’s goal is to learn a strategy that maximizes its total reward over time.

But what’s actually happening inside that agent? How does it go from receiving raw pixels or data to making intelligent choices? This document will take you inside the agent’s “mind” to explore the three key internal components it can use to learn and make decisions. We will demystify the core concepts of the Policy, the Value Function, and the Model, showing you the fundamental building blocks of an artificial intelligence that learns from experience.

2. The Agent’s Blueprint for Action: The Policy

The most fundamental component of an RL agent is its policy. You can think of a policy as the agent’s “rulebook” or “strategy guide” — it defines the agent’s behavior at any given moment.

Simply put, a policy is a mapping from an agent’s state (its current situation or observation) to the actions it should take. Policies generally come in two flavors:

Policy Type:

Deterministic Policy

This is a strict rulebook. For any given state, the policy outputs the exact same action every single time. It’s a direct command: “When you see this, do that.”

Stochastic Policy

This is a more flexible strategy guide. Instead of picking one action, it outputs probabilities for every possible action. For example: “When you see this, there’s a 70% chance you should go left, a 20% chance you should go right, and a 10% chance you should go forward.” This is a more common and general approach as it allows for exploration and more nuanced behavior.

To make this concrete, imagine an agent in a maze whose goal is to reach an exit. The optimal policy could be visualized as a grid with an arrow in every square. Each arrow points in the best direction to move from that square to reach the goal as quickly as possible. That set of arrows is the policy — a complete guide for what to do, no matter where the agent is. But to develop such a guide, the agent first needs a way to judge whether being in one square is better than another. This requires predicting future rewards, a job for the value function.

3. Predicting the Future: The Value Function

The value function is the agent’s “crystal ball.” It’s a predictive tool that evaluates how good a situation is. Crucially, a value function is always tied to a specific policy. It answers the question: “Given a specific policy (or strategy), how much total reward can I expect to get if I start from this state and follow that policy?”

The value function is a prediction of the expected cumulative reward, or “expected return.” It’s not just about the immediate reward from the next action; it considers the entire stream of future rewards. Its primary purpose is to help the agent make better decisions by evaluating the long-term consequences of being in a particular state. It’s what allows an agent to understand that it’s sometimes better to sacrifice immediate reward to gain more long-term reward.

Returning to our maze example, the value function could be visualized as a number in every square. A square far from the goal might have a value of -20 (representing the -1 penalty for each of the 20 steps it will take to get to the end), while a square right next to the goal would have a value of -1. These numbers quantify how “good” it is to be in any given state under a certain policy, giving the agent a clear basis for choosing between different paths. A policy and its corresponding value function are the core of many RL agents, allowing them to choose actions and evaluate their strategy. However, some agents go a step further, attempting to understand the underlying mechanics of the environment itself. This brings us to the agent’s internal model.

4. Building an Internal World: The Model

The model is an optional but powerful component that represents the agent’s understanding of how the environment works. You can think of it as the agent’s “internal simulator” or its own personal “map of the world.” A model’s job is to predict what the environment will do next.

Specifically, the model addresses two key questions:

• Predicting the next state: Given my current state and the action I take, what will the world look like next?

• Predicting the next reward: How much reward will I get for taking this action in this state?

The key benefit of having a model is that it allows the agent to engage in planning. Planning is the ability to “think ahead” by simulating the consequences of different action sequences internally, without ever having to take a step in the real world. The agent can use its internal simulator to explore possibilities and come up with a good plan before acting.

However, a model learned from experience is not always a perfect representation of reality. For instance, an agent in a maze might build an internal model of the layout. If it has never explored a certain corner, its model might be inaccurate and show a wall where there is actually an open path. Planning with this inaccurate model could lead to a sub-optimal strategy. We’ve now seen the three primary components an agent can possess: a policy for acting, a value function for evaluating, and a model for predicting. The way these components are assembled defines the fundamental type of RL agent we are building.

5. Assembling the Agent: A Taxonomy of Learners

Reinforcement learning agents can be categorized based on which of the three core components — Policy, Value Function, and Model — they explicitly use to make decisions.

Value-Based

It uses its value function to estimate the long-term reward of taking different actions from the current state. It then chooses the action with the highest predicted value.

Policy-Based

It directly uses its policy, which is a learned mapping from the current state to an action (or probabilities of actions), without needing to consult a separate value function.

Actor-Critic

Policy (the “Actor”) and Value Function (the “Critic”)

The Actor (policy) proposes an action, and the Critic (value function) evaluates that action, providing feedback that helps the Actor improve its decisions over time.

Finally, agents are also divided by one other major distinction: whether or not they use a model of the environment.

• Model-Free Agents: These agents do not have an internal model of how the world works. They learn the best actions to take through direct trial and error in the environment. Value-Based, Policy-Based, and Actor-Critic agents can all be model-free.

• Model-Based Agents: These agents first build a model of the environment and then use that model to plan their actions.

6. Conclusion: The Building Blocks of Intelligence

Understanding what goes on inside an RL agent reveals the building blocks it uses to learn and achieve its goals. By combining these three core components in different ways, we can create a wide variety of intelligent systems.

Let’s recap their roles one last time:

• The Policy: The agent’s strategy or rulebook for acting.

• The Value Function: The agent’s crystal ball for predicting future rewards under a specific policy.

• The Model: The agent’s internal simulator of the world.

Grasping how these three pieces fit together is the first major step toward understanding the fascinating science of how machines can learn to make decisions on their own.