A multi-armed bandit is a reinforcement learning problem where an agent repeatedly chooses among several actions, called arms, and tries to maximize cumulative reward under uncertainty.
Intuition
The core tradeoff is exploration versus exploitation:
- Exploration tries arms whose reward is still uncertain.
- Exploitation chooses the arm that currently looks best.
If the agent only explores, it wastes reward on bad options. If it only exploits too early, it may miss a better option.
How It Works
At each round:
- Choose an arm.
- Observe the reward.
- Update the estimated reward distribution for that arm.
- Use the updated estimates to choose the next arm.
Common strategies include epsilon-greedy, upper confidence bound, and Thompson sampling.
Relationship To AB Testing
Traditional AB Testing often assigns fixed traffic proportions until the experiment ends. A multi-armed bandit is an adaptive experiment: it can shift more traffic toward better-performing variants while still reserving some traffic for learning.
Failure Modes
- Early noisy results can push traffic toward the wrong arm.
- Non-stationary rewards require continued exploration because the best arm may change over time.
- Optimizing only short-term reward can conflict with long-term learning or fairness.
One-line Interview Answer
A multi-armed bandit balances exploration and exploitation to maximize cumulative reward while learning which action is best.