Teaching a Mobile Network to Tune Itself, in Simulation
Before an agent can learn to run a network, someone has to build its playground.
A cellular network is full of knobs: how much power each base station transmits, when a phone should hand over from one cell to the next. Today most of them are set by hand-tuned rules. The idea behind deep reinforcement learning (DRL) for networks is to let an agent learn those settings by trial and error, rewarding it when users get better service.
You can't let an untrained agent experiment on a real network, so you train it in a simulator. The 5g_opt_drl repository from our lab at RBCCPS, IISc is the scaffolding for doing that with ns-3, the standard open-source network simulator. Most of the work is on the ns3_dev branch, and this post describes what is there.
The loop every RL setup needs
Reinforcement learning is a loop. The agent observes the state of the world, takes an action, and gets a reward. Repeat thousands of times, and a good algorithm learns which actions lead to high reward.
The project connects ns-3 to Python using ns3-gym, which makes a running simulation look like an OpenAI Gym environment. The simulation side has to answer five questions: what can be observed, what actions exist, what the current observation is, what the reward is, and how to apply an action.
The experiment: cars, cells and transmit power
In network/lte_rl, a car drives through a street grid simulated by SUMO, the urban traffic simulator, coupled to ns-3 over its TraCI interface. Two base stations sit at fixed positions, and the car's phone receives a UDP stream while it moves and hands over between them (ns-3's built-in A2-A4 RSRQ handover rule).
- Observation: the received power from the two cells, the serving cell's ID, and the lane the car is on.
- Action: one of four transmit power levels (30, 40, 50 or 60 dBm).
- Reward: downlink throughput over the last step (every 2 s), in Mbit/s.
The Python side adds a nice idea. As cars drive, it counts which lane follows which and builds a transition table of driving behaviour. From that it guesses the car's next lane and replaces the lane ID with the predicted future distance to the serving base station. The agent then sees where the user is likely going, not just where it is. A folder named HMM extends this towards path prediction with a hidden Markov model.
Plugging in real learning algorithms
The ray-de folder is the learning engine. It wraps any ns3-gym environment so that Ray RLlib, a widely used RL library, can train on it, with the algorithm (PPO, SAC and others), network size and number of parallel workers set in one YAML file. There are Docker files for running a Ray cluster too.
Its worked example is ns3-gym's own interference-pattern demo, copied as-is: a radio has to learn which channel a jammer will hit next and avoid it. The config trains PPO with a small 256 by 256 network for 2,000 iterations. A handover-optimization folder runs ns-3's two built-in handover algorithms and plots signal strength and serving cell, a baseline to beat.
Where it honestly stands
This is early-stage research infrastructure, not a finished result:
- It's LTE, not 5G NR. The simulations use ns-3's LTE module (ns-3.33); the 5G part is the goal in the name.
- The traffic experiment doesn't learn yet.
rl_agent.pypicks a random power level every step (action_space.sample()); no trained policy is connected to it. An episode also never ends on its own, and the chosen power is applied to both base stations at once. - The RLlib pipeline is only shown on the copied demo, and it needs a manual patch to Ray 1.12's internals, which the README itself calls a temporary hack. The README says the demo reaches a cumulative reward of about 98, but no training logs are in the repo.
- There are no results, plots or comparisons of a learned policy against the built-in handover rules.
What the repo does give you is the hard plumbing: traffic simulation feeding a network simulation, the network exposed as an RL environment, and a scalable trainer ready to plug in. The next step is clear: point the RLlib trainer at the car-and-cells environment and measure it against the built-in rules. The full source is on GitHub.