A Q-learning Algorithm-based Frequency Resource Allocation Scheme for User Links in High Throughput Satellite Communication

High-throughput satellites (HTS) with frequency flexibilities have attracted increasing attention due to their high-capacity and high-speed communication, which enables a lower cost per bps for end users. In this chapter, we model a beam resource allocation problem as a beam-series finite Markov decision process and propose a frequency allocation scheme for beams in HTS communication systems using a Q-learning algorithm. Under the constraint of the limited bandwidth allocated to the beam, the frequency resource allocations to user equipment (UE) are conducted using a dynamic programming algorithm, which is known to provide an efficient solver for 0-1 knapsack problems in order to obtain an optimal solution according to the policies of a network operation center. Here, the state of the beam is determined from the frequency allocation in a previous beam, the action of the beam is defined as an allocated bandwidth, and the rewards to update the Q-table are calculated from the link allocations. The proposed scheme was evaluated through computer simulation and compared with a full search scheme and an immediate reward maximization scheme. The proposed allocation scheme outperforms the full search scheme in terms of computational complexity. In addition, simulation results show that the proposed scheme outperforms the immediate reward maximization scheme. The system throughput of the proposed allocation scheme reached 96.1% of the full search, and that of the immediate reward maximization scheme reached only 80.8% when a policy that maximizes the system throughput was adopted. In addition, with the proposed allocation scheme, the average number of the allocated UE was 96.1% of the full search, and that of the immediate reward maximization scheme reached only 76.7% when a policy that maximizes fairness among the UE was adopted.