Reinforcement Learning Technique for Finding the Feedback Capacity

One of the classic problems in information theory is solving the feedback capacity of noisy channels with memory. The capacity is expressed analytically by an optimization problem over a multi-letter objective. This is the main obstacle to directly solving the feedback capacity analytically. In the last decade, some channels with memory where solved by formulating the capacity objective as a Markov decision process, and then applying dynamic programming algorithms. However, those solutions were restricted by the channel cardinality and were computationally tractable only for channels with binary alphabet. In this paper, we propose a novel method to compute the feedback capacity of channels with memory using reinforcement learning (RL). The main advantage of this approach is its computational efficiency, even for channels with with large cardinality. The outcome of the RL algorithm sheds light on the properties of the optimal solution, which in our case, is the optimal input distribution of the channel. These insights can be converted into analytic, single-letter capacity objectives by solving corresponding lower and upper bounds. We demonstrate the efficiency of this method by analytically solving the feedback capacity of the well-known Ising channel with cardinality smaller than 9. The proposed method is used to extract the structure of the optimal input distribution, which is followed by an analytic solution for the feedback capacity and a capacity achieving coding scheme. However, we can show that the coding scheme derived for small cardinality is no longer optimal for cardinality larger or equal to 9. Insights on the solution are supplied by a new upper-bound for large cardinality. Also, we present an optimal coding scheme for asymptotic alphabet size. The proposed methodology is a step in the course of developing strong numerical tools for channels with large cardinality. Furthermore, the insights obtained by the analysis of large cardinality sheds light on the behaviour of the optimal solution as the cardinality increases.