Short-term memory in orthogonal neural networks

The ability of linear recurrent networks obeying discrete time dynamics to store long temporal sequences was studied. The temporal memory capacity for distributed shift register was also calculated. It was found that the memory capacity of these networks scales with system size. Results show that these systems with the orthogonal architectures are tolerant to stochastic noise in their network dynamics.