Redundancy maximization raises Hopfield network memory capacity to 1.59 versus 0.14 in simulations
Authors in Nature Communications trained each neuron to maximize redundant information, raising capacity from 0.14 to 1.59 in Hopfield network models.
A study in Nature Communications reports that a learning goal based on redundant information raises the memory capacity of Hopfield networks to 1.59. The classical network trained with the Hebbian rule reaches 0.14. nature.com
Nature Communicationsnature.comOpen source ↗ The authors describe this as a more than tenfold improvement and report that it also outperforms recent state-of-the-art implementations of Hopfield networks. The work is computational: model neurons are analysed and trained in simulated networks.
Associative memory is the retrieval of stored patterns from partial or noisy cues. A Hopfield network stores patterns as attractors of its dynamics, and its weights are trained with the patterns as a teaching signal. Started from a corrupted pattern, the network updates its neuron states and converges to the stored pattern. Hopfield networks were introduced in 1982 and originally trained with the Hebbian rule. Later learning rules improved capacity and stability.
Information analysis of single neurons
The authors used Partial Information Decomposition. It splits the contribution of inputs to an output into unique information from each input, redundant information shared across inputs, and synergistic information that arises only from combining them. Each neuron was compared with two inputs: its recurrent input and a target input that carries the ground-truth pattern but does not drive the neuron.
The analysis covered a classical Hebbian network of 500 neurons. Memory load is the ratio of the number of stored patterns to the number of neurons. Memory capacity is the largest load at which all patterns are still recalled, about 0.14 for the Hebbian rule. Below capacity, redundancy between the external pattern input and the internal recurrent input was high. Synergy and unique information stayed close to zero until the capacity was exceeded and performance dropped steeply. nature.com
Nature Communicationsnature.comOpen source ↗
Redundancy as a learning goal
Based on this observation, the authors built infomorphic associative memory neurons. Each neuron maximizes a local goal function of redundancy. According to the authors, this builds on and improves a proof of concept by Makkeh et al. With this goal the network's memory capacity reached 1.59.
The authors conclude that redundancy maximization can serve as a design principle for associative memories. They state that it opens a way to new associative memory models based on information-theoretic goals.
