Storage capacities of committee machines with overlapping and non-overlapping receptive fields

Abstract

We present theoretical investigations via the replica theory of the storage capacities of committee machines with a large number M of hidden units and spherical weights. Difficulties arise in the solution of this problem in the limit of large M. In the case of overlapping receptive fields, as the number of patterns increases, both permutation symmetry and replica symmetry are broken, which leads to the appearance of many order parameters and causes additional difficulty. We observe that the relations among these order parameters yield a set of quantities which are small in the limit of large M, making the asymptotic calculation tractable. Using the one-step replica symmetry breaking scheme, we compute the asymptotic value alpha(c) of the storage capacity per input unit in the limit of large M. We find that alpha(c) similar or equal to (8 root 2/(pi-2))M root ln M. The shift to the case of non-overlapping receptive fields can be made easily; we then find alpha c similar or equal to (8 root 2/pi)root ln M. Both values satisfy the bound of Mitchison and Durbin.