Depth Separation for Neural Networks

Let f : S^d-1 × S^d-1 → R be a function of the form f(x, x⁰) = g(hx, x⁰i) for g : [-1, 1] → R. We give a simple proof that shows that poly-size depth two neural networks with (exponentially) bounded weights cannot approximate f whenever g cannot be approximated by a low degree polynomial. Moreover, for many g’s, such as g(x) = sin(πd³x), the number of neurons must be 2^{Ω(d log(d))}. Furthermore, the result holds w.r.t. the uniform distribution on S^d-1 × S^d-1. As many functions of the above form can be well approximated by poly-size depth three networks with poly-bounded weights, this establishes a separation between depth two and depth three networks w.r.t.