On the Sample Complexity of Two-Layer Networks: Lipschitz Vs. Element-Wise Lipschitz Activation

This study delves into the sample complexity of two-layer neural networks. For a given reference matrix W⁰ ∈ R^T ×^d (typically representing initial training weights) and an O(1)-Lipschitz activation function σ : R → R, we examine the class This bound is optimal, barring logarithmic factors, and depends logarithmically on the width T . This finding improves on Vardi et al. (2022), who established a similar outcome for W⁰ = 0. Our motivation stems from the real-world observation that trained weights often remain close to their initial counterparts, implying that kWk_Frobenius << kW + W⁰k_Frobenius. To arrive at our conclusion, we employed and enhanced a recently new norm-based bounds method, the Approximate Description Length (ADL), as proposed by Daniely and Granot (2019). Finally, our results underline the crucial role of the element-wise nature of σ for achieving a logarithmic width-dependent bound. We prove that there exists an O(1)-Lipschitz (non-element-wise) activation function Ψ: R^T → R^T where the sample complexity of H_W^Ψ_0,B,R,r increases linearly with the width.