The simultaneous communication of disjointness with applications to data streams

We study k-party number-in-hand set disjointness in the simultaneous message-passing model, and show that even if each element i ∈ [n] is guaranteed to either belong to all k parties or to at most O(1) parties in expectation (and to at most O(log n) parties with high probability), then Ω(n min(log 1/δ, log k)/k) communication is required by any δ-error communication protocol for this problem (assuming k = Ω(log n)). We use the strong promise of our lower bound, together with a recent characterization of turnstile streaming algorithms as linear sketches, to obtain new lower bounds for the well-studied problem in data streams of approximating the frequency moments. We obtain a space lower bound of Ω(n^1−2/pε⁻² logM log 1/δ) bits for any algorithm giving a (1 + ε)-approximation to the p-th moment (Formula Presented.) of an n-dimensional vector x ∈ {±M}ⁿ with probability 1 − δ, for any (Formula Presented.). Our lower bound improves upon a prior Ω(n^1−2/pε⁻² log M) lower bound which did not capture the dependence on δ, and our bound is optimal whenever ε ≤ 1/poly(log n). This is the first example of a lower bound in data streams which uses a characterization in terms of linear sketches to obtain stronger lower bounds than obtainable via the one-way communication model; indeed, our set disjointness lower bound provably cannot hold in the one-way model.