TY - GEN

T1 - From average case complexity to improper learning complexity

AU - Daniely, Amit

AU - Linial, Nati

AU - Shalev-Shwartz, Shai

PY - 2014

Y1 - 2014

N2 - The basic problem in the PAC model of computational learning theory is to determine which hypothesis classes are efficiently learnable. There is presently a dearth of results showing hardness of learning problems. Moreover, the existing lower bounds fall short of the best known algorithms. The biggest challenge in proving complexity results is to establish hardness of improper learning (a.k.a. representation independent learning). The dificulty in proving lower bounds for improper learning is that the standard reductions from NP-hard problems do not seem to apply in this context. There is essentially only one known approach to proving lower bounds on improper learning. It was initiated in [21] and relies on cryptographic assumptions. We introduce a new technique for proving hardness of improper learning, based on reductions from problems that are hard on average. We put forward a (fairly strong) generalization of Feige's assumption [13] about the complexity of refuting random constraint satisfaction problems. Combining this assumption with our new technique yields far reaching implications. In particular, • Learning DNF's is hard. • Agnostically learning halfspaces with a constant approximation ratio is hard. • Learning an intersection of ω(1) halfspaces is hard.

AB - The basic problem in the PAC model of computational learning theory is to determine which hypothesis classes are efficiently learnable. There is presently a dearth of results showing hardness of learning problems. Moreover, the existing lower bounds fall short of the best known algorithms. The biggest challenge in proving complexity results is to establish hardness of improper learning (a.k.a. representation independent learning). The dificulty in proving lower bounds for improper learning is that the standard reductions from NP-hard problems do not seem to apply in this context. There is essentially only one known approach to proving lower bounds on improper learning. It was initiated in [21] and relies on cryptographic assumptions. We introduce a new technique for proving hardness of improper learning, based on reductions from problems that are hard on average. We put forward a (fairly strong) generalization of Feige's assumption [13] about the complexity of refuting random constraint satisfaction problems. Combining this assumption with our new technique yields far reaching implications. In particular, • Learning DNF's is hard. • Agnostically learning halfspaces with a constant approximation ratio is hard. • Learning an intersection of ω(1) halfspaces is hard.

KW - Average case complexity

KW - CSP problems

KW - DNFs

KW - Halfspaces

KW - Hardness of improper learning

KW - Resolution lower bounds

UR - http://www.scopus.com/inward/record.url?scp=84904336181&partnerID=8YFLogxK

U2 - 10.1145/2591796.2591820

DO - 10.1145/2591796.2591820

M3 - Conference contribution

AN - SCOPUS:84904336181

SN - 9781450327107

T3 - Proceedings of the Annual ACM Symposium on Theory of Computing

SP - 441

EP - 448

BT - STOC 2014 - Proceedings of the 2014 ACM Symposium on Theory of Computing

PB - Association for Computing Machinery

T2 - 4th Annual ACM Symposium on Theory of Computing, STOC 2014

Y2 - 31 May 2014 through 3 June 2014

ER -