We report a series of robust empirical observations, demonstrating that deep Neural Networks learn the examples in both the training and test sets in a similar order. This phenomenon is observed in all the commonly used benchmarks we evaluated, including many image classification benchmarks, and one text classification benchmark. While this phenomenon is strongest for models of the same architecture, it also crosses architectural boundaries - models of different architectures start by learning the same examples, after which the more powerful model may continue to learn additional examples. We further show that this pattern of results reflects the interplay between the way neural networks learn benchmark datasets. Specifically, when fixing the architecture, we describe synthetic datasets for which this pattern is no longer observed. When fixing the dataset, we show that other learning paradigms may learn the data in a different order. We hypothesize that our results reflect how neural networks discover structure in natural datasets.
|Original language||American English|
|Title of host publication||37th International Conference on Machine Learning, ICML 2020|
|Editors||Hal Daume, Aarti Singh|
|Publisher||International Machine Learning Society (IMLS)|
|Number of pages||11|
|State||Published - 2020|
|Event||37th International Conference on Machine Learning, ICML 2020 - Virtual, Online|
Duration: 13 Jul 2020 → 18 Jul 2020
|Name||37th International Conference on Machine Learning, ICML 2020|
|Conference||37th International Conference on Machine Learning, ICML 2020|
|Period||13/07/20 → 18/07/20|
Bibliographical noteFunding Information:
This work was supported in part by a grant from the Israel Science Foundation (ISF) and by the Gatsby Charitable Foundations. We thank Reviewer #2 for proposing a more intuitive graphical representation of the results. This representation features prominently throughout the paper.
© International Conference on Machine Learning, ICML 2020. All rights reserved.