Recent work has shown that infusing layout features into language models (LMs) improves processing of visually-rich documents such as scientific papers. Layout-infused LMs are often evaluated on documents with familiar layout features (e.g., papers from the same publisher), but in practice models encounter documents with unfamiliar distributions of layout features, such as new combinations of text sizes and styles, or new spatial configurations of textual elements. In this work, we test whether layout-infused LMs are robust to layout distribution shifts. As a case study, we use the task of scientific document structure recovery, segmenting a scientific paper into its structural categories (e.g., TITLE, CAPTION, REFERENCE). To emulate distribution shifts that occur in practice, we re-partition the GROTOAP2 dataset. We find that under layout distribution shifts model performance degrades by up to 20 F1. Simple training strategies, such as increasing training diversity, can reduce this degradation by over 35% relative F1; however, models fail to reach in-distribution performance in any tested out-of-distribution conditions. This work highlights the need to consider layout distribution shifts during model evaluation, and presents a methodology for conducting such evaluations.
|Original language||American English|
|Title of host publication||Findings of the Association for Computational Linguistics, ACL 2023|
|Publisher||Association for Computational Linguistics (ACL)|
|Number of pages||16|
|State||Published - 2023|
|Event||61st Annual Meeting of the Association for Computational Linguistics, ACL 2023 - Toronto, Canada|
Duration: 9 Jul 2023 → 14 Jul 2023
|Name||Proceedings of the Annual Meeting of the Association for Computational Linguistics|
|Conference||61st Annual Meeting of the Association for Computational Linguistics, ACL 2023|
|Period||9/07/23 → 14/07/23|
Bibliographical notePublisher Copyright:
© 2023 Association for Computational Linguistics.