Decoding Stumpers: Large Language Models vs. Human Problem-Solvers

Alon Goldstein; Miriam Havin; Roi Reichart; Ariel Goldstein

doi:10.18653/v1/2023.findings-emnlp.779

Decoding Stumpers: Large Language Models vs. Human Problem-Solvers

Alon Goldstein^*
, Miriam Havin
, Roi Reichart
, Ariel Goldstein

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

1 Scopus citations

Abstract

This paper investigates the problem-solving capabilities of Large Language Models (LLMs) by evaluating their performance on stumpers, unique single-step intuition problems that pose challenges for human solvers but are easily verifiable. We compare the performance of four state-of-the-art LLMs (Davinci-2, Davinci-3, GPT-3.5-Turbo, GPT-4) to human participants. Our findings reveal that the new-generation LLMs excel in solving stumpers and surpass human performance. However, humans exhibit superior skills in verifying solutions to the same problems. This research enhances our understanding of LLMs' cognitive abilities and provides insights for enhancing their problem-solving potential across various domains.

Original language	English
Title of host publication	Findings of the Association for Computational Linguistics
Subtitle of host publication	EMNLP 2023
Publisher	Association for Computational Linguistics (ACL)
Pages	11644-11653
Number of pages	10
ISBN (Electronic)	9798891760615
DOIs	https://doi.org/10.18653/v1/2023.findings-emnlp.779
State	Published - 2023
Event	2023 Findings of the Association for Computational Linguistics: EMNLP 2023 - Hybrid, Singapore Duration: 6 Dec 2023 → 10 Dec 2023

Publication series

Name	Findings of the Association for Computational Linguistics: EMNLP 2023

Conference

Conference	2023 Findings of the Association for Computational Linguistics: EMNLP 2023
Country/Territory	Singapore
City	Hybrid
Period	6/12/23 → 10/12/23

Bibliographical note

Publisher Copyright:
© 2023 Association for Computational Linguistics.

Access to Document

10.18653/v1/2023.findings-emnlp.779

Cite this

Goldstein, A., Havin, M., Reichart, R., & Goldstein, A. (2023). Decoding Stumpers: Large Language Models vs. Human Problem-Solvers. In Findings of the Association for Computational Linguistics: EMNLP 2023 (pp. 11644-11653). (Findings of the Association for Computational Linguistics: EMNLP 2023). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2023.findings-emnlp.779

@inproceedings{84a9c0b94e854953b65b466bb9ed2e96,

title = "Decoding Stumpers: Large Language Models vs. Human Problem-Solvers",

abstract = "This paper investigates the problem-solving capabilities of Large Language Models (LLMs) by evaluating their performance on stumpers, unique single-step intuition problems that pose challenges for human solvers but are easily verifiable. We compare the performance of four state-of-the-art LLMs (Davinci-2, Davinci-3, GPT-3.5-Turbo, GPT-4) to human participants. Our findings reveal that the new-generation LLMs excel in solving stumpers and surpass human performance. However, humans exhibit superior skills in verifying solutions to the same problems. This research enhances our understanding of LLMs' cognitive abilities and provides insights for enhancing their problem-solving potential across various domains.",

author = "Alon Goldstein and Miriam Havin and Roi Reichart and Ariel Goldstein",

note = "Publisher Copyright: {\textcopyright} 2023 Association for Computational Linguistics.; 2023 Findings of the Association for Computational Linguistics: EMNLP 2023 ; Conference date: 06-12-2023 Through 10-12-2023",

year = "2023",

doi = "10.18653/v1/2023.findings-emnlp.779",

language = "אנגלית",

series = "Findings of the Association for Computational Linguistics: EMNLP 2023",

publisher = "Association for Computational Linguistics (ACL)",

pages = "11644--11653",

booktitle = "Findings of the Association for Computational Linguistics",

address = "ארצות הברית",

}

Goldstein, A, Havin, M, Reichart, R & Goldstein, A 2023, Decoding Stumpers: Large Language Models vs. Human Problem-Solvers. in Findings of the Association for Computational Linguistics: EMNLP 2023. Findings of the Association for Computational Linguistics: EMNLP 2023, Association for Computational Linguistics (ACL), pp. 11644-11653, 2023 Findings of the Association for Computational Linguistics: EMNLP 2023, Hybrid, Singapore, 6/12/23. https://doi.org/10.18653/v1/2023.findings-emnlp.779

Decoding Stumpers: Large Language Models vs. Human Problem-Solvers. / Goldstein, Alon; Havin, Miriam; Reichart, Roi et al.
Findings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational Linguistics (ACL), 2023. p. 11644-11653 (Findings of the Association for Computational Linguistics: EMNLP 2023).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Decoding Stumpers

T2 - 2023 Findings of the Association for Computational Linguistics: EMNLP 2023

AU - Goldstein, Alon

AU - Havin, Miriam

AU - Reichart, Roi

AU - Goldstein, Ariel

PY - 2023

Y1 - 2023

N2 - This paper investigates the problem-solving capabilities of Large Language Models (LLMs) by evaluating their performance on stumpers, unique single-step intuition problems that pose challenges for human solvers but are easily verifiable. We compare the performance of four state-of-the-art LLMs (Davinci-2, Davinci-3, GPT-3.5-Turbo, GPT-4) to human participants. Our findings reveal that the new-generation LLMs excel in solving stumpers and surpass human performance. However, humans exhibit superior skills in verifying solutions to the same problems. This research enhances our understanding of LLMs' cognitive abilities and provides insights for enhancing their problem-solving potential across various domains.

AB - This paper investigates the problem-solving capabilities of Large Language Models (LLMs) by evaluating their performance on stumpers, unique single-step intuition problems that pose challenges for human solvers but are easily verifiable. We compare the performance of four state-of-the-art LLMs (Davinci-2, Davinci-3, GPT-3.5-Turbo, GPT-4) to human participants. Our findings reveal that the new-generation LLMs excel in solving stumpers and surpass human performance. However, humans exhibit superior skills in verifying solutions to the same problems. This research enhances our understanding of LLMs' cognitive abilities and provides insights for enhancing their problem-solving potential across various domains.

UR - https://www.scopus.com/pages/publications/85183300736

U2 - 10.18653/v1/2023.findings-emnlp.779

DO - 10.18653/v1/2023.findings-emnlp.779

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85183300736

T3 - Findings of the Association for Computational Linguistics: EMNLP 2023

SP - 11644

EP - 11653

BT - Findings of the Association for Computational Linguistics

PB - Association for Computational Linguistics (ACL)

Y2 - 6 December 2023 through 10 December 2023

ER -

Goldstein A, Havin M, Reichart R, Goldstein A. Decoding Stumpers: Large Language Models vs. Human Problem-Solvers. In Findings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational Linguistics (ACL). 2023. p. 11644-11653. (Findings of the Association for Computational Linguistics: EMNLP 2023). doi: 10.18653/v1/2023.findings-emnlp.779

Decoding Stumpers: Large Language Models vs. Human Problem-Solvers

Abstract

Publication series

Conference

Bibliographical note

Access to Document

Other files and links

Fingerprint

Cite this