TY - JOUR
T1 - Dynamic Speculation Lookahead Accelerates Speculative Decoding of Large Language Models
AU - Mamou, Jonathan
AU - Pereg, Oren
AU - Korat, Daniel
AU - Berchansky, Moshe
AU - Timor, Nadav
AU - Wasserblat, Moshe
AU - Schwartz, Roy
N1 - Publisher Copyright:
© 2024 Proceedings of Machine Learning Research.
PY - 2024
Y1 - 2024
N2 - Speculative decoding is commonly used for reducing the inference latency of large language models. Its effectiveness depends highly on the speculation lookahead (SL) - the number of tokens generated by the draft model at each iteration. In this work we show that the common practice of using the same SL for all iterations (static SL) is suboptimal. We introduce DISCO (DynamIc SpeCulation lookahead Optimization), a novel method for dynamically selecting the SL. Our experiments with four datasets show that DISCO reaches an average speedup of 10% compared to the best static SL baseline, while generating the exact same text.
AB - Speculative decoding is commonly used for reducing the inference latency of large language models. Its effectiveness depends highly on the speculation lookahead (SL) - the number of tokens generated by the draft model at each iteration. In this work we show that the common practice of using the same SL for all iterations (static SL) is suboptimal. We introduce DISCO (DynamIc SpeCulation lookahead Optimization), a novel method for dynamically selecting the SL. Our experiments with four datasets show that DISCO reaches an average speedup of 10% compared to the best static SL baseline, while generating the exact same text.
UR - http://www.scopus.com/inward/record.url?scp=85216793244&partnerID=8YFLogxK
M3 - ???researchoutput.researchoutputtypes.contributiontojournal.conferencearticle???
AN - SCOPUS:85216793244
SN - 2640-3498
VL - 262
SP - 456
EP - 467
JO - Proceedings of Machine Learning Research
JF - Proceedings of Machine Learning Research
T2 - 4th NeurIPS Efficient Natural Language and Speech Processing Workshop
Y2 - 14 December 2024
ER -