Constructing Multiclass Classifiers using Binary Classifiers under Log-Loss

Assaf Ben-Yishai; Or Ordentlich

doi:10.1109/ISIT45174.2021.9518152

Constructing Multiclass Classifiers using Binary Classifiers under Log-Loss

Assaf Ben-Yishai, Or Ordentlich

The Rachel and Selim Benin School of Engineering and Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

The construction of multiclass classifiers from binary classifiers is studied in this paper, and performance is quantified by the regret, defined with respect to the Bayes optimal log-loss. We start by proving that the regret of the well known One vs. All (OVA) method is upper bounded by the sum of the regrets of its constituent binary classifiers. We then present a new method called Conditional OVA (COVA), and prove that its regret is given by the weighted sum of the regrets corresponding to the constituent binary classifiers. Lastly, we present a method termed Leveraged COVA (LCOVA), designated to reduce the regret of a multiclass classifier by breaking it down to independently optimized binary classifiers.

Original language	American English
Title of host publication	2021 IEEE International Symposium on Information Theory, ISIT 2021 - Proceedings
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	2435-2440
Number of pages	6
ISBN (Electronic)	9781538682098
DOIs	https://doi.org/10.1109/ISIT45174.2021.9518152
State	Published - 12 Jul 2021
Event	2021 IEEE International Symposium on Information Theory, ISIT 2021 - Virtual, Melbourne, Australia Duration: 12 Jul 2021 → 20 Jul 2021

Publication series

Name	IEEE International Symposium on Information Theory - Proceedings
Volume	2021-July
ISSN (Print)	2157-8095

Conference

Conference	2021 IEEE International Symposium on Information Theory, ISIT 2021
Country/Territory	Australia
City	Virtual, Melbourne
Period	12/07/21 → 20/07/21

Bibliographical note

Funding Information:
VI. ACKNOWLEDGMENT The authors would like to thank Meir Feder and Yury Polyanskiy for many fruitful discussions. This work was supported by the ISF under Grant 1791/17.

Publisher Copyright:
© 2021 IEEE.

Access to Document

10.1109/ISIT45174.2021.9518152

Cite this

Ben-Yishai, A., & Ordentlich, O. (2021). Constructing Multiclass Classifiers using Binary Classifiers under Log-Loss. In 2021 IEEE International Symposium on Information Theory, ISIT 2021 - Proceedings (pp. 2435-2440). (IEEE International Symposium on Information Theory - Proceedings; Vol. 2021-July). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ISIT45174.2021.9518152

@inproceedings{5319982f3d1f42d8b0fa329424a1dde2,

title = "Constructing Multiclass Classifiers using Binary Classifiers under Log-Loss",

abstract = "The construction of multiclass classifiers from binary classifiers is studied in this paper, and performance is quantified by the regret, defined with respect to the Bayes optimal log-loss. We start by proving that the regret of the well known One vs. All (OVA) method is upper bounded by the sum of the regrets of its constituent binary classifiers. We then present a new method called Conditional OVA (COVA), and prove that its regret is given by the weighted sum of the regrets corresponding to the constituent binary classifiers. Lastly, we present a method termed Leveraged COVA (LCOVA), designated to reduce the regret of a multiclass classifier by breaking it down to independently optimized binary classifiers.",

author = "Assaf Ben-Yishai and Or Ordentlich",

note = "Funding Information: VI. ACKNOWLEDGMENT The authors would like to thank Meir Feder and Yury Polyanskiy for many fruitful discussions. This work was supported by the ISF under Grant 1791/17. Publisher Copyright: {\textcopyright} 2021 IEEE.; 2021 IEEE International Symposium on Information Theory, ISIT 2021 ; Conference date: 12-07-2021 Through 20-07-2021",

year = "2021",

month = jul,

day = "12",

doi = "10.1109/ISIT45174.2021.9518152",

language = "American English",

series = "IEEE International Symposium on Information Theory - Proceedings",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "2435--2440",

booktitle = "2021 IEEE International Symposium on Information Theory, ISIT 2021 - Proceedings",

address = "United States",

}

Ben-Yishai, A & Ordentlich, O 2021, Constructing Multiclass Classifiers using Binary Classifiers under Log-Loss. in 2021 IEEE International Symposium on Information Theory, ISIT 2021 - Proceedings. IEEE International Symposium on Information Theory - Proceedings, vol. 2021-July, Institute of Electrical and Electronics Engineers Inc., pp. 2435-2440, 2021 IEEE International Symposium on Information Theory, ISIT 2021, Virtual, Melbourne, Australia, 12/07/21. https://doi.org/10.1109/ISIT45174.2021.9518152

Constructing Multiclass Classifiers using Binary Classifiers under Log-Loss. / Ben-Yishai, Assaf; Ordentlich, Or.
2021 IEEE International Symposium on Information Theory, ISIT 2021 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2021. p. 2435-2440 (IEEE International Symposium on Information Theory - Proceedings; Vol. 2021-July).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Constructing Multiclass Classifiers using Binary Classifiers under Log-Loss

AU - Ben-Yishai, Assaf

AU - Ordentlich, Or

N1 - Funding Information: VI. ACKNOWLEDGMENT The authors would like to thank Meir Feder and Yury Polyanskiy for many fruitful discussions. This work was supported by the ISF under Grant 1791/17. Publisher Copyright: © 2021 IEEE.

PY - 2021/7/12

Y1 - 2021/7/12

N2 - The construction of multiclass classifiers from binary classifiers is studied in this paper, and performance is quantified by the regret, defined with respect to the Bayes optimal log-loss. We start by proving that the regret of the well known One vs. All (OVA) method is upper bounded by the sum of the regrets of its constituent binary classifiers. We then present a new method called Conditional OVA (COVA), and prove that its regret is given by the weighted sum of the regrets corresponding to the constituent binary classifiers. Lastly, we present a method termed Leveraged COVA (LCOVA), designated to reduce the regret of a multiclass classifier by breaking it down to independently optimized binary classifiers.

AB - The construction of multiclass classifiers from binary classifiers is studied in this paper, and performance is quantified by the regret, defined with respect to the Bayes optimal log-loss. We start by proving that the regret of the well known One vs. All (OVA) method is upper bounded by the sum of the regrets of its constituent binary classifiers. We then present a new method called Conditional OVA (COVA), and prove that its regret is given by the weighted sum of the regrets corresponding to the constituent binary classifiers. Lastly, we present a method termed Leveraged COVA (LCOVA), designated to reduce the regret of a multiclass classifier by breaking it down to independently optimized binary classifiers.

UR - http://www.scopus.com/inward/record.url?scp=85115058479&partnerID=8YFLogxK

U2 - 10.1109/ISIT45174.2021.9518152

DO - 10.1109/ISIT45174.2021.9518152

M3 - Conference contribution

AN - SCOPUS:85115058479

T3 - IEEE International Symposium on Information Theory - Proceedings

SP - 2435

EP - 2440

BT - 2021 IEEE International Symposium on Information Theory, ISIT 2021 - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2021 IEEE International Symposium on Information Theory, ISIT 2021

Y2 - 12 July 2021 through 20 July 2021

ER -

Constructing Multiclass Classifiers using Binary Classifiers under Log-Loss

Abstract

Publication series

Conference

Bibliographical note

Access to Document

Other files and links

Fingerprint

Cite this