Practical estimation of the optimal classification error with soft labels and calibration

ArXi:2505.20761v3 Announce Type: replace While the performance of machine learning systems has experienced significant improvement in recent years, relatively little attention has been paid to the fundamental question: to what extent can we improve our models? This paper provides a means of answering this question in the setting of binary classification, which is practical and theoretically ed. We extend a previous work that utilizes soft labels for estimating the Bayes error, the optimal error rate, in two important ways.