Cheat-sheetKeywords: ROC analysis, operating points, discriminants, multi-class
Problem: How to perform ROC analysis in multi-class problems?
Solution: The ROC analysis is performed by output weighting using the sdroc, similarly to the two-class problem
First, we generate a multi-class dataset and divide it into training and test sets. Second, we train the mixture of Gaussian classifier and execute it on the validation set, storing its soft outputs.
>> a=sddata(gendatm(1000))
'Multi-Class Problem' 1000 by 2 sddata, 8 classes: [122 123 110 146 114 138 129 118]
>> [tr,ts]=randsubset(a,0.5);
>> p=sdlinear(tr);
>> out=ts*p
Multi-Class Problem, 498 by 2 sddata, 8 classes: [65 55 55 72 58 59 68 66]
Multi-class ROC is estimated using the sdroc command analogously to the two-class case. By default, the sdroc estimates a sub-optimal ROC using a greedy optimizer:
>> r=sdroc(out)
>> sddrawroc(r)
Use cursor keys to flip through the available per-class error measures in the ROC plot. The weights for a given operating point may be shown in the plot title by pressing the 'w' key. Press the 's' key (save) on the keyboard to store the selected operating point as default.