perClass Mira provides a convenient Cross-validation tool significantly simplifying statistical validation of models.
What is cross-validation?
Cross-validation is a procedure of repeated retraining of a model and re-testing on different fractions of the data. The goal is to assess variability of model performance on a given problem. Repeated testing provides us with performance estimates accompanied with their respective standard deviations. Cross-validation allows us to compare two machine learning models at certain significance level. In other words, it allows us to conclude whether one model performs significantly better than other.
Cross-validation in perClass Mira
The cross-validation tool in perClass Mira can support classification, regression, or custom external analysis work-flows. We will explain Cross-validation tool based on a regression example. We are estimating mixing proportion of two powders using regression modeling. We have placed the Cross-validation panel in the center under the image view. On the right side, we have a Regression panel with results of a model, trained on a set of scans. Note, that three images were manually flagged as testing. Therefore, we observe three green points in the Regression plot.
For this example, it is useful to clarify the scanning task and the structure of file names. We have a set of plastic vials with mixed powder. Each vial containing one specific mixing proportion. We scan each vial alone multiple times. Each time, we record which mixing proportion the vial contains by an integer below. The letterdenotes which replica scan of the same vial we have acquired. Replicas A, B, and C mean three repeated scans of the same powder container,
In order to use the cross-validation, we need to make a selection of images. In most cases, we wish to cross-validate on all images in the project. Therefore may select all using the Ctrl+A keyboard shortcut.
When an image selection is created, the Cross-validation panel will fill the selected images in a tableand its controlswill become enabled.
We will first explain default cross-validation over images.