Cheat-sheetProblem: How to reliably measure classifier execution speed?
Solution: Use sdexe command to measure only the buffer processing, not overhead of runtime setup.
31.1. Introduction ↩
perClass was designed to provide fast classifier execution in real-time applications, such as on-line sorting or defect detection. Apart of accurate classifier, the major requirement in such applications is fast processing. The classifier has only a limited time to return the results, typically few milliseconds. In this article, we discuss how to reliably measure classifier execution speed.
31.2. Speed measurement in Matlab ↩
Matlab provides tic/toc commands to measure time needed for execution
of a block of code.
Let us make a small example. We will train a Parzen classifier on fruit data set:
>> a
'Fruit set' 260 by 2 sddata, 3 classes: 'apple'(100) 'banana'(100) 'stone'(60)
>> p=sdparzen(a)
.....sequential pipeline 2x1 'Parzen model+Decision'
1 Parzen model 2x3 260 prototypes, h=0.8
2 Decision 3x1 weighting, 3 classes
To time execution, we run it on 10000 random samples:
>> data=rand(10000,2);
>> tic; data*p; toc
Elapsed time is 0.095310 seconds.
>> tic; data*p; toc
Elapsed time is 0.092284 seconds.
>> tic; data*p; toc
Elapsed time is 0.093810 seconds.
The tic/toc timing showed that our classifier can process 10000 samples
in about 94 milliseconds.
This, however, includes also the time needed to setup large memory buffers and preparing classifier for execution. In our deployed application, we are performing such tasks as well, but only once. What is more important is the time needed to process 10000 samples in a continuous manner.
In perClass, we can measure that using the sdexe command. It returns, as
the second output, the time needed only to process the input buffer
(without setup and allocation time).
>> [out,t]=sdexe(p,data);
>> t
t =
0.0534
>> [out,t]=sdexe(p,data);
>> t
t =
0.0536
>> [out,t]=sdexe(p,data);
>> t
t =
0.0534
As we can see, the relevant processing time only 54 milliseconds.
31.3. Speed measurement out of Matlab ↩
When we export our classifier for execution using sdexport, we may
measure the execution time out-of-Matlab.
>> sdexport(p,'/tmp/out.ppl')
Exporting pipeline..ok
This pipeline requires perClass runtime version 4.0 (11-jul-2013) or higher.
The fastest way is to use the sdrun utility, bundled with perClass in
SDK/interfaces/sdrun directory.
When provide with the pipeline file, it only prints pipeline details:
$ ./sdrun /tmp/out.ppl
Pipeline name: 'Parzen model+Decision'
Minimum required runtime version: 4.0 (11-jul-2013)
Input type: double, dimensionality: 2
Output type: int, dimensionality: 1, decisions
Operating point count: 1, current: 1
Possible decisions: 1:apple, 2:banana, 3:stone
When called with the -t switch, sdrun will perform time
measurements. By default on 100000 samples. We may provide a different buffer size as an extra
argument:
$ ./sdrun /tmp/out.ppl -t 10000
Measuring execution speed for 'Parzen model+Decision'
double precision, 2D input: ..........
Average speed on 10 rounds of 10000 samples: 56.0 msec.
Total throughput: 178.51 kilo samples/sec.
The timing is performed ten times, only measuring the time needed to process data in an existing buffer. As we can see, also here we get average speed of 56 milliseconds.
31.4. Speed measurement via perClass Runtime API ↩
Finally, you may perform precise speed measurements via sd_Tic and
sd_Toc functions of perClass Runtime API.
double elapsed_sec;
...
pk=sd_InitKernel(NULL);
...
sd_Tic(pk);
/* your code to time */
elapsed_sec=sd_Toc(pk);
31.5. Summary ↩
When measuring execution speed of your classifiers, make sure you measure
only the time needed to perform processing of input buffer in a tight
control loop of your system. This can be easily done via sdexe command in
Matlab or sdrun utility on the command line.