The design is evaluated based on accuracy and throughput.
ACCURACY
Accuracy is measured using the F1-score, where:
F1-score = (2 * precision * recall) / (precision + recall)
precision = global_true_positives / (global_true_positives + global_false_positives)
recall = global_true_positives / (global_true_positives + global_false_negatives)
True positives are when a reported object matches the type and location (IoU > 0.5) of an object from the golden data.
The minimum accuracy should be TBD, otherwise a penalty is applied.
THROUGHPUT
Throughput is measured in FPS. The time to read images from the SD card is not included.
The design should achieve at least 5 FPS (subject to change), otherwise a penatly is applied.
SCORING FUNCTION
The score for a team is calculated as follows (Subject to change):
team score = F1-score^2 * fps
Scoring Script
A score.py script is provided that will provide f1-score and fps.