Inference and Evaluation

This guides explains how to run inference or a complete evaluation using command-line tools. Inference produces probability of TB presence for input images, while evaluation will analyze such output against existing annotations and produce performance figures.

Inference

In inference (or prediction) mode, we input data, the trained model, and output a CSV file containing the prediction outputs for every input image.

To run inference, use the sub-command predict to run prediction on an existing dataset:

binseg predict -vv <model> -w <path/to/model.pth> <dataset>

Replace <model> and <dataset> by the appropriate configuration files. Replace <path/to/model.pth> to a path leading to the pre-trained model.

Evaluation

In evaluation, we input a dataset and predictions to generate performance summaries that help analysis of a trained model. Evaluation is done using the evaluate command followed by the model and the annotated dataset configuration, and the path to the pretrained weights via the --weight argument.

Use binseg evaluate --help for more information.

E.g. run evaluation on predictions from the Montgomery set, do the following:

binseg evaluate -vv montgomery -p /predictions/folder -o /eval/results/folder

Comparing Systems

To compare multiple systems together and generate combined plots and tables, use the compare command. Use --help for a quick guide.

binseg compare -vv A A/metrics.csv B B/metrics.csv --output-figure=plot.pdf --output-table=table.txt --threshold=0.5