Inference and Evaluation

This guides explains how to run inference or a complete evaluation using command-line tools. Inference produces probability of TB presence for input images, while evaluation will analyze such output against existing annotations and produce performance figures.

Inference

In inference (or prediction) mode, we input data, the trained model, and output a CSV file containing the prediction outputs for every input image.

To run inference, use the sub-command predict to run prediction on an existing dataset:

$ bob tb predict -vv <model> -w <path/to/model.pth> <dataset>

Replace <model> and <dataset> by the appropriate configuration files. Replace <path/to/model.pth> to a path leading to the pre-trained model.

Tip

An option to generate grad-CAMs is available for the DensenetRS model. To activate it, use the --grad-cams argument.

Tip

An option to generate a relevance analysis plot is available. To activate it, use the --relevance-analysis argument.

Evaluation

In evaluation, we input a dataset and predictions to generate performance summaries that help analysis of a trained model. Evaluation is done using the evaluate command followed by the model and the annotated dataset configuration, and the path to the pretrained weights via the --weight argument.

Use bob tb evaluate --help for more information.

E.g. run evaluation on predictions from the Montgomery set, do the following:

bob tb evaluate -vv montgomery -p /predictions/folder -o /eval/results/folder

Comparing Systems

To compare multiple systems together and generate combined plots and tables, use the compare command. Use --help for a quick guide.

$ bob tb compare -vv A A/metrics.csv B B/metrics.csv --output-figure=plot.pdf --output-table=table.txt --threshold=0.5