Summary

The ie-eval all command can be used to compute all available metrics, globally and for each semantic category.

Metrics

The command will display the following metrics:

• Bag-of-Words F1-score
• Bag-of-Tagged-Words F1-score
• Bag-of-Entities F1-score
• ECER and EWER scores
• Nerval F1-score

Parameters

Here are the available parameters for this metric:

Parameter	Description	Type	Default
--label-dir	Path to the directory containing BIO label files.	pathlib.Path
--prediction-dir	Path to the directory containing BIO prediction files.	pathlib.Path
--nerval-threshold	Percentage of transcription error to allow [0, 100]	float	30.0
--by-category	Whether to display the metric for each category.	bool	False

The parameters are also described when running ie-eval all --help.

Examples

Global evaluation

Use the following command to compute the overall BoE metrics:

ie-eval all \
    --label-dir tests/data/labels/ \
    --prediction-dir tests/data/predictions/

It will output the results in Markdown format:

| Category | BoW-F1 (%) | BoTW-F1 (%) | BoE-F1 (%) | ECER (%) | EWER (%) | Nerval-F1 (%) | N documents |
|:---------|:----------:|:-----------:|:----------:|:--------:|:--------:|:-------------:|------------:|
| total    |   80.73    |    77.06    |   71.43    |  21.33   |  28.27   |     82.14     |           5 |

Evaluation for each category

Use the following command to compute the BoE metrics for each semantic category:

ie-eval boe
    --label-dir Simara/labels/ \
    --prediction-dir Simara/predictions/ \
    --by-category

It will output the results in Markdown format:

| Category            | BoW-F1 (%) | BoTW-F1 (%) | BoE-F1 (%) | N documents |
|:--------------------|:----------:|:-----------:|:----------:|------------:|
| total               |   84.88    |    84.26    |   77.20    |         804 |
| intitule            |   81.66    |    81.66    |   43.91    |         804 |
| cote_article        |   96.78    |    96.78    |   97.49    |         676 |
| cote_serie          |   97.49    |    97.49    |   97.71    |         676 |
| precisions_sur_cote |   89.07    |    89.07    |   88.21    |         675 |
| analyse_compl       |   80.22    |    80.22    |   50.78    |         771 |
| date                |   97.52    |    97.52    |   94.46    |         751 |
| classement          |   86.64    |    86.64    |   74.03    |          77 |