Built-in Recipes

Mark entity spans in a text by highlighting them and selecting the respective labels. The model is used to tokenize the text to allow less sensitive highlighting, since the token boundaries are used to set the entity spans. The label set can be defined as a comma-separated list on the command line or as a path to a text file with one label per line. If no labels are specified, Prodigy will check if labels are present in the model. This recipe does not require an entity recognizer, and doesn’t do any active learning.

Create gold-standard data for NER by correcting the model’s suggestions. The spaCy pipeline will be used to predict entities contained in the text, which the annotator can remove and correct if necessary.

Collect the best possible training data for a named entity recognition model with the model in the loop. Based on your annotations, Prodigy will decide which questions to ask next. If the suggested entity is fully correct, you can accept it. If it’s entirely or partially wrong, you should reject it. As of v1.11, the recipe will also ask you about examples containing no entities at all, which can improve overall accuracy of your model. So if you see an example with no highlighted suggestions, you can accept it if the text contains no entities, or reject it if it does contain entities of the labels you’re annotating.

Take existing “silver” datasets with binary accept/reject annotations, merge the annotations to find the best possible analysis given the constraints defined in the annotations, and manually edit it to create a perfect and complete “gold” dataset.

Load two models and a stream of text, compare their predictions and select which result you prefer. The outputs will be randomized, so you won’t know which model is which. When you stop the server, the results are calculated. This recipe is especially helpful if you’re updating an existing model or if you’re trying out a new strategy on the same problem. Even if two models achieve similar accuracy, one of them can still be subjectively “better”, so this recipe lets you analyze that.

Leverage a model to add NER annotations to the database. You can repeat this multiple times with different models so that you may easily compare their predictions using the review recipe and curate examples where models disagree.

For more information on this method of annotating you can consult this guide.

Getting labels from the 'ner' component
Using 1 labels: ['PERSON']
100%|███████████████████████████████████████████████████████|
201/201 [00:00<00:00, 395.14it/s]

Mark entity spans in a text by highlighting them and selecting the respective labels. The model is used to tokenize the text to allow less sensitive highlighting, since the token boundaries are used to set the entity spans. The label set can be defined as a comma-separated list on the command line or as a path to a text file with one label per line. If no labels are specified, Prodigy will check if labels are present in the model. This recipe does not require an entity recognizer, and doesn’t do any active learning.

If you’re using a custom suggester function for the span categorizer, you can provide it via the --suggester argument and Prodigy will validate submitted annotations against it as you annotate. If you’re not using a suggester, data-to-spacy and train will infer the best-matching ngram suggester based on the available span annotations in your data.

suggester.py
from spacy import registry
from spacy.pipeline.spancat import build_ngram_suggester

@registry.misc("123_ngram_suggester.v1")
def custom_ngram_suggester():
    return build_ngram_suggester(sizes=[1, 2, 3])  # all ngrams of size 1, 2 and 3

Create gold-standard data for span categorization by correcting the model’s predictions. Requires a spaCy pipeline with a trained span categorizer and will show all spans in the given group. To customize the span group to read from, you can use the --key argument.

Leverage a model to add span annotations to the database. You can repeat this multiple times with different models so that you may easily compare their predictions using the review recipe and curate examples where models disagree.

For more information on this method of annotating you can consult this guide.

Getting labels from the 'spancat' component
Using 3 labels: ['PERSON', 'COMPANY', 'CITY']
100%|███████████████████████████████████████████████████████|
201/201 [00:00<00:00, 395.14it/s]

Manually annotate categories that apply to a text. If only one label is set, the classification interface is used. If more than one label is specified, the choice interface is used and categories are added as multiple choice options. If the --exclusive flag is set, categories become mutually exclusive, meaning that only one can be selected during annotation.

Create training data for an existing trained text classification model by correcting the model’s suggestions. The --threshold is used to determine whether a label should be pre-selected, e.g. if it’s set to 0.5 (default), all labels with a score of 0.5 and above will be checked automatically. Prodigy will automatically infer whether the categories are mutually exclusive, based on the component configuration.

Collect the best possible training data for a text classification model by using a model in the loop. Based on your annotations, Prodigy will decide which questions to ask next. All annotations will be stored in the database. If a patterns file is supplied via the --patterns argument, the matches will be included in the stream and the matched spans are highlighted, so you’re able to tell which words or phrases the selection was based on. Note that the exact pattern matches have no influence when updating the model – they’re only used to help pre-select examples for annotation.

Leverage a model to add texcat annotations to the database. You can repeat this multiple times with different models so that you may easily compare their predictions using the review recipe and curate examples where models disagree.

For more information on this method of annotating you can consult this guide.

Getting labels from the 'textcat' component.
Using 3 labels: ['POLICY', 'VC', 'AI']
100%|███████████████████████████████████████████████████████|
201/201 [00:00<00:00, 395.14it/s]

Create gold-standard data for part-of-speech tagging by correcting the model’s suggestions. The spaCy pipeline will be used to predict fine-grained part-of-speech tags (Token.tag_), which the annotator can remove and correct if necessary. It’s often more efficient to focus on a few labels at a time, instead of annotating all labels jointly.

Collect the best possible training data for a part-of-speech tagging model with the model in the loop. Based on your annotations, Prodigy will decide which questions to ask next. It’s often more efficient to focus on a few labels at a time, instead of annotating all labels jointly.

Create gold-standard data for sentence segmentation by correcting the model’s suggestions. The spaCy pipeline will be used to predict sentence boundaries, which the annotator can correct if necessary. The recipe uses the label S to mark tokens that start a sentence. You can double-click a sentence start token in the UI to add a new sentence boundary, or click on an incorrect prediction to remove it.

Collect the best possible training data for a sentence segmentation model with the model in the loop. Based on your annotations, Prodigy will decide which questions to ask next. The recipe uses S to mark tokens that start sentences and I for all other tokens. You can then hit accept or reject, depending on whether the suggested token is correctly labelled as a sentence start or other token.

Create gold-standard data for dependency parsing by correcting the model’s suggestions. The spaCy pipeline will be used to predict dependencies for the given labels, which the annotator can remove and correct if necessary. If --update is set, the model in the loop will be updated with the annotations and its updated predictions will be reflected in future batches. The recipe performs no example selection and all texts will be shown as they come in.

Collect the best possible training data for a dependency parsing model with the model in the loop. Based on your annotations, Prodigy will decide which questions to ask next. It’s often more efficient to focus on a few most relevant labels at a time, instead of annotating all labels jointly.

Create training data for coreference resolution. Coreference resolution is the challenge of linking ambiguous mentions such as “her” or “that woman” back to an antecedent providing more context about the entity in question. This recipe allows you to focus on nouns, proper nouns and pronouns specifically, by disabling all other tokens. You can customize the labels used to extract those using the recipe arguments. Also see the usage guide on coreference annotation.

Annotate directional relations and dependencies between tokens and expressions by selecting the head, child and dependency label and optionally assign labelled spans for named entities or other expressions. This workflow is extremely powerful and can be used for basic dependency annotation, as well as joint named entity and entity relation annotation. If --span-label defines additional span labels, a second mode for span highlighting is added.

The recipe lets you take advantage of several efficiency tricks: spans can be pre-defined using an existing NER dataset, entities or noun phrases from a model or fully custom match patterns. You can also disable certain tokens to make them unselectable. This lets you focus on what matters and prevents annotators from introducing mistakes. For more details and examples, check out the usage guide on custom relation annotation and see the task-specific recipes dep.correct and coref.manual that include pre-defined configurations.

disable_patterns.jsonl
{"pattern": [{"is_punct": true}]}
{"pattern": [{"pos": "VERB"}]}
{"pattern": [{"lower": {"in": ["'s", "’s"]}}]}

Annotate images by drawing rectangular bounding boxes and polygon shapes. Each shape will be added to the task’s "spans" with its label and a "points" property containing the [x, y] pixel coordinate tuples. See here for more details on the JSONL format. You can click and drag or click and release to draw boxes. Polygon shapes can also be closed by double-clicking when adding the last point, similar to closing a shape in Photoshop or Illustrator. Clicking on the label will select a shape so you can change the label or delete it.

If you organize your images in subdirectories, you can set --loader pages to group them together in a single interface using the pages UI. This can be especially useful for multi-page documents or collections of images that should be viewed together. If you’re working with PDFs, the Prodigy-PDF plugin also supports loading paginated documents.

Manually label regions for the given labels in the audio or video file. The recipe expects a directory of audio files as the source argument and will use the audio loader (default) to load the data. To load video files instead, you can set --loader video. Each added region will be added to the "audio_spans" with a start and end timestamp and the selected label.

Manually transcribe audio and video files by typing the transcript into a text field. The recipe expects a directory of audio files as the source argument and will use the audio loader (default) to load the data. To load video files instead, you can set --loader video. The transcript will be stored as the key "transcript". To make it easier to toggle play and pause as you transcribe and to prevent clashes with the text input field (like with the default enter), this recipe lets you customize the keyboard shortcuts. To toggle play/pause, you can press command/option/alt/ctrl+enter or provide your own overrides via --playpause-key.

Train a model with one or more components (NER, text classification, tagger, parser, sentence recognizer or span categorizer) using one or more Prodigy datasets with annotations. The recipe calls into spaCy directly and can update an existing model or train a new model from scratch. For each component, you can provide optional datasets for evaluation using the eval: prefix, e.g. --ner dataset,eval:eval_dataset. If no evaluation sets are specified, the --eval-split is used to determine the percentage held back for evaluation.

Datasets will be merged and conflicts will be filtered out. If your data contains potentially conflicting annotations, it’s recommended to first use review to resolve them. If you specify an output directory as the first argument, the best model will be saved at the end. You can then load it into spaCy by pointing spacy.load at the directory.

Components: ner
Merging training and evaluation data for 1 components
  - [ner] Training: 685 | Evaluation: 300 (from datasets)
Training: 685 | Evaluation: 300
Labels: ner (1)

The example below shows how you might run the same command but with overrides. Notice how this training run uses less steps and has a different learning rate.

Components: ner
Merging training and evaluation data for 1 components
  - [ner] Training: 685 | Evaluation: 300 (from datasets)
Training: 685 | Evaluation: 300
Labels: ner (1)

Train a model with one or more components (NER, text classification, tagger, parser, sentence recognizer or span categorizer) with different portions of the training examples and print the accuracy figures and accuracy improvements with more data. This recipe takes pretty much the same arguments as train. --n-samples sets the number of sample models to train at different stages. For instance, 10 will train models for 10% of the examples, 20%, 30% and so on. This recipe is useful to determine the quality of the collected annotations, and whether more training examples will improve the accuracy. As a rule of thumb, if accuracy improves within the last 25%, training with more examples will likely result in better accuracy.

%      Score    ner
----   ------   ------
  0%   0.00     0.00
 25%   0.31 ▲   0.31 ▲
 50%   0.44 ▲   0.44 ▲
 75%   0.43 ▼   0.43 ▼
100%   0.56 ▲   0.56 ▲

    ┌──────────────────────────────────┐
0.56┤                                 •│
0.44┤                 •           •••• │
0.43┤              ••• •••••••••••     │
    │           •••                    │
0.31┤        •••                       │
    │      ••                          │
    │    ••                            │
    │  ••                              │
0.00┤••                                │
    └┬───────┬────────┬───────┬───────┬┘
     0%     25%      50%     75%    100%

Combine multiple datasets, merge annotations on the same examples and output training and evaluation data in spaCy’s binary .spacy format, which you can use with spacy train. The command takes an output directory and generates all data required to train a pipeline with spaCy, including the config and pre-generated labels data to speed up the training process. This recipe will merge annotations for the different pipeline components and outputs a combined training corpus. If an example is only present in one dataset type, its annotations for the other components will be missing values. It’s recommended to use the review recipe on the different annotation types first to resolve conflicts properly.

Components: ner, textcat
Merging training and evaluation data for 2 components
  - [ner] Training: 685 | Evaluation: 300 (from datasets)
  - [textcat] Training: 538 | Evaluation: 230 (30% split)
Training: 1223 | Evaluation: 530
Labels: ner (3) | textcat (5)

To use this data for training with spaCy, you can run:
python -m spacy train ./corpus/config.cfg --paths.train ./corpus/train.spacy --paths.dev ./corpus/dev.spacy

Generate a starter config for training from Prodigy datasets which you can use with spacy train. A custom reader will be used that merges annotations on the same examples. It’s recommended to use the review recipe on the different annotation types first to resolve conflicts properly (instead of relying on this recipe to just filter conflicting annotations and decide on one).

Build a terminology list interactively using a model’s word vectors and seed terms, either a comma-separated list or a text file containing one term per line. Based on the seed terms, a target vector is created and only terms similar to that target vector are shown. As you annotate, the recipe iterates over the vector model’s vocab and updates the target vector with the words you accept.

Convert a dataset collected with terms.teach or sense2vec.teach to a JSONL-formatted patterns file. You can optionally provide a spaCy pipeline for tokenization to create token-based patterns and make them case-insensitive. If no model is provided, the patterns will be generated as exact string matches. Pattern files can be used in Prodigy to bootstrap annotation and pre-highlight suggestions, for example in ner.manual. You can also use them with spaCy’s EntityRuler for rule-based named entity recognition.

✨ Exported 59 patterns
./prog_lang_patterns.jsonl

This recipe marks entity predictions obtained from a large language model configured by spacy-llm and allows you to accept them as correct, or to manually curate them. This recipe may require environment variables to be set, depending on the large language model that you’re using. The details of this are explained in this section

Example spacy-llm config file for NER
[nlp]
lang = "en"
pipeline = ["llm"]

[components]

[components.llm]
factory = "llm"

[components.llm.task]
@llm_tasks = "spacy.NER.v2"
labels = ["DISH", "INGREDIENT", "EQUIPMENT"]

[components.llm.model]
@llm_models = "spacy.GPT-4.v1"
config = {"temperature": 0.3}

[components.llm.cache]
@llm_misc = "spacy.BatchCache.v1"
path = "local-ner-cache"
batch_size = 3
max_batches_in_mem = 10

The ner.llm.correct recipe fetches examples from large language models while annotating, but this recipe that can fetch a large batch of examples upfront. After downloading such a batch of examples you can use ner.manual to correct the annotations. This recipe may require environment variables to be set, depending on the large language model that you’re using. The details of this are explained in this section

100%|████████████████████████████| 50/50 [00:12<00:00, 3.88it/s]

This recipe marks entity predictions obtained from a large language model configured by spacy-llm and allows you to accept them as correct, or to manually curate them. This recipe may require environment variables to be set, depending on the large language model that you’re using. The details of this are explained in this section

Example spacy-llm config file for textcat
[nlp]
lang = "en"
pipeline = ["llm"]

[components]

[components.llm]
factory = "llm"

[components.llm.task]
@llm_tasks = "spacy.TextCat.v2"
labels = ["RECIPE", "QUESTION", "FEEDBACK"]

[components.llm.model]
@llm_models = "spacy.GPT-4.v1"
config = {"temperature": 0.3}

[components.llm.cache]
@llm_misc = "spacy.BatchCache.v1"
path = "local-ner-cache"
batch_size = 3
max_batches_in_mem = 10

The textcat.llm.correct recipe fetches examples from large language models while annotating, but this recipe that can fetch a large batch of examples upfront. After downloading such a batch of examples you can use textcat.manual to correct the annotations. This recipe may require environment variables to be set, depending on the large language model that you’re using. The details of this are explained in this section

100%|████████████████████████████| 50/50 [00:12<00:00, 3.88it/s]

This recipe marks overlapping span predictions obtained from a large language model configured by spacy-llm and allows you to accept them as correct, or to manually curate them. This recipe may require environment variables to be set, depending on the large language model that you’re using. The details of this are explained in this section.

Basic spacy-llm config for spans
[nlp]
lang = "en"
pipeline = ["llm"]

[components]

[components.llm]
factory = "llm"
save_io = true

[components.llm.task]
@llm_tasks = "spacy.SpanCat.v2"
labels = ["DISH", "INGREDIENT", "EQUIPMENT"]

[components.llm.task.label_definitions]
DISH = "Extract the name of a known dish."
INGREDIENT = "Extract the name of a cooking ingredient, including herbs and spices."
EQUIPMENT = "Extract any mention of cooking equipment. e.g. oven, cooking pot, grill"

[components.llm.model]
@llm_models = "spacy.GPT-3-5.v1"
config = {"temperature": 0.3}

[components.llm.task.examples]
@misc = "spacy.FewShotReader.v1"
path = "span_examples.yaml"

[components.llm.cache]
@llm_misc = "spacy.BatchCache.v1"
path = "local-cached"
batch_size = 3
max_batches_in_mem = 10

span_examples.yaml
- text: 'Mac and Cheese is a popular American pasta variant.'
  entities:
    INGREDIENT: ['Cheese']
    DISH: ['Mac and Cheese']

The spans.llm.correct recipe fetches examples from large language models while annotating, but this recipe that can fetch a large batch of examples upfront. After downloading such a batch of examples you can use spancat.manual to correct the annotations. This recipe may require environment variables to be set, depending on the large language model that you’re using. The details of this are explained in this section.

100%|████████████████████████████| 50/50 [00:12<00:00, 3.88it/s]

This recipe generates terms and phrases obtained from a large language model. These terms can be curated and turned into patterns files, which can help with downstream annotation tasks. The recipe works by iteratively requesting terms from the LLM and by deduplicating the results on each batch. It can be helpful for this recipe to choose a high temperature setting for the LLM.

This recipe may require environment variables to be set, depending on the large language model that you’re using. The details of this are explained in this section.

Example spacy-llm config for terms
[nlp]
lang = "en"
pipeline = ["llm"]

[components]

[components.llm]
factory = "llm"

[components.llm.task]
@llm_tasks = "prodigy.Terms.v1"
batch_size = 50

[components.llm.model]
@llm_models = "spacy.GPT-3-5.v1"
config = {"temperature": 0.3}

100%|████████████████████████████| 50/50 [00:12<00:00, 3.88it/s]

The goal of this recipe is to quickly compare the quality of outputs from a collection of prompts by leveraging a tournament. It uses the Glicko rating system internally to determine the duels as well as the best performing prompt. You can also compare different LLM backends using this recipe.

This recipe may require environment variables to be set, depending on the large language model that you’re using. The details of this are explained in this section.

Example spacy-llm config for the tournament recipe
[nlp]
lang = "en"
pipeline = ["llm"]

[components]

[components.llm]
factory = "llm"

[components.llm.task]
@llm_tasks = "prodigy.TextPrompter.v1"

[components.llm.model]
@llm_models = "spacy.GPT-3-5.v1"
config = {"temperature": 0.3}

Example spacy-llm config for the tournament recipe
[nlp]
lang = "en"
pipeline = ["llm"]

[components]

[components.llm]
factory = "llm"

[components.llm.task]
@llm_tasks = "prodigy.TextPrompter.v1"

[components.llm.model]
@llm_models = "spacy.GPT-4.v1"
config = {"temperature": 0.3}

prompt1.jinja2
Write a haiku about {{topic}} that rhymes.

prompt2.jinja2
Write a super funny haiku about {{topic}} that rhymes.

{"topic": "star wars"}
{"topic": "python"}
{"topic": "stroopwafels"}

comparison                                                  prob   trials
[prompt1.jinja2 + cfg1.cfg] > [prompt1.jinja2 + cfg2.cfg]   0.50        0
[prompt1.jinja2 + cfg1.cfg] > [prompt2.jinja2 + cfg1.cfg]   0.50        0
[prompt1.jinja2 + cfg1.cfg] > [prompt2.jinja2 + cfg1.cfg]   0.71        1

comparison                                                  prob   trials
[prompt1.jinja2 + cfg1.cfg] > [prompt1.jinja2 + cfg2.cfg]   0.96       15
[prompt1.jinja2 + cfg1.cfg] > [prompt2.jinja2 + cfg1.cfg]   0.94       12
[prompt1.jinja2 + cfg1.cfg] > [prompt2.jinja2 + cfg1.cfg]   0.89       10

This recipe marks entity predictions obtained from a large language model and allows you to accept them as correct, or to manually curate them. This allows you to quickly gather a gold-standard dataset through zero-shot or few-shot learning. It’s very much like using the ner.correct recipe, but with GPT-3 as a backend model to make predictions.

The ner.openai.correct recipe fetches examples from OpenAI while annotating, but this recipe that can fetch a large batch of examples upfront. After downloading such a batch of examples loaded you can use ner.manual to correct the OpenAI annotations.

100%|████████████████████████████| 50/50 [00:12<00:00, 3.88it/s]

This recipe enables you to classify texts by correcting the annotations from an OpenAI language model. OpenAI will also provide a “reason” to explain why a particular label was chosen.

The textcat.openai.correct recipe fetches examples from OpenAI while annotating, but this recipe that can fetch a large batch of examples upfront. This is helpful when you are with a highly-imbalanced data and interested only in rare examples. After downloading such a batch of examples loaded you can use ner.manual to correct the OpenAI annotations.

100%|████████████████████████████| 50/50 [00:12<00:00, 3.88it/s]

This recipe generates terms and phrases obtained from a large language model. These terms can be curated and turned into patterns files, which can help with downstream annotation tasks.

100%|████████████████████████████| 100/100 [00:12<00:00, 3.88it/s]

The goal of this recipe is to quickly compare the quality of outputs from two prompts in a quantifiable and blind way.

✔ You preferred prompt1.jinja2
prompt1.jinja2   11
prompt2.jinja2    5

The goal of this recipe is to quickly compare the quality of outputs from a collection of prompts by leveraging a tournament. It uses the Glicko rating system internally to determine the duels as well as the best performing prompt. Your can read more about the expectations of the jsonl/jinja2 files in the tournaments guide

comparison                          value  count
P(prompt3.jinja2 > prompt1.jinja2)   0.60      7
P(prompt3.jinja2 > prompt2.jinja2)   0.83      3

comparison                          value   count
P(prompt3.jinja2 > prompt1.jinja2)   0.95      19
P(prompt3.jinja2 > prompt2.jinja2)   0.97       5

Review existing annotations created by multiple annotators and resolve potential conflicts by creating one final “master annotation”. Can be used for both binary and manual annotations and supports all interfaces except image_manual and compare. If the annotations were created with a manual interface, the “most popular” version, e.g. the version most sessions agreed on, will be pre-selected automatically.

Compare the output of your model and the output of a baseline on the same inputs. To prevent bias during annotation, Prodigy will randomly decide which output to suggest as the correct answer. When you exit the application, you’ll see detailed stats, including the preferred output. Expects two JSONL files where each entry has an "id" (to match up the outputs on the same input), and an "input" and "output" object with the content to render, e.g. the "text".

model_a.jsonl
{"id": 1, "input": {"text": "FedEx von weltweiter Cyberattacke getroffen"}, "output": {"text": "FedEx hit by worldwide cyberattack"}}

model_b.jsonl
{"id": 1, "input": {"text": "FedEx von weltweiter Cyberattacke getroffen"}, "output": {"text": "FedEx from worldwide Cyberattacke hit"}}

Compute the inter-annotator agreement (IAA) for document-level annotations using percent agreement, Krippendorff’s Alpha, and Gwet’s AC2 as metrics. The algorithm implemention is ported from: https://github.com/pmbaumgartner/prodigy-iaa and is benchmarked on Gwet, 2015 paper. The current implementation supports two types of annotations: multiclass and multilabel (for binary annotations please see metric.iaa.binary). Importantly, the annotations are grouped by the _input_hash i.e. all annotations that have the same _input_hash are considered to be the same annotation task. For details on other source data assumptions and the interpretation of the results please see the metrics guide.

The command will output the results to the terminal:

Attribute                     Value
--------------------------    -----
Examples                      100
Categories                    3
Coincident Examples*          50
Single Annotation Examples    4
Annotators                    2
Avg. Annotations per Example  2.00
* (>1 annotation)

Statistic                     Value
--------------------------    -----
Percent (Simple) Agreement    0.4167
Krippendorff's Alpha          0.1809
Gwet's AC2                    0.1640

Compute the inter-annotator agreement (IAA) for span-level text annotations using micro-averaged pairwise F1 score as the metric. For computing IAA on reject/accept decisions of span annotations see metric.iaa.binary. Importantly, the annotations are grouped by the _input_hash i.e. all annotations that have the same _input_hash are considered to be the same annotation task. For more details on other source data assumptions and the interpretation of the results please see the metrics guide.

The command will output the results to the terminal:

Attribute                     Value
--------------------------    -----
Examples                      22
Categories                    2
Coincident Examples*          12
Single Annotation Examples    0
Annotators                    2
Avg. Annotations per Example  2.00
\* (>1 annotation)

Label                         Value
--------------------------    -----
PER                           0.0
ORG                           0.5

        PER    ORG   NONE
---    ----   ----   ----
PER    0.00   0.00   0.00
ORG    0.00   0.67   0.33
NONE   0.00   0.60   0.40

Compute the inter-annotator agreement (IAA) for binary annotations using percent agreement, Krippendorff’s Alpha, and Gwet’s AC2 as metrics. The algorithm implemention is ported from: https://github.com/pmbaumgartner/prodigy-iaa and is benchmarked on Gwet, 2015 paper. Importantly, the annotations are grouped by the _input_hash i.e. all annotations that have the same _input_hash are considered to be the same annotation task. For details on other source data assumptions and the interpretation of the results please see the metrics guide.

The command will output the results to the terminal:

Attribute                     Value
--------------------------    -----
Examples                      37
Categories                    2
Coincident Examples*          17
Single Annotation Examples    3
Annotators                    2
Avg. Annotations per Example  1.85
\* (>1 annotation)

Statistic                     Value
--------------------------    -----
Percent (Simple) Agreement    0.5294
Krippendorff's Alpha          0.1833
Gwet's AC2                    0.1681

Start the annotation server, display whatever comes in with a given interface and collect binary annotations. At the end of the annotation session, a breakdown of the answer counts is printed. The --view-id lets you specify one of the existing annotation interfaces – just make sure your input data includes everything the interface needs, since this recipe does no preprocessing and will just show you whatever is in the data. The recipe is also very useful if you want to re-annotate data exported with db-out.

Select examples based on match patterns and accept or reject the result. Unlike ner.manual with patterns, this recipe will only show examples if they contain pattern matches. It can be used for NER and text classification annotations – for instance, to bootstrap a text category if the classes are very imbalanced and not enough positive examples are presented during manual annotation or textcat.teach. The --label-task and --label-span flags can be used to specify where the label should be added. This will also be reflected via the "label" property (on the top-level task or the spans) in the data you create with the recipe. If --combine-matches is set, all matches will be presented together. Otherwise, each match will be presented as a separate task.