Prodigy Plugins

Some Prodigy recipes require a 3rd party library in order to work. To keep Prodigy lightweight we’ve separated some of these recipes out into their own packages so that you may install them as a plugin. These plugins always target the most recent version of Prodigy with regards to compatibility.

This section of the docs showscases such plugins. Note that you can also explore these recipes on Github to serve as a source of inspiration to customise further.

🤗 Prodigy-HF

This plugin contains recipes that interact with the Hugging Face stack. Some recipes will allow you to directly train transformer models on top of your annotations while other recipes allow you to upload artifacts to HF cloud environment.

To use these recipes, you’ll first need to install the plugin.

Install prodigy-hf
pip install "prodigy-hf @ git+https://github.com/explosion/prodigy-hf"

Once it is installed you can explore some of the new recipes.

Training Hugging Face models

The first recipe that you may enjoy from this plugin is the recipe to train custom NER models.

Once the model is done training you’ll be able to inspect the hf-model-dir folder to find all the trained state.

You can also choose to re-use this trained model to help you annotate data. The plugin features a hf.ner.correct recipe that works similarily to ner.correct except here we get to use a Hugging Face model. This means that you can also use models from the Hugging Face Hub. This recipe will internally map the predictions from the transformer model to spaCy tokens.

Note that this plugin also offers variants of these recipes for text classification. Check out the API docs for hf.train.textcat and hf.correct.textcat for more details.

Interacting with Hugging Face Hub

Alternatively, you may also use these plugin to upload your annotated datasets to Hugging Face Hub.

Internally this recipe will validate the dataset for consistency and will attempt to anonymise the annotators before uploading. You can turn this behavior off with flags and you can also specify that you want the dataset not to appear publicly.

API

Prodigy-PDF

This plugin contains recipes for annotating PDF files using the familiar image-based image_manual interface, as well as recipes for OCR (Optical Character Recognition) to extract text-based content from documents.

To use these recipes, you’ll first need to install the plugin. In order for the recipes to work, you may also need to install system dependencies for tesseract.

Install prodigy-pdf
pip install "prodigy-pdf @ git+https://github.com/explosion/prodigy-pdf"

brew install tesseract  # macOs
sudo apt install tesseract-ocr  # Linux

Once it is installed, you can start annotating PDFs as images via pdf.image.manual:

If you like, you can re-use the pdf annotations with the pdf.ocr.correct recipe to apply OCR to the annotated segments. This recipe uses pytessaract under the hood to give suggestions that you can correct.

API

🤫 Prodigy-Whisper

OpenAI released an open model for audio annotation called Whisper. It’s a model that can be downloaded locally, it has support for multiple languages and you’re even able to pick from a selection of models. The model isn’t perfect, but when you’re transcribing text, it can really help to have such a model provide a starting point. The goal of this plugin is to help you get started with this right away.

To use this plugin, you’ll need to install it first.

Install prodigy-whisper
pip install "prodigy-whisper @ git+https://github.com/explosion/prodigy-whisper"

In order to use the plugin you’ll also need to have ffmpeg installed. Most package managers should have these available so you should be able to use one of the following commands.

Install ffmpeg
# on Ubuntu or Debian
sudo apt update && sudo apt install ffmpeg
# on Arch Linux
sudo pacman -S ffmpeg
# on MacOS using Homebrew (https://brew.sh/)
brew install ffmpeg
# on Windows using Chocolatey (https://chocolatey.org/)
choco install ffmpeg
# on Windows using Scoop (https://scoop.sh/)
scoop install ffmpeg

Once the plugin is installed you can use the whisper.audio.transcribe recipe. It is very similar to audio.transcribe recipe that Prodigy provides, but this recipe uses Whisper to provide an initial transcription.

In the base form you can already see that Whisper does a pretty good job at transcription. But it may be easier to correct short pieces of audio instead of a long one. This is where Wishper can help out as well. It is able to segment a long audio clip into shorter segments and each of these segments can then be annotated in Prodigy.

To use this feature, you can add the --segment flag to the recipe call.

Now, you can go through the segments one by one and each segment will have metadata attached so that you can link it back to the timestamps in the original file. This is what the first segment would look like.

This is what the second segment would look like.

API

Prodigy-Segment

Sometimes you’re interested in selecting pixels from an image, as opposed to merely selecting a bounding box. Selecting the right pixels can be tedious work so you may want to use a model in the loop to help you. A good choice for such a model is Meta’s Segment Anything model, which we’ve integrated into Prodigy via the prodigy-segment plugin.

This model is able to take bounding box annotations from Prodigy to construct a pixel segmentation map under the hood. From the UI, that might look like this:

Using Prodigy-Segment

For a quick overview of the features, you may also enjoy this Youtube tutorial.

Before you’ll be able to use recipes, you’ll want to make sure you’ve downloaded the appropriate model checkpoint beforehand. You can check the available models here but this tutorial will assume the “default” model-type. The weights for this model can be downloaded via:

Download the weights for the `default` model-type
wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth

Once the model is downloaded you can get started by running the segment.image.manual recipe.

When you run this model, you may notice that it’s fairly slow. This isn’t a big suprise given the size of the model but it can be a serious burden, especially if your machine does not have a GPU. For a better experience, you may want to pre-compute the features ahead of annotation time and cache those results to disk. It may take a while to precompute all the images, but once they are done the annotation experience feels seamless and realtime again.

To precompute a cache, you can use the segment.fill-cache recipe.

This will store all the features in a folder (configurable via the --cache flag) which the segment.image.manual recipe can immediately pick up.

The pixel maps, once annotated, are stored under the spans key in your examples. You can explore these maps one by one in a Jupyter notebook using the script shown below.

Script to loop over all annotated examples
import base64
from io import BytesIO
from PIL import Image

from prodigy.components.db import connect

db = connect()
examples = db.get_dataset_examples("<dataset-name>")

def mask_to_pil(mask_str):
    indicator = "base64,"
    mask_str = mask_str[mask_str.find(indicator) + len(indicator):]
    bytes = BytesIO(base64.b64decode(mask_str))
    return Image.open(bytes)

# Loop over all the examples and display them.
for ex in examples:
    print(ex['path'])
    for span in ex.get("spans", []):
        # Use builtin `display` to view pixel map
        display(mask_to_pil(span['mask']))

From here you can re-use the Pillow library to either store these pixel maps into the required format for your pipeline or you can stream them directly into a learning algorithm from Python.

API

Prodigy-ANN

Sometimes you may want to query your examples to find a relevant subset for annotation. A modern method for doing this is to use numeric vectors to represent text and you can use approximate neighest neighbor (ANN) techniques to fetch relevant examples. The goal is to spend more time looking at examples that matter, like examples similar to items that the model gets wrong. Curating these examples first might be a pragmatic method to steer the model in the right direction.

If you’re interested to see a quick demo for Prodigy-ANN applied to a text dataset, you may appreciate this Prodigy short on Youtube.

To use this plugin, you’ll need to install it first.

Install prodigy-ann
pip install "prodigy-ann @ git+https://github.com/explosion/prodigy-ann"

As a first step for this approach you’ll first need to generate an index with vector representations of your text. To encode the text this library uses sentence-transformers and it uses hnswlib as an index for these vectors.

To index your documents, you can run the ann.text.index recipe.

indexing: 100%|███████████████████████████| 2210/2210 [00:09<00:00, 243.64it/s]

Once it is indexed you can use text queries find and curate interesting subsets. A general method to prepare these subsets is to use ann.text.fetch. This will fetch a subset of vectors that are close in vector space and save the associated examples on disk. From there you can use any Prodigy recipe you like.

More interfaces

As a convenience this plugin also provides the textcat.ann.manual, ner.ann.manual and spans.ann.manual so that you may query and annotate directly. These recipes have the same arguments as their native Prodigy textcat.manual, ner.manual and spans.manual counterparts but add a --query parameter so that you may pass your query.

Interactive Queries

Sometimes you may want to update the stream while you’re annotating. You can do that without restarting the server by using the --allow-reset flag when you’re starting the textcat.ann.manual, ner.ann.manual or spans.ann.manual recipes.

Here’s an example of what the experience might look like from the UI.

Retreiving Images

You can use these embedding retreival techniques for images too. Models like CLIP allow you to embed images and text in the same space, which means that you can query the images by using text.

The approach for images is very similar to the approach for text too. To get started you’ll first want to run an indexing recipe over a folder of images via the ann.image.index recipe.

indexing: 100%|███████████████████████████| 210/210 [01:49<00:00]

Once the index is built, you can query it. You can choose to query it to prepare a .jsonl file to re-use later via the ann.image.fetch recipe.

Alternatively the plugin also provides a wrapper around the familiar image.manual recipe. This will retreive the images before passing it on to the image_manual interface. This interface also allows you to reset the stream via the --allow-reset flag.

Here’s an example of what the experience might look like from the UI.

API

Prodigy-Lunr

Instead of using semantic vectors with approximate nearest neighbors to find relevant subsets you can also resort to the “regular” search techniques. To accomodate these techniques we’ve added support for recipes that use lunr. These recipes are very similar to their ann.* counterparts but will rely on string matching techniques to retreive relevant examples.

To use this plugin, you’ll need to install it first.

Install prodigy-lunr
pip install "prodigy-lunr @ git+https://github.com/explosion/prodigy-lunr"

To index your documents, you can run the ann.text.index recipe. This will generate an index and serialize it to disk by writing it into a gzipped json file.

indexing: 100%|███████████████████████████| 2210/2210 [00:09<00:00, 243.64it/s]

Once it is indexed you can use text queries find and curate interesting subsets. A general method to prepare these subsets is to use lunr.text.fetch. This will fetch a subset of vectors that are close in vector space and save the associated examples on disk. From there you can use any Prodigy recipe you like.

More interfaces

As a convenience this plugin also provides the textcat.lunr.manual, ner.lunr.manual and spans.lunr.manual so that you may query and annotate directly. These recipes have the same arguments as their native Prodigy textcat.manual, ner.manual and spans.manual counterparts but add a --query parameter so that you may pass your query.

Interactive Queries

Sometimes you may want to update the stream while you’re annotating. You can do that without restarting the server by using the --allow-reset flag when you’re starting the textcat.lunr.manual, ner.lunr.manual or spans.lunr.manual recipes.

API

Sense2vec

sense2vec (Trask et. al, 2015) is a nice twist on word2vec that lets you learn more interesting and detailed word vectors. This library is a simple Python implementation for loading, querying and training sense2vec models. To explore the semantic similarities across all Reddit comments of 2015 and 2019, see the interactive demo. There are also more details in this blogpost.

To see a demo on how to use this tool with Prodigy, you may enjoy this Youtube video where we use it to detect video games in text.

To use sense2vec, you’ll first need to install it.

python -m pip install sense2vec

To use the pre-trained vectors in Prodigy you’ll need to download the archive(s) and extract them. Large files have been split into multi-part downloads. All the available versions can be found below.

To merge the multi-part archives, you can run the following:

cat s2v_reddit_2019_lg.tar.gz.* > s2v_reddit_2019_lg.tar.gz

Once downloaded (and merged) you should be able to unarchive via:

tar -xvf s2v_reddit_lg.tar.gz

Now that the archive is extracted you can point the sense2vec.teach recipe to it. This will allow Prodigy to suggest similar terms based on the most similar phrases from sense2vec, and the suggestions will be adjusted as you annotate and accept similar phrases. For each seed term, the best matching sense according to the sense2vec vectors will be used.

After curating the generated examples you can choose to export the collected phrases as pattern files which can be used with spaCy’s EntityRuler or recipes like ner.manual by using the sense2vec.to-patterns recipe.

This will generate a patterns.jsonl file locally that has contents that may look like:

{"label": "VIDEO_GAME", "pattern": [{"LOWER": "mass"}, {"LOWER": "effect"}]}
{"label": "VIDEO_GAME", "pattern": [{"LOWER": "knights"}, {"LOWER": "of"}, {"LOWER": "the"}, {"LOWER": "old"}, {"LOWER": "republic"}]}
{"label": "VIDEO_GAME", "pattern": [{"LOWER": "halo"}, {"LOWER": "3"}]}
{"label": "VIDEO_GAME", "pattern": [{"LOWER": "jade"}, {"LOWER": "empire"}]}

More recipes

Sense2vec also has the sense2vec.eval, sense2vec.eval-most-similar and sense2vec.eval-ab recipes. These may be interesting if you’re interested in evaluating a sense2vec model. For more information on those, you can check the README on the Github repository.

🔎 Prodigy-evaluate

This Prodigy plugin allows you to evaluate your spaCy pipeline overall or on a per-example basis. To use these recipes, you’ll first need to install the plugin.

Install prodigy-evaluate
pip install "prodigy-evaluate @ git+https://github.com/explosion/prodigy-evaluate"

Once installed, you can make use of the two main recipes in this plugin: evaluate.evaluate and evaluate.evaluate-example.

TOK     100.00
NER P   92.80
NER R   99.58
NER F   96.07
SPEED   26868

                 P        R       F
SKILL        92.53    99.55   95.91
EXPERIENCE   96.88   100.00   98.41

Example                     ents_f
-----------------           ------
I live in london.           0.0
My name is Freya.           0.0
Where is Antonia?           0.0