How do I perform entity extraction with Haystack?

To perform entity extraction with Haystack, you’ll use its pipeline-based architecture to process text and identify entities like names, dates, or locations. Haystack provides prebuilt components and integration with popular NLP models, allowing you to build a workflow tailored to your data. The process typically involves initializing a model (like a transformer-based NER model), setting up a preprocessing step, and configuring a pipeline to extract and return structured results.

First, install Haystack and its dependencies using pip install farm-haystack. Next, choose a model for Named Entity Recognition (NER). For example, the dslim/bert-base-NER model from Hugging Face detects four entity types: locations (LOC), organizations (ORG), persons (PER), and miscellaneous (MISC). Initialize the model using Haystack’s TransformersNERPredictor or a similar component. You’ll also need a PreProcessor to split large documents into manageable chunks, ensuring the model’s token limit isn’t exceeded. Configure a pipeline with these components, connecting them in sequence: preprocessing → inference → entity extraction.

Here’s a simplified example:

from haystack import Pipeline
from haystack.nodes import TransformersNERPredictor, PreProcessor

# Initialize components
processor = PreProcessor(split_by="sentence")
model = TransformersNERPredictor(model_name="dslim/bert-base-NER")

# Build pipeline
pipeline = Pipeline()
pipeline.add_node(component=processor, name="preprocessor", inputs=["File"])
pipeline.add_node(component=model, name="ner_model", inputs=["preprocessor"])

# Run extraction
results = pipeline.run(file_paths=["document.txt"])
entities = results["results"]

This code processes a text file, splits it into sentences, and identifies entities in each chunk. The output is a list of entities with their types and positions in the text.

When implementing this, consider model limitations and data preprocessing. For example, transformer models have token limits (often 512 tokens), so splitting documents is critical. Adjust the PreProcessor’s split_length or split_overlap to avoid cutting entities in half. You might also post-process results to merge entities split across chunks or filter low-confidence predictions. If your use case requires custom entity types (like product codes), you’ll need to fine-tune a model or use a rule-based approach with regex in combination with NER. Haystack’s flexibility allows mixing these methods in a single pipeline.