gliner.data_processing.collator module

class gliner.data_processing.collator.BaseDataCollator(config, data_processor=None, return_tokens=False, return_id_to_classes=False, return_entities=False, prepare_labels=True)

Bases: ABC

Abstract base class for all data collators.

Provides common functionality for collating batches and preparing model inputs. Subclasses should implement processor-specific logic and field handling.

Initialize the base data collator.

Parameters:

• config – Configuration object containing model/training parameters.

• data_processor (BaseProcessor | None) – Processor instance for handling data transformations. If None, subclass should provide a default processor.

• return_tokens (bool) – Whether to include tokenized text in output.

• return_id_to_classes (bool) – Whether to include class ID to name mappings.

• return_entities (bool) – Whether to include entity annotations.

• prepare_labels (bool) – Whether to prepare labels for training.

__init__(config, data_processor=None, return_tokens=False, return_id_to_classes=False, return_entities=False, prepare_labels=True)

Initialize the base data collator.

Parameters:

• config – Configuration object containing model/training parameters.

• data_processor (BaseProcessor | None) – Processor instance for handling data transformations. If None, subclass should provide a default processor.

• return_tokens (bool) – Whether to include tokenized text in output.

• return_id_to_classes (bool) – Whether to include class ID to name mappings.

• return_entities (bool) – Whether to include entity annotations.

• prepare_labels (bool) – Whether to prepare labels for training.

collate_batch(input_x, **kwargs)

Collate raw input examples into a batch.

Parameters:

• input_x (List[Dict[str, Any]]) – List of raw input examples.

• **kwargs – Additional arguments passed to the processor’s collate_raw_batch.

Returns:

Dict containing collated raw batch data.

Return type:

Dict[str, Any]

collate_function(raw_batch, **kwargs)

Transform raw batch into model input format.

Parameters:

• raw_batch (Dict[str, Any]) – Raw collated batch from collate_batch.

• **kwargs – Additional arguments passed to the processor’s collate_fn.

Returns:

Dict containing model-ready inputs.

Return type:

Dict[str, Any]

class gliner.data_processing.collator.BaseSpanCollator(config, data_processor=None, return_tokens=False, return_id_to_classes=False, return_entities=False, prepare_labels=True)

Bases: BaseDataCollator

Base collator for span-based processors.

Provides common logic for handling span indices, span masks, and span labels. Used by all span-level NER/RE models.

Initialize the base data collator.

Parameters:

• config – Configuration object containing model/training parameters.

• data_processor (BaseProcessor | None) – Processor instance for handling data transformations. If None, subclass should provide a default processor.

• return_tokens (bool) – Whether to include tokenized text in output.

• return_id_to_classes (bool) – Whether to include class ID to name mappings.

• return_entities (bool) – Whether to include entity annotations.

• prepare_labels (bool) – Whether to prepare labels for training.

class gliner.data_processing.collator.BaseTokenCollator(config, data_processor=None, return_tokens=False, return_id_to_classes=False, return_entities=False, prepare_labels=True)

Bases: BaseDataCollator

Base collator for token-based processors.

Provides common logic for handling token-level annotations and entity IDs. Used by all token-level NER models.

Initialize the base data collator.

Parameters:

• config – Configuration object containing model/training parameters.

• data_processor (BaseProcessor | None) – Processor instance for handling data transformations. If None, subclass should provide a default processor.

• return_tokens (bool) – Whether to include tokenized text in output.

• return_id_to_classes (bool) – Whether to include class ID to name mappings.

• return_entities (bool) – Whether to include entity annotations.

• prepare_labels (bool) – Whether to prepare labels for training.

class gliner.data_processing.collator.SpanDataCollator(config, data_processor=None, return_tokens=False, return_id_to_classes=False, return_entities=False, prepare_labels=True, prepare_entities=True)

Bases: BaseSpanCollator

Unified data collator for all span-based processors.

Handles span-based NER with various architectures: - UniEncoder: Single encoder with span classification - BiEncoder: Separate encoders for text and entity types - EncoderDecoder: Generative entity typing with decoder

Automatically adapts behavior based on processor type.

Required Processors: UniEncoderSpanProcessor, BiEncoderSpanProcessor,

or UniEncoderSpanDecoderProcessor

Initialize unified span collator.

Parameters:

• config – Configuration object.

• data_processor (UniEncoderSpanProcessor | BiEncoderSpanProcessor | UniEncoderSpanDecoderProcessor | None) – Span processor instance (Uni/Bi/EncoderDecoder).

• return_tokens (bool) – Whether to return tokenized text.

• return_id_to_classes (bool) – Whether to return class mappings.

• return_entities (bool) – Whether to return entity annotations.

• prepare_labels (bool) – Whether to prepare training labels.

• prepare_entities (bool) – Whether to encode entity types (BiEncoder only).

__init__(config, data_processor=None, return_tokens=False, return_id_to_classes=False, return_entities=False, prepare_labels=True, prepare_entities=True)

Initialize unified span collator.

Parameters:

• config – Configuration object.

• data_processor (UniEncoderSpanProcessor | BiEncoderSpanProcessor | UniEncoderSpanDecoderProcessor | None) – Span processor instance (Uni/Bi/EncoderDecoder).

• return_tokens (bool) – Whether to return tokenized text.

• return_id_to_classes (bool) – Whether to return class mappings.

• return_entities (bool) – Whether to return entity annotations.

• prepare_labels (bool) – Whether to prepare training labels.

• prepare_entities (bool) – Whether to encode entity types (BiEncoder only).

class gliner.data_processing.collator.TokenDataCollator(config, data_processor=None, return_tokens=False, return_id_to_classes=False, return_entities=False, prepare_labels=True, prepare_entities=True)

Bases: BaseTokenCollator

Unified data collator for all token-based processors.

Handles token-level NER with various architectures: - UniEncoder: Single encoder with BIO/BIOES tagging - BiEncoder: Separate encoders for text and entity types

Automatically adapts behavior based on processor type.

Required Processors: UniEncoderTokenProcessor or BiEncoderTokenProcessor

Initialize unified token collator.

Parameters:

• config – Configuration object.

• data_processor (UniEncoderTokenProcessor | BiEncoderTokenProcessor | None) – Token processor instance (Uni/Bi).

• return_tokens (bool) – Whether to return tokenized text.

• return_id_to_classes (bool) – Whether to return class mappings.

• return_entities (bool) – Whether to return entity annotations.

• prepare_labels (bool) – Whether to prepare training labels.

• prepare_entities (bool) – Whether to encode entity types (BiEncoder only).

__init__(config, data_processor=None, return_tokens=False, return_id_to_classes=False, return_entities=False, prepare_labels=True, prepare_entities=True)

Initialize unified token collator.

Parameters:

• config – Configuration object.

• data_processor (UniEncoderTokenProcessor | BiEncoderTokenProcessor | None) – Token processor instance (Uni/Bi).

• return_tokens (bool) – Whether to return tokenized text.

• return_id_to_classes (bool) – Whether to return class mappings.

• return_entities (bool) – Whether to return entity annotations.

• prepare_labels (bool) – Whether to prepare training labels.

• prepare_entities (bool) – Whether to encode entity types (BiEncoder only).

class gliner.data_processing.collator.RelationExtractionSpanDataCollator(config, data_processor=None, return_tokens=False, return_id_to_classes=False, return_entities=False, return_rel_id_to_classes=False, return_relations=False, prepare_labels=True)

Bases: BaseSpanCollator

Data collator for RelationExtractionSpanProcessor.

Handles joint entity and relation extraction at span level. Produces both entity labels and relation adjacency matrices.

This collator is kept separate due to its unique handling of: - Relation adjacency matrices - Dual classification (entities + relations) - Relation-specific configuration

Required Processor: RelationExtractionSpanProcessor

Initialize RelationExtraction span collator.

Parameters:

• config – Configuration object.

• data_processor (RelationExtractionSpanProcessor | None) – RelationExtractionSpanProcessor instance.

• return_tokens (bool) – Whether to return tokenized text.

• return_id_to_classes (bool) – Whether to return entity class mappings.

• return_entities (bool) – Whether to return entity annotations.

• return_rel_id_to_classes (bool) – Whether to return relation class mappings.

• return_relations (bool) – Whether to return relation annotations.

• prepare_labels (bool) – Whether to prepare training labels.

__init__(config, data_processor=None, return_tokens=False, return_id_to_classes=False, return_entities=False, return_rel_id_to_classes=False, return_relations=False, prepare_labels=True)

Initialize RelationExtraction span collator.

Parameters:

• config – Configuration object.

• data_processor (RelationExtractionSpanProcessor | None) – RelationExtractionSpanProcessor instance.

• return_tokens (bool) – Whether to return tokenized text.

• return_id_to_classes (bool) – Whether to return entity class mappings.

• return_entities (bool) – Whether to return entity annotations.

• return_rel_id_to_classes (bool) – Whether to return relation class mappings.

• return_relations (bool) – Whether to return relation annotations.

• prepare_labels (bool) – Whether to prepare training labels.

collate_batch(input_x, entity_types=None, relation_types=None, ner_negatives=None, rel_negatives=None, **kwargs)

Collate raw batch data for relation extraction.

Parameters:

• input_x (List[Dict[str, Any]]) – List of input examples.

• entity_types (List[str] | List[List[str]] | None) – Optional entity type specifications.

• relation_types (List[str] | List[List[str]] | None) – Optional relation type specifications.

• ner_negatives (List[str] | None) – Optional negative entity types for sampling.

• rel_negatives (List[str] | None) – Optional negative relation types for sampling.

• **kwargs – Additional arguments.

Returns:

Collated raw batch with entity and relation information.

Return type:

Dict[str, Any]

class gliner.data_processing.collator.UniEncoderSpanDataCollator(config, data_processor=None, return_tokens=False, return_id_to_classes=False, return_entities=False, prepare_labels=True, prepare_entities=True)

Bases: SpanDataCollator

Backward compatibility alias for SpanDataCollator with UniEncoderSpanProcessor.

Use SpanDataCollator directly for new code.

Initialize unified span collator.

Parameters:

• config – Configuration object.

• data_processor (UniEncoderSpanProcessor | BiEncoderSpanProcessor | UniEncoderSpanDecoderProcessor | None) – Span processor instance (Uni/Bi/EncoderDecoder).

• return_tokens (bool) – Whether to return tokenized text.

• return_id_to_classes (bool) – Whether to return class mappings.

• return_entities (bool) – Whether to return entity annotations.

• prepare_labels (bool) – Whether to prepare training labels.

• prepare_entities (bool) – Whether to encode entity types (BiEncoder only).

class gliner.data_processing.collator.BiEncoderSpanDataCollator(config, data_processor=None, return_tokens=False, return_id_to_classes=False, return_entities=False, prepare_labels=True, prepare_entities=True)

Bases: SpanDataCollator

Backward compatibility alias for SpanDataCollator with BiEncoderSpanProcessor.

Use SpanDataCollator directly for new code.

Initialize unified span collator.

Parameters:

• config – Configuration object.

• data_processor (UniEncoderSpanProcessor | BiEncoderSpanProcessor | UniEncoderSpanDecoderProcessor | None) – Span processor instance (Uni/Bi/EncoderDecoder).

• return_tokens (bool) – Whether to return tokenized text.

• return_id_to_classes (bool) – Whether to return class mappings.

• return_entities (bool) – Whether to return entity annotations.

• prepare_labels (bool) – Whether to prepare training labels.

• prepare_entities (bool) – Whether to encode entity types (BiEncoder only).

class gliner.data_processing.collator.UniEncoderSpanDecoderDataCollator(config, data_processor=None, return_tokens=False, return_id_to_classes=False, return_entities=False, prepare_labels=True, prepare_entities=True)

Bases: SpanDataCollator

Backward compatibility alias for SpanDataCollator with EncoderDecoderSpanProcessor.

Use SpanDataCollator directly for new code.

Initialize unified span collator.

Parameters:

• config – Configuration object.

• data_processor (UniEncoderSpanProcessor | BiEncoderSpanProcessor | UniEncoderSpanDecoderProcessor | None) – Span processor instance (Uni/Bi/EncoderDecoder).

• return_tokens (bool) – Whether to return tokenized text.

• return_id_to_classes (bool) – Whether to return class mappings.

• return_entities (bool) – Whether to return entity annotations.

• prepare_labels (bool) – Whether to prepare training labels.

• prepare_entities (bool) – Whether to encode entity types (BiEncoder only).

class gliner.data_processing.collator.UniEncoderTokenDataCollator(config, data_processor=None, return_tokens=False, return_id_to_classes=False, return_entities=False, prepare_labels=True, prepare_entities=True)

Bases: TokenDataCollator

Backward compatibility alias for TokenDataCollator with UniEncoderTokenProcessor.

Use TokenDataCollator directly for new code.

Initialize unified token collator.

Parameters:

• config – Configuration object.

• data_processor (UniEncoderTokenProcessor | BiEncoderTokenProcessor | None) – Token processor instance (Uni/Bi).

• return_tokens (bool) – Whether to return tokenized text.

• return_id_to_classes (bool) – Whether to return class mappings.

• return_entities (bool) – Whether to return entity annotations.

• prepare_labels (bool) – Whether to prepare training labels.

• prepare_entities (bool) – Whether to encode entity types (BiEncoder only).

class gliner.data_processing.collator.BiEncoderTokenDataCollator(config, data_processor=None, return_tokens=False, return_id_to_classes=False, return_entities=False, prepare_labels=True, prepare_entities=True)

Bases: TokenDataCollator

Backward compatibility alias for TokenDataCollator with BiEncoderTokenProcessor.

Use TokenDataCollator directly for new code.

Initialize unified token collator.

Parameters:

• config – Configuration object.

• data_processor (UniEncoderTokenProcessor | BiEncoderTokenProcessor | None) – Token processor instance (Uni/Bi).

• return_tokens (bool) – Whether to return tokenized text.

• return_id_to_classes (bool) – Whether to return class mappings.

• return_entities (bool) – Whether to return entity annotations.

• prepare_labels (bool) – Whether to prepare training labels.

• prepare_entities (bool) – Whether to encode entity types (BiEncoder only).