gliner.modeling.encoder module

class gliner.modeling.encoder.Transformer(model_name, config, from_pretrained=False, labels_encoder=False, cache_dir=None)

Bases: Module

Flexible transformer wrapper supporting multiple architectures and configurations.

This class provides a unified interface for various transformer models including encoder-only (BERT, DeBERTa), encoder-decoder (T5), and decoder-only models (LLaMA, Mistral) with bidirectional adaptations. It handles model initialization, adapter loading, and specialized forward passes for different architectures.

model

The underlying transformer model instance.

layers_fuser

Optional layer fusion module when config.fuse_layers is True.

config

Configuration object containing model hyperparameters.

Initializes the transformer wrapper.

Parameters:

• model_name (str) – Name or path of the pretrained model to load.

• config (Any) – Configuration object containing model hyperparameters. Must have attributes like encoder_config, labels_encoder_config, vocab_size, _attn_implementation, and fuse_layers.

• from_pretrained (bool) – If True, loads pretrained weights. If False, initializes from config only. Defaults to False.

• labels_encoder (bool) – If True, initializes as a labels encoder using config.labels_encoder_config. Defaults to False.

• cache_dir (str | Path | None) – Optional directory for caching downloaded models. Defaults to None.

Raises:

MissedPackageException – If required packages (llm2vec, peft) are not installed when needed for specific model types.

__init__(model_name, config, from_pretrained=False, labels_encoder=False, cache_dir=None)

Initializes the transformer wrapper.

Parameters:

• model_name (str) – Name or path of the pretrained model to load.

• config (Any) – Configuration object containing model hyperparameters. Must have attributes like encoder_config, labels_encoder_config, vocab_size, _attn_implementation, and fuse_layers.

• from_pretrained (bool) – If True, loads pretrained weights. If False, initializes from config only. Defaults to False.

• labels_encoder (bool) – If True, initializes as a labels encoder using config.labels_encoder_config. Defaults to False.

• cache_dir (str | Path | None) – Optional directory for caching downloaded models. Defaults to None.

Raises:

MissedPackageException – If required packages (llm2vec, peft) are not installed when needed for specific model types.

forward(*args, **kwargs)

Forward pass through the transformer model.

Handles different attention mask configurations and model architectures, including support for pair attention masks for packed sequences.

Parameters:

• *args (Any) – Variable positional arguments passed to the model.

• **kwargs (Any) –

  Variable keyword arguments. Special arguments include: - pair_attention_mask: Optional pairwise attention mask of shape

  (batch_size, seq_len, seq_len) for packed sequences.

  • attention_mask: Standard attention mask of shape (batch_size, seq_len).

  • input_ids: Input token IDs of shape (batch_size, seq_len).

  • Other model-specific arguments.

Returns:

Encoded representations of shape (batch_size, seq_len, hidden_size). If config.fuse_layers is True, returns fused layer outputs, otherwise returns the last hidden state.

Return type:

Tensor

class gliner.modeling.encoder.Encoder(config, from_pretrained=False, cache_dir=None)

Bases: Module

Standard encoder module wrapping a transformer model with optional projection.

This class provides a high-level interface for encoding text sequences, including support for inference-time packing to improve throughput. It handles embedding extraction and optional projection to a different hidden size.

bert_layer

The underlying Transformer instance.

projection

Optional linear projection layer when config.hidden_size differs from the model’s native hidden size.

Initializes the encoder.

Parameters:

• config (Any) – Configuration object containing model hyperparameters including model_name, hidden_size, and transformer-specific settings.

• from_pretrained (bool) – If True, loads pretrained weights for the transformer. Defaults to False.

• cache_dir (str | Path | None) – Optional directory for caching downloaded models. Defaults to None.

__init__(config, from_pretrained=False, cache_dir=None)

Initializes the encoder.

Parameters:

• config (Any) – Configuration object containing model hyperparameters including model_name, hidden_size, and transformer-specific settings.

• from_pretrained (bool) – If True, loads pretrained weights for the transformer. Defaults to False.

• cache_dir (str | Path | None) – Optional directory for caching downloaded models. Defaults to None.

resize_token_embeddings(new_num_tokens, pad_to_multiple_of=None)

Resizes token embeddings to accommodate new vocabulary size.

Parameters:

• new_num_tokens (int) – New vocabulary size.

• pad_to_multiple_of (int | None) – Optional value to pad vocabulary size to a multiple. Defaults to None.

Returns:

The resized embedding layer.

Return type:

Embedding

get_input_embeddings()

Gets the input embedding layer.

Returns:

The model’s input embedding layer.

Return type:

Embedding

encode_text(input_ids, attention_mask, *args, **kwargs)

Encodes input text sequences into contextualized embeddings.

Supports inference-time packing to batch multiple variable-length sequences efficiently when packing_config is provided and not in training mode.

Parameters:

• input_ids (Tensor) – Input token IDs of shape (batch_size, seq_len).

• attention_mask (Tensor) – Attention mask of shape (batch_size, seq_len) where 1 indicates valid tokens and 0 indicates padding.

• *args (Any) – Additional positional arguments passed to the transformer.

• **kwargs (Any) – Additional keyword arguments including: - packing_config: Optional InferencePackingConfig for efficient batching. - pair_attention_mask: Optional pairwise attention mask for packed sequences.

Returns:

Token embeddings of shape (batch_size, seq_len, hidden_size).

Return type:

Tensor

forward(*args, **kwargs)

Forward pass through the encoder.

Parameters:

• *args (Any) – Positional arguments passed to encode_text.

• **kwargs (Any) – Keyword arguments passed to encode_text.

Returns:

Token embeddings of shape (batch_size, seq_len, hidden_size).

Return type:

Tensor

class gliner.modeling.encoder.BiEncoder(config, from_pretrained=False, cache_dir=None)

Bases: Encoder

Bi-encoder architecture with separate encoders for text and labels.

This encoder processes text sequences and label sequences through potentially different transformer models, producing aligned representations for both. The label representations are mean-pooled to create fixed-size embeddings.

bert_layer

Inherited text encoder from Encoder.

projection

Inherited optional projection from Encoder.

labels_encoder

Separate Transformer instance for encoding labels.

labels_projection

Optional projection for label embeddings when label encoder hidden size differs from config.hidden_size.

Initializes the bi-encoder.

Parameters:

• config (Any) – Configuration object containing model hyperparameters including labels_encoder (model name for label encoder) and hidden_size.

• from_pretrained (bool) – If True, loads pretrained weights for both encoders. Defaults to False.

• cache_dir (str | Path | None) – Optional directory for caching downloaded models. Defaults to None.

__init__(config, from_pretrained=False, cache_dir=None)

Initializes the bi-encoder.

Parameters:

• config (Any) – Configuration object containing model hyperparameters including labels_encoder (model name for label encoder) and hidden_size.

• from_pretrained (bool) – If True, loads pretrained weights for both encoders. Defaults to False.

• cache_dir (str | Path | None) – Optional directory for caching downloaded models. Defaults to None.

mean_pooling(token_embeddings, attention_mask)

Applies mean pooling over token embeddings using attention mask.

Computes the average of token embeddings weighted by the attention mask, ignoring padded positions.

Parameters:

• token_embeddings (Tensor) – Token-level embeddings of shape (batch_size, seq_len, hidden_size).

• attention_mask (Tensor) – Binary mask of shape (batch_size, seq_len) where 1 indicates valid tokens and 0 indicates padding.

Returns:

Pooled embeddings of shape (batch_size, hidden_size).

Return type:

Tensor

encode_labels(input_ids, attention_mask, *args, **kwargs)

Encodes label sequences into fixed-size embeddings.

Processes labels through the dedicated labels encoder and applies mean pooling to produce sentence-level representations.

Parameters:

• input_ids (Tensor) – Label token IDs of shape (batch_size, seq_len).

• attention_mask (Tensor) – Attention mask of shape (batch_size, seq_len).

• *args (Any) – Additional positional arguments.

• **kwargs (Any) – Additional keyword arguments (packing_config and pair_attention_mask are removed as they’re not supported for labels).

Returns:

Pooled label embeddings of shape (batch_size, hidden_size).

Return type:

Tensor

forward(input_ids, attention_mask, labels_input_ids=None, labels_attention_mask=None, *args, **kwargs)

Forward pass through the bi-encoder.

Encodes both text sequences (token-level) and label sequences (pooled) to produce aligned representations.

Parameters:

• input_ids (Tensor) – Text token IDs of shape (batch_size, seq_len).

• attention_mask (Tensor) – Text attention mask of shape (batch_size, seq_len).

• labels_input_ids (Tensor | None) – Label token IDs of shape (batch_size, label_seq_len).

• labels_attention_mask (Tensor | None) – Label attention mask of shape (batch_size, label_seq_len).

• *args (Any) – Additional positional arguments.

• **kwargs (Any) – Additional keyword arguments.

Returns:

• token_embeddings: Text embeddings of shape (batch_size, seq_len, hidden_size).

• labels_embeddings: Pooled label embeddings of shape (batch_size, hidden_size).

Return type:

A tuple containing