gliner.modeling.cache module

Session cache support for decoder-backed span inference.

class gliner.modeling.cache.ReusableDynamicLayer(initial_capacity=256, max_cache_length=None)

Bases: DynamicLayer

Append-only KV storage that grows geometrically instead of every turn.

__init__(initial_capacity=256, max_cache_length=None)

lazy_initialization(key_states, value_states)

update(key_states, value_states, *args, **kwargs)

Update the key and value caches in-place, and return the necessary keys and value states.

Parameters:

• key_states (torch.Tensor) – The new key states to cache.

• value_states (torch.Tensor) – The new value states to cache.

• cache_kwargs (dict[str, Any], optional) – Additional arguments for the cache.

Returns:

The key and value states.

Return type:

tuple[torch.Tensor, torch.Tensor]

get_seq_length()

Returns the sequence length of the cached states.

get_max_cache_shape()

Returns the maximum sequence length of the cache object. DynamicLayer does not have a maximum length.

property max_batch_size: int

property max_cache_len: int

reset()

Resets the cache values while preserving the objects

crop(max_length)

Crop the past key values up to a new max_length in terms of tokens. max_length can also be negative to remove max_length tokens.

batch_repeat_interleave(repeats)

Repeat the cache repeats times in the batch dimension.

batch_select_indices(indices)

Only keep the indices in the batch dimension of the cache.

move_to(device)

class gliner.modeling.cache.ReusableDynamicCache(config=None, initial_capacity=256, max_cache_length=None)

Bases: DynamicCache

DynamicCache with stable, capacity-backed tensors when supported.

__init__(config=None, initial_capacity=256, max_cache_length=None)

move_to(device)

gliner.modeling.cache.create_reusable_cache(config, *, initial_capacity=256, max_cache_length=None)

Create a session KV cache whose storage is reused between appends.

class gliner.modeling.cache.CacheState(past_key_values=None, input_ids=<factory>, attention_mask=<factory>, token_word_mask=<factory>, past_word_embeddings=None, past_word_mask=None, prompts_embedding=None, prompts_mask=None, cached_length=0, next_position_id=None, session_id=None, labels=(), text='', tokens=<factory>, char_starts=<factory>, char_ends=<factory>, span_logits=<factory>, metadata=<factory>)

Bases: object

All reusable state belonging to one streaming-span session.

past_key_values: Any = None

input_ids: torch.Tensor

attention_mask: torch.Tensor

token_word_mask: torch.Tensor

past_word_embeddings: torch.Tensor | None = None

past_word_mask: torch.Tensor | None = None

prompts_embedding: torch.Tensor | None = None

prompts_mask: torch.Tensor | None = None

cached_length: int = 0

next_position_id: int | None = None

session_id: str | None = None

labels: tuple[str, ...] = ()

text: str = ''

tokens: list[str]

char_starts: list[int]

char_ends: list[int]

span_logits: dict[tuple[int, int], torch.Tensor]

metadata: dict[str, Any]

property next_position: int

Return the absolute decoder position for the next token.

property word_length: int

Return the authoritative cached word count without a device sync.

to(device)

Move reusable model tensors only when the target device changes.

__init__(past_key_values=None, input_ids=<factory>, attention_mask=<factory>, token_word_mask=<factory>, past_word_embeddings=None, past_word_mask=None, prompts_embedding=None, prompts_mask=None, cached_length=0, next_position_id=None, session_id=None, labels=(), text='', tokens=<factory>, char_starts=<factory>, char_ends=<factory>, span_logits=<factory>, metadata=<factory>)

class gliner.modeling.cache.SessionCacheManager

Bases: object

Thread-safe ownership and lifecycle management for session states.

__init__()

get(session_id)

put(state)

clear(session_ids=None)

gliner.modeling.cache.copy_past_key_values(past_kv, *, config=None, max_cache_length=None)

Return an independent copy of a transformers-style KV cache.

Streaming cache layers append in place. Tests, transactional helpers, and dynamic batching therefore need an explicit copy operation rather than a shallow copy.copy of the cache container.

gliner.modeling.cache.stack_past_key_values(caches, *, config=None, max_cache_length=None)

Stack equal-length, batch-size-one caches along their batch dimension.

gliner.modeling.cache.split_past_key_values(past_kv, lengths, *, config=None, max_cache_length=None)

Split a batched cache into compact batch-size-one session caches.