gliner.streaming module

Synchronous fixed-batch and asynchronous streaming inference surfaces.

class gliner.streaming.AsyncStreamingEngine(model, *, max_batch_size=32, batch_wait_timeout_ms=2.0, queue_capacity=4096)

Bases: object

Dynamically microbatch independently arriving streaming sessions.

__init__(model, *, max_batch_size=32, batch_wait_timeout_ms=2.0, queue_capacity=4096)

async start()

Start the scheduler lazily on the current event loop.

async append(session_id, text, labels, *, threshold=0.5, flat_ner=True, multi_label=False, return_class_probs=False, recompute=False)

Queue one append; concurrent callers are dynamically microbatched.

async stream(session_id, chunks, labels, **kwargs)

Yield a session snapshot for every chunk from sync or async input.

async clear_session(session_id)

Wait for prior work on a session, then discard its cache.

async close()

Drain queued work and stop the scheduler. Idempotent.

class gliner.streaming.StreamingBatch(model, session_ids, labels)

Bases: object

Persistent fixed-order batch for synchronous streaming inference.

The session-to-row mapping is immutable, allowing the decoder KV cache to remain batched between calls without stacking or splitting historical keys.

__init__(model, session_ids, labels)

property closed: bool

property initialized: bool

append(texts, *, threshold=0.5, flat_ner=True, multi_label=False, return_class_probs=False, recompute=False)

Append one text chunk per fixed session row and return snapshots.

reset()

Discard the entire persistent cache while retaining row ownership.

close()

Release the persistent batch cache. Idempotent.