ds_provider_xledger_py_lib.dataset.engines

File: __init__.py Region: ds_provider_xledger_py_lib/dataset/engines

Description

Dataset execution engines.

Submodules

• ds_provider_xledger_py_lib.dataset.engines._read_checkpoint
• ds_provider_xledger_py_lib.dataset.engines._read_incremental
• ds_provider_xledger_py_lib.dataset.engines.read

Classes

ReadEngine

Execute Xledger read requests including pagination handling.

Package Contents

class ds_provider_xledger_py_lib.dataset.engines.ReadEngine

Execute Xledger read requests including pagination handling.

This engine is intentionally stateful: - output accumulates collected rows page-by-page - checkpoint tracks the last valid pagination state

connection: ds_protocol_http_py_lib.utils.http.provider.Http

host: str

deserializer: ds_provider_xledger_py_lib.serde.deserializer.XledgerDeserializer

metadata: ds_provider_xledger_py_lib.utils.introspection.MetaData

output: list[pandas.DataFrame] = []

checkpoint: ds_provider_xledger_py_lib.dataset.engines._read_checkpoint.Checkpoint

execute(*, read_settings: ds_provider_xledger_py_lib.dataset.xledger.XledgerReadSettings, checkpoint: dict[str, Any] | None = None) → None

Execute read flow and update output/checkpoint state.

Parameters:

• read_settings – Effective read settings for query rendering/pagination.

• checkpoint – Existing checkpoint state to continue from.

_resolve_effective_read_settings(*, read_settings: ds_provider_xledger_py_lib.dataset.xledger.XledgerReadSettings, incremental: ds_provider_xledger_py_lib.utils.introspection.IncrementalMetaData | None) → ds_provider_xledger_py_lib.dataset.xledger.XledgerReadSettings

Resolve effective read settings before the request loop.

Parameters:

• read_settings – Caller-provided read settings.

• incremental – Incremental section from read metadata, when configured.

Returns:

Read settings updated with resume cursor and checkpoint boundary filter.

_build_payload(*, read_settings: ds_provider_xledger_py_lib.dataset.xledger.XledgerReadSettings, query_fields: list[str], request_first: int) → dict[str, Any]

Build the GraphQL payload for one read request.

Parameters:

• read_settings – Effective read settings for the current request.

• query_fields – Effective query fields for the read request.

• request_first – Resolved page size for this request.

Returns:

GraphQL payload ready to send through the linked service connection.

_should_continue_pagination(*, has_next_page: bool, end_cursor: str | None) → bool

Return whether the engine should request the next page.

Parameters:

• has_next_page – Provider-reported pagination state.

• end_cursor – Provider-reported end cursor for the current page.

Returns:

True when pagination should continue, otherwise False.

_configured_page_size(*, read_settings: ds_provider_xledger_py_lib.dataset.xledger.XledgerReadSettings) → int

Resolve the configured page size from read settings or metadata.

Parameters:

read_settings – Effective read settings for the current request.

Returns:

The configured page size to request from the source for the next page.

Raises:

RuntimeError – If neither settings nor metadata provide a page size.

_resolve_request_first(*, read_settings: ds_provider_xledger_py_lib.dataset.xledger.XledgerReadSettings, total_rows: int) → int

Resolve the request page size for the next read request.

Parameters:

• read_settings – Effective read settings for the current request.

• total_rows – Number of rows already collected in the current read.

Returns:

Page size to request from the source for the next page.

Raises:

RuntimeError – If neither settings nor metadata provide a page size.

_merge_read_checkpoint(*, incremental: ds_provider_xledger_py_lib.utils.introspection.IncrementalMetaData | None, observed_incremental_values: list[Any], completed_scope: bool) → None

Merge read results into checkpoint state after the read loop.

Pagination is cleared only when completed_scope is true (the configured scope was fully exhausted). Partial runs (row limit, missing cursor, repeated cursor) leave pagination and incremental state as set during the loop.

Parameters:

• incremental – Incremental metadata from read metadata, when configured.

• observed_incremental_values – List of observed incremental values from the current read.

• completed_scope – Whether the read completed the configured scope.