ds_provider_mock_py_lib.dataset.engines.read

File: read.py Region: ds_provider_mock_py_lib/dataset/engines

Read engine for paginated mock execution.

This engine is intentionally stateful: - output accumulates collected rows page-by-page - checkpoint tracks incremental watermark and pagination resume state - metadata records batch, page, and op counts

Example

>>> from ds_provider_mock_py_lib.dataset.engines.read import ReadEngine
>>> from ds_provider_mock_py_lib.dataset.settings import MockDatasetSettings
>>> from ds_provider_mock_py_lib.linked_service import MockBackend, MockLinkedServiceSettings
>>> engine = ReadEngine(
...     connection=MockBackend(MockLinkedServiceSettings()),
...     settings=MockDatasetSettings(row_count=2),
...     dataset_name="mock-ds",
...     provider_type="ds.resource.dataset.mock",
... )
>>> engine.execute(checkpoint={})
>>> len(engine.output.index)
2

Attributes

logger

Classes

ReadEngine

Execute mock reads including pagination, resume, and failure injection.

Module Contents

ds_provider_mock_py_lib.dataset.engines.read.logger

class ds_provider_mock_py_lib.dataset.engines.read.ReadEngine

Execute mock reads including pagination, resume, and failure injection.

connection: ds_provider_mock_py_lib.linked_service.mock.MockBackend

settings: ds_provider_mock_py_lib.dataset.settings.MockDatasetSettings

dataset_name: str

provider_type: str

output: pandas.DataFrame

checkpoint: ds_provider_mock_py_lib.dataset.engines._read_checkpoint.Checkpoint

metadata: dict[str, Any]

_emitted_ops: list[str] = []

execute(checkpoint: dict[str, Any] | None = None) → None

Execute read flow and update output / checkpoint state.

Parameters:

checkpoint – Existing checkpoint state to continue from.

Raises:

• ReadError – If settings are invalid or a wrapped backend failure occurs.

• ConnectionError – If dataset-level injection is configured as a connection error.

• MockBackendError – If the mock backend raises and the caller must wrap it.

_validate_settings() → None

Raise when settings cannot fulfil a read.

_resolve_start() → tuple[int, int, bool]

Resolve batch, first page, and whether this call is a resume.

Returns:

(batch, first_page, resumed).

Raises:

ReadError – If a stored pagination cursor cannot be decoded.

_save_pagination(*, batch: int, page: int) → None

Persist pagination resume state without advancing the watermark.

_assign_output(pages: list[pandas.DataFrame]) → None

Assign concatenated pages to output, including rows already read on failure.

Parameters:

pages – Pages collected so far in this read.

_finish_complete(*, batch: int, pages: list[pandas.DataFrame], total_pages: int, resumed: bool) → None

Complete a successful read of the configured scope.

Parameters:

• batch – Completed batch number written to the incremental watermark.

• pages – Pages collected during this read.

• total_pages – Page count for the current batch.

• resumed – Whether this call continued from a pagination cursor.

_set_metadata(*, batch: int, total_pages: int, resumed: bool) → None

Populate engine metadata used by self.operation.metadata.

Parameters:

• batch – Batch number that was read.

• total_pages – Page count for the current batch.

• resumed – Whether this call continued from a pagination cursor.

_inject_failure(details: dict[str, Any]) → None

Raise the configured dataset-level contract exception.

Parameters:

details – Page context attached to the raised exception.

Raises:

• ConnectionError – When raise_as is connection_error.

• ReadError – When raise_as is read_error.