ds_provider_unimicro_py_lib.dataset.unimicro¶

File: unimicro.py Region: ds_provider_unimicro_py_lib/dataset/unimicro

This module contains dataset-related classes and functions for the Unimicro provider.

Example

>>> import ds_provider_unimicro_py_lib.dataset.unimicro

Attributes¶

`logger`
`UnimicroDatasetSettingsType`
`UnimicroLinkedServiceType`

Classes¶

`UnimicroReadSettings`	Read settings specific to the Unimicro provider.
`UnimicroDatasetSettings`	Dataset settings specific to the Unimicro provider.
`UnimicroDataset`	UnimicroDataset represents a dataset specific to the Unimicro provider.

Module Contents¶

ds_provider_unimicro_py_lib.dataset.unimicro.logger¶

class ds_provider_unimicro_py_lib.dataset.unimicro.UnimicroReadSettings[source]¶

Bases: ds_common_serde_py_lib.Serializable

Read settings specific to the Unimicro provider.

id: int | None = None¶: The unique identifier of the record to read from the Unimicro dataset. Optional.

page_size: int = 1000¶: The number of records to fetch per page from the Unimicro dataset. Default is 1000.

fields: list[str] | None = None¶: The specific fields to read from the Unimicro dataset. Optional.

filters: str | None = None¶: Custom filters to apply when reading from the Unimicro dataset. Knowledge of the filter syntax is required. Optional.

class ds_provider_unimicro_py_lib.dataset.unimicro.UnimicroDatasetSettings[source]¶

Bases: ds_resource_plugin_py_lib.common.resource.dataset.DatasetSettings

Dataset settings specific to the Unimicro provider.

data_product: str¶: The specific data product within the Unimicro provider.

company_key: str¶: The company key used for fetching data from the Unimicro provider.

read: UnimicroReadSettings¶: Read settings for the Unimicro dataset.

ds_provider_unimicro_py_lib.dataset.unimicro.UnimicroDatasetSettingsType¶

ds_provider_unimicro_py_lib.dataset.unimicro.UnimicroLinkedServiceType¶

class ds_provider_unimicro_py_lib.dataset.unimicro.UnimicroDataset[source]¶

Bases: ds_resource_plugin_py_lib.common.resource.dataset.TabularDataset[UnimicroLinkedServiceType, UnimicroDatasetSettingsType, ds_resource_plugin_py_lib.common.serde.serialize.PandasSerializer, ds_resource_plugin_py_lib.common.serde.deserialize.PandasDeserializer], Generic[UnimicroLinkedServiceType, UnimicroDatasetSettingsType]

UnimicroDataset represents a dataset specific to the Unimicro provider.

linked_service: UnimicroLinkedServiceType¶

settings: UnimicroDatasetSettingsType¶

serializer: ds_resource_plugin_py_lib.common.serde.serialize.PandasSerializer | None¶

deserializer: ds_resource_plugin_py_lib.common.serde.deserialize.PandasDeserializer | None¶

property type: ds_provider_unimicro_py_lib.enums.ResourceType¶: Get the type of the dataset.

property supports_checkpoint: bool¶

Whether this provider supports checkpointing for incremental reads.

The checkpoint has this structure:

{
    "incremental": {"value": <timestamp or None>},
    "pagination": {"value": <record offset>},
}

During a read, pagination.value is used as the initial API skip offset (defaulting to 0). Pagination is handled within the read call: each subsequent request skips the initial offset plus the number of records already fetched during that call.

When incremental.value is set, requests filter for records whose CreatedAt or UpdatedAt timestamp is greater than that value. After a successful read, the checkpoint is replaced with the latest valid timestamp found in the returned records, and pagination.value is reset to 0. If no valid timestamps are found, the incremental value is None.

If a request fails, the checkpoint is updated with the number of records fetched during that read attempt and retains the prior incremental value. Consumers should persist checkpoints separately for each dataset and company.

Returns:: True, because this provider supports incremental checkpointing.
Return type:: bool

read() → None[source]¶

Reads data from the Unimicro dataset.

This method should implement the logic to fetch data from the Unimicro API, respecting the current checkpoint for incremental loading.

Returns:: None

create() → None[source]¶

Insert all rows in self.input into the target as a single atomic transaction. Must not delete, update, or overwrite existing data.

Raises:

CreateError – If the operation fails.
NotSupportedError – If the provider does not support create.

See also

Full contract: docs/DATASET_CONTRACT.md – create()

delete() → None[source]¶

Remove specific rows from the target matched by identity columns defined in self.settings. Atomic. Idempotent.

Raises:

DeleteError – If the operation fails.
NotSupportedError – If the provider does not support delete.

See also

Full contract: docs/DATASET_CONTRACT.md – delete()

update() → None[source]¶

Update existing rows in the target matched by identity columns defined in self.settings. Atomic. Must not insert new rows.

Raises:

UpdateError – If the operation fails.
NotSupportedError – If the provider does not support update.

See also

Full contract: docs/DATASET_CONTRACT.md – update()

rename() → None[source]¶

Rename the resource in the backend. Atomic. Not idempotent.

Raises:

RenameError – If the operation fails.
NotSupportedError – If the provider does not support renaming.

See also

Full contract: docs/DATASET_CONTRACT.md – rename()

list() → None[source]¶

Discover available resources and populate self.output with a DataFrame of resources and their metadata. Idempotent.

Raises:

ListError – If the operation fails.
NotSupportedError – If the provider does not support listing.

See also

Full contract: docs/DATASET_CONTRACT.md – list()

upsert() → None[source]¶

Insert rows that do not exist, update rows that do, matched by identity columns defined in self.settings. Atomic.

Raises:

UpsertError – If the operation fails.
NotSupportedError – If the provider does not support upsert.

See also

Full contract: docs/DATASET_CONTRACT.md – upsert()

purge() → None[source]¶

Remove all content from the target. self.input is not used. Atomic. Idempotent.

Raises:

PurgeError – If the operation fails.
NotSupportedError – If the provider does not support purge.

See also

Full contract: docs/DATASET_CONTRACT.md – purge()

_fetch_data(session: Any) → None[source]¶

Fetches data from the Unimicro API using the provided session.

Parameters:: session (Any) – The session object used to make API requests.
Returns:: None

greatest_incremental_value(all_records: pandas.DataFrame) → str | None[source]¶

Determine the greatest incremental value (last modified date) from the fetched records.

Parameters:: all_records (pd.DataFrame) – The dataframe of all fetched records.
Returns:: The greatest incremental value found, or None if no records exist.
Return type:: str | None

_build_checkpoint(last_loaded_records: int | None, last_modified_date: str | None) → dict[str, Any][source]¶

Build a checkpoint dictionary to track the last successfully read page.

Include last_modified_date from read settings.

Parameters:

last_loaded_records (int | None) – The number of records that were successfully read.
last_modified_date (str | None) – The last modified date to include in the checkpoint.

Returns:

A checkpoint dictionary containing the last loaded records information.

Return type:

dict[str, Any]

_build_incremental_filter(last_modified_date: str) → str[source]¶: Build the provider-specific incremental filter.

_build_url_with_filters(page_size: int, records_to_skip: int | None, last_modified_date: str | None) → str[source]¶

Constructs the filter string for the Unimicro API request based on the current settings.

Parameters:

page_size (int) – The number of records per page for pagination.
records_to_skip (int | None) – The number of records to skip for pagination.
last_modified_date (str | None) – The last modified date to filter the data.

Returns:

The constructed filter string.

Return type:

str

_build_url() → str[source]¶

Constructs the URL for accessing the Unimicro API.

Returns:: The constructed URL.
Return type:: str

close() → None[source]¶

Release any resources held by the dataset.

Connection lifecycle is managed by the linked service.