ds_provider_unimicro_py_lib.dataset.unimicro ============================================ .. py:module:: ds_provider_unimicro_py_lib.dataset.unimicro .. autoapi-nested-parse:: **File:** ``unimicro.py`` **Region:** ``ds_provider_unimicro_py_lib/dataset/unimicro`` This module contains dataset-related classes and functions for the Unimicro provider. .. rubric:: Example >>> import ds_provider_unimicro_py_lib.dataset.unimicro Attributes ---------- .. autoapisummary:: ds_provider_unimicro_py_lib.dataset.unimicro.logger ds_provider_unimicro_py_lib.dataset.unimicro.UnimicroDatasetSettingsType ds_provider_unimicro_py_lib.dataset.unimicro.UnimicroLinkedServiceType Classes ------- .. autoapisummary:: ds_provider_unimicro_py_lib.dataset.unimicro.UnimicroReadSettings ds_provider_unimicro_py_lib.dataset.unimicro.UnimicroDatasetSettings ds_provider_unimicro_py_lib.dataset.unimicro.UnimicroDataset Module Contents --------------- .. py:data:: logger .. py:class:: UnimicroReadSettings Bases: :py:obj:`ds_common_serde_py_lib.Serializable` Read settings specific to the Unimicro provider. .. py:attribute:: id :type: int | None :value: None The unique identifier of the record to read from the Unimicro dataset. Optional. .. py:attribute:: page_size :type: int :value: 1000 The number of records to fetch per page from the Unimicro dataset. Default is 1000. .. py:attribute:: fields :type: list[str] | None :value: None The specific fields to read from the Unimicro dataset. Optional. .. py:attribute:: filters :type: str | None :value: None Custom filters to apply when reading from the Unimicro dataset. Knowledge of the filter syntax is required. Optional. .. py:class:: UnimicroDatasetSettings Bases: :py:obj:`ds_resource_plugin_py_lib.common.resource.dataset.DatasetSettings` Dataset settings specific to the Unimicro provider. .. py:attribute:: data_product :type: str The specific data product within the Unimicro provider. .. py:attribute:: company_key :type: str The company key used for fetching data from the Unimicro provider. .. py:attribute:: read :type: UnimicroReadSettings Read settings for the Unimicro dataset. .. py:data:: UnimicroDatasetSettingsType .. py:data:: UnimicroLinkedServiceType .. py:class:: UnimicroDataset Bases: :py:obj:`ds_resource_plugin_py_lib.common.resource.dataset.TabularDataset`\ [\ :py:obj:`UnimicroLinkedServiceType`\ , :py:obj:`UnimicroDatasetSettingsType`\ , :py:obj:`ds_resource_plugin_py_lib.common.serde.serialize.PandasSerializer`\ , :py:obj:`ds_resource_plugin_py_lib.common.serde.deserialize.PandasDeserializer`\ ], :py:obj:`Generic`\ [\ :py:obj:`UnimicroLinkedServiceType`\ , :py:obj:`UnimicroDatasetSettingsType`\ ] UnimicroDataset represents a dataset specific to the Unimicro provider. .. py:attribute:: linked_service :type: UnimicroLinkedServiceType .. py:attribute:: settings :type: UnimicroDatasetSettingsType .. py:attribute:: serializer :type: ds_resource_plugin_py_lib.common.serde.serialize.PandasSerializer | None .. py:attribute:: deserializer :type: ds_resource_plugin_py_lib.common.serde.deserialize.PandasDeserializer | None .. py:property:: type :type: ds_provider_unimicro_py_lib.enums.ResourceType Get the type of the dataset. .. py:property:: supports_checkpoint :type: bool Whether this provider supports checkpointing for incremental reads. The checkpoint has this structure:: { "incremental": {"value": }, "pagination": {"value": }, } During a read, ``pagination.value`` is used as the initial API ``skip`` offset (defaulting to 0). Pagination is handled within the read call: each subsequent request skips the initial offset plus the number of records already fetched during that call. When ``incremental.value`` is set, requests filter for records whose ``CreatedAt`` or ``UpdatedAt`` timestamp is greater than that value. After a successful read, the checkpoint is replaced with the latest valid timestamp found in the returned records, and ``pagination.value`` is reset to 0. If no valid timestamps are found, the incremental value is None. If a request fails, the checkpoint is updated with the number of records fetched during that read attempt and retains the prior incremental value. Consumers should persist checkpoints separately for each dataset and company. :returns: True, because this provider supports incremental checkpointing. :rtype: bool .. py:method:: read() -> None Reads data from the Unimicro dataset. This method should implement the logic to fetch data from the Unimicro API, respecting the current checkpoint for incremental loading. :returns: None .. py:method:: create() -> None Insert all rows in ``self.input`` into the target as a single atomic transaction. Must not delete, update, or overwrite existing data. :raises CreateError: If the operation fails. :raises NotSupportedError: If the provider does not support create. .. seealso:: Full contract: ``docs/DATASET_CONTRACT.md`` -- ``create()`` .. py:method:: delete() -> None Remove specific rows from the target matched by identity columns defined in ``self.settings``. Atomic. Idempotent. :raises DeleteError: If the operation fails. :raises NotSupportedError: If the provider does not support delete. .. seealso:: Full contract: ``docs/DATASET_CONTRACT.md`` -- ``delete()`` .. py:method:: update() -> None Update existing rows in the target matched by identity columns defined in ``self.settings``. Atomic. Must not insert new rows. :raises UpdateError: If the operation fails. :raises NotSupportedError: If the provider does not support update. .. seealso:: Full contract: ``docs/DATASET_CONTRACT.md`` -- ``update()`` .. py:method:: rename() -> None Rename the resource in the backend. Atomic. Not idempotent. :raises RenameError: If the operation fails. :raises NotSupportedError: If the provider does not support renaming. .. seealso:: Full contract: ``docs/DATASET_CONTRACT.md`` -- ``rename()`` .. py:method:: list() -> None Discover available resources and populate ``self.output`` with a DataFrame of resources and their metadata. Idempotent. :raises ListError: If the operation fails. :raises NotSupportedError: If the provider does not support listing. .. seealso:: Full contract: ``docs/DATASET_CONTRACT.md`` -- ``list()`` .. py:method:: upsert() -> None Insert rows that do not exist, update rows that do, matched by identity columns defined in ``self.settings``. Atomic. :raises UpsertError: If the operation fails. :raises NotSupportedError: If the provider does not support upsert. .. seealso:: Full contract: ``docs/DATASET_CONTRACT.md`` -- ``upsert()`` .. py:method:: purge() -> None Remove all content from the target. ``self.input`` is not used. Atomic. Idempotent. :raises PurgeError: If the operation fails. :raises NotSupportedError: If the provider does not support purge. .. seealso:: Full contract: ``docs/DATASET_CONTRACT.md`` -- ``purge()`` .. py:method:: _fetch_data(session: Any) -> None Fetches data from the Unimicro API using the provided session. :param session: The session object used to make API requests. :type session: Any :returns: None .. py:method:: greatest_incremental_value(all_records: pandas.DataFrame) -> str | None Determine the greatest incremental value (last modified date) from the fetched records. :param all_records: The dataframe of all fetched records. :type all_records: pd.DataFrame :returns: The greatest incremental value found, or None if no records exist. :rtype: str | None .. py:method:: _build_checkpoint(last_loaded_records: int | None, last_modified_date: str | None) -> dict[str, Any] Build a checkpoint dictionary to track the last successfully read page. Include last_modified_date from read settings. :param last_loaded_records: The number of records that were successfully read. :type last_loaded_records: int | None :param last_modified_date: The last modified date to include in the checkpoint. :type last_modified_date: str | None :returns: A checkpoint dictionary containing the last loaded records information. :rtype: dict[str, Any] .. py:method:: _build_incremental_filter(last_modified_date: str) -> str Build the provider-specific incremental filter. .. py:method:: _build_url_with_filters(page_size: int, records_to_skip: int | None, last_modified_date: str | None) -> str Constructs the filter string for the Unimicro API request based on the current settings. :param page_size: The number of records per page for pagination. :type page_size: int :param records_to_skip: The number of records to skip for pagination. :type records_to_skip: int | None :param last_modified_date: The last modified date to filter the data. :type last_modified_date: str | None :returns: The constructed filter string. :rtype: str .. py:method:: _build_url() -> str Constructs the URL for accessing the Unimicro API. :returns: The constructed URL. :rtype: str .. py:method:: close() -> None Release any resources held by the dataset. Connection lifecycle is managed by the linked service.