ds_provider_powerofficego_py_lib.dataset.powerofficego ====================================================== .. py:module:: ds_provider_powerofficego_py_lib.dataset.powerofficego .. autoapi-nested-parse:: **File:** ``powerofficego.py`` **Region:** ``ds_provider_powerofficego_py_lib/dataset/powerofficego`` This module contains dataset-related classes and functions for the PowerOfficeGo provider. Attributes ---------- .. autoapisummary:: ds_provider_powerofficego_py_lib.dataset.powerofficego.logger ds_provider_powerofficego_py_lib.dataset.powerofficego.PowerOfficeGoDatasetSettingsType ds_provider_powerofficego_py_lib.dataset.powerofficego.PowerOfficeGoLinkedServiceType Classes ------- .. autoapisummary:: ds_provider_powerofficego_py_lib.dataset.powerofficego.ReadSettings ds_provider_powerofficego_py_lib.dataset.powerofficego.PowerOfficeGoDatasetSettings ds_provider_powerofficego_py_lib.dataset.powerofficego.PowerOfficeGoDataset Module Contents --------------- .. py:data:: logger .. py:class:: ReadSettings Bases: :py:obj:`ds_common_serde_py_lib.Serializable` Settings for reading from PowerOfficeGo dataset. .. attribute:: page_size Number of records to read per page. Default is 20,000. :type: int .. attribute:: fields List of fields to include in the response. Optional. :type: list[str] | None .. attribute:: filters Additional filters for the API request. Optional. :type: dict[str, Any] | None .. py:attribute:: page_size :type: int :value: 20000 Number of records to read per page. Default is 20,000. .. py:attribute:: fields :type: list[str] | None :value: None List of fields to include in the response. Optional. .. py:attribute:: filters :type: dict[str, Any] | None :value: None Additional filters for the API request. Optional. .. py:class:: PowerOfficeGoDatasetSettings Bases: :py:obj:`ds_resource_plugin_py_lib.common.resource.dataset.DatasetSettings` Settings for PowerOfficeGo dataset. .. attribute:: data_product Data product to read from PowerOfficeGo API. Required. Must match product name as used in DS Config. :type: str .. attribute:: read Settings for reading from PowerOfficeGo dataset. Contains pagination and filtering options. :type: ReadSettings .. py:attribute:: data_product :type: str Data product to read from PowerOfficeGo API. Required. Must match product name as used in DS Config. .. py:attribute:: read :type: ReadSettings Settings for reading from PowerOfficeGo dataset. .. py:data:: PowerOfficeGoDatasetSettingsType .. py:data:: PowerOfficeGoLinkedServiceType .. py:class:: PowerOfficeGoDataset Bases: :py:obj:`ds_resource_plugin_py_lib.common.resource.dataset.TabularDataset`\ [\ :py:obj:`PowerOfficeGoLinkedServiceType`\ , :py:obj:`PowerOfficeGoDatasetSettingsType`\ , :py:obj:`ds_resource_plugin_py_lib.common.serde.serialize.PandasSerializer`\ , :py:obj:`ds_resource_plugin_py_lib.common.serde.deserialize.PandasDeserializer`\ ], :py:obj:`Generic`\ [\ :py:obj:`PowerOfficeGoLinkedServiceType`\ , :py:obj:`PowerOfficeGoDatasetSettingsType`\ ] PowerOfficeGoDataset represents a dataset for the PowerOfficeGo provider. .. attribute:: linked_service The linked service used to connect to PowerOfficeGo. :type: PowerOfficeGoLinkedServiceType .. attribute:: settings The settings for the dataset, including data product and read settings. :type: PowerOfficeGoDatasetSettingsType .. attribute:: serializer The serializer used for the dataset. Defaults to JSON format. :type: PandasSerializer | None .. attribute:: deserializer The deserializer used for the dataset. Defaults to JSON format. :type: PandasDeserializer | None .. py:attribute:: linked_service :type: PowerOfficeGoLinkedServiceType .. py:attribute:: settings :type: PowerOfficeGoDatasetSettingsType .. py:attribute:: serializer :type: ds_resource_plugin_py_lib.common.serde.serialize.PandasSerializer | None .. py:attribute:: deserializer :type: ds_resource_plugin_py_lib.common.serde.deserialize.PandasDeserializer | None .. py:property:: type :type: ds_provider_powerofficego_py_lib.enums.ResourceType Get the type of the dataset. .. py:property:: supports_checkpoint :type: bool Whether this provider supports incremental loads via ``self.checkpoint``. The checkpoint is a dictionary that tracks pagination and incremental state: - On a full load, ``self.checkpoint`` is expected to be empty (``{}``) or ``None``. In this case, :meth:`read` starts from page ``1``. - After each successfully read page, ``self.checkpoint`` is updated with at least ``{"last_page": page, ...}``. - If incremental loading is possible (i.e., the data contains a ``lastChangedDateTimeOffset`` field), the checkpoint will also include an ``incremental`` key with the latest observed value: ``{"incremental": {"last_modified_date": ...}}``. - On a subsequent run, if ``self.checkpoint`` contains a ``"last_page"`` entry, :meth:`read` resumes from ``last_page + 1`` and continues fetching data from the PowerOfficeGo API. - If ``self.checkpoint`` contains an ``incremental`` key, the loader will use the stored ``last_modified_date`` to filter for new/changed records. This allows consumers to perform incremental loads by persisting and reusing the checkpoint between executions, avoiding re-reading pages that were already processed successfully. The checkpoint structure is designed to support both paginated and incremental (watermark-based) loading. :returns: True if checkpointing is supported, False otherwise. :rtype: bool .. py:method:: read() -> None Read data from PowerOfficeGo API from requested endpoint. :raises ReadError: If there is an error during the read operation. .. py:method:: create() -> None Insert all rows in ``self.input`` into the target as a single atomic transaction. Must not delete, update, or overwrite existing data. :raises CreateError: If the operation fails. :raises NotSupportedError: If the provider does not support create. .. seealso:: Full contract: ``docs/DATASET_CONTRACT.md`` -- ``create()`` .. py:method:: delete() -> None Remove specific rows from the target matched by identity columns defined in ``self.settings``. Atomic. Idempotent. :raises DeleteError: If the operation fails. :raises NotSupportedError: If the provider does not support delete. .. seealso:: Full contract: ``docs/DATASET_CONTRACT.md`` -- ``delete()`` .. py:method:: update() -> None Update existing rows in the target matched by identity columns defined in ``self.settings``. Atomic. Must not insert new rows. :raises UpdateError: If the operation fails. :raises NotSupportedError: If the provider does not support update. .. seealso:: Full contract: ``docs/DATASET_CONTRACT.md`` -- ``update()`` .. py:method:: rename() -> None Rename the resource in the backend. Atomic. Not idempotent. :raises RenameError: If the operation fails. :raises NotSupportedError: If the provider does not support renaming. .. seealso:: Full contract: ``docs/DATASET_CONTRACT.md`` -- ``rename()`` .. py:method:: list() -> None Discover available resources and populate ``self.output`` with a DataFrame of resources and their metadata. Idempotent. :raises ListError: If the operation fails. :raises NotSupportedError: If the provider does not support listing. .. seealso:: Full contract: ``docs/DATASET_CONTRACT.md`` -- ``list()`` .. py:method:: upsert() -> None Insert rows that do not exist, update rows that do, matched by identity columns defined in ``self.settings``. Atomic. :raises UpsertError: If the operation fails. :raises NotSupportedError: If the provider does not support upsert. .. seealso:: Full contract: ``docs/DATASET_CONTRACT.md`` -- ``upsert()`` .. py:method:: purge() -> None Remove all content from the target. ``self.input`` is not used. Atomic. Idempotent. :raises PurgeError: If the operation fails. :raises NotSupportedError: If the provider does not support purge. .. seealso:: Full contract: ``docs/DATASET_CONTRACT.md`` -- ``purge()`` .. py:method:: _fetch_data(session: Any) -> None Fetch data from PowerOfficeGo API using the provided session. :param session: The session object to use for making API requests. :type session: Any :raises ReadError: If there is an error during the data fetching process. .. py:method:: _parse_iso8601_timestamp(value: str) -> datetime.datetime :staticmethod: Parse an ISO-8601 timestamp string to an aware UTC datetime for comparison. Handles up to 7 digits of fractional seconds by truncating to 6 (Python max). .. py:method:: _greatest_time_field_value(values: collections.abc.Sequence[Any]) -> Any Return the original value that sorts last by parsed UTC ``datetime``. .. py:method:: greatest_incremental_value(values: collections.abc.Sequence[Any], *, kind: str) -> Any | None Return the greatest watermark among observed values for the given strategy. :param values: Non-empty sequence of observed watermark candidates (nulls should be excluded by callers). :param kind: Incremental strategy from metadata (e.g. ``time_field``). :returns: The winning original value from ``values``, or ``None`` when ``values`` is empty. :raises InvalidIncrementalWatermarkException: When ``time_field`` values are not strings or parsing fails. :raises UnsupportedIncrementalKindException: When ``kind`` is not supported. .. py:method:: _build_checkpoint(last_page: int | None, last_modified_date: str | None) -> dict[str, Any] Build a checkpoint dictionary to track the last successfully read page. Include last_modified_date from read settings. :param last_page: The last page number that was successfully read. :type last_page: int | None :param last_modified_date: The last modified date to include in the checkpoint. :type last_modified_date: str | None :returns: A checkpoint dictionary containing the last page information. :rtype: dict[str, Any] .. py:method:: _build_params(page: int, last_modified_date: str | None) -> dict[str, Any] Build the query parameters for the PowerOfficeGo API request based on the dataset settings and pagination. :param page: The page number to fetch. :type page: int :param last_modified_date: The last modified date to filter results. :type last_modified_date: str | None :returns: A dictionary of query parameters for the API request. :rtype: dict[str, Any] .. py:method:: _build_url() -> str Build the URL for the PowerOfficeGo API request based on the dataset settings. :returns: The constructed URL for the API request. :rtype: str .. py:method:: close() -> None Release any resources held by the dataset. Connection lifecycle is managed by the linked service.