ds_provider_postgresql_py_lib ============================= .. py:module:: ds_provider_postgresql_py_lib .. autoapi-nested-parse:: **File:** ``__init__.py`` **Region:** ``ds-provider-postgresql-py-lib`` Description ----------- A Python package from the ds-provider-postgresql-py-lib library. .. rubric:: Example .. code-block:: python from ds_provider_postgresql_py_lib import __version__ print(f"Package version: {__version__}") Submodules ---------- .. toctree:: :maxdepth: 1 /autoapi/ds_provider_postgresql_py_lib/dataset/index /autoapi/ds_provider_postgresql_py_lib/enums/index /autoapi/ds_provider_postgresql_py_lib/linked_service/index /autoapi/ds_provider_postgresql_py_lib/utils/index Attributes ---------- .. autoapisummary:: ds_provider_postgresql_py_lib.__version__ Classes ------- .. autoapisummary:: ds_provider_postgresql_py_lib.PostgreSQLDataset ds_provider_postgresql_py_lib.PostgreSQLDatasetSettings ds_provider_postgresql_py_lib.PostgreSQLLinkedService ds_provider_postgresql_py_lib.PostgreSQLLinkedServiceSettings Package Contents ---------------- .. py:class:: PostgreSQLDataset Bases: :py:obj:`ds_resource_plugin_py_lib.common.resource.dataset.TabularDataset`\ [\ :py:obj:`PostgreSQLLinkedServiceType`\ , :py:obj:`PostgreSQLDatasetSettingsType`\ , :py:obj:`ds_resource_plugin_py_lib.common.serde.serialize.PandasSerializer`\ , :py:obj:`ds_resource_plugin_py_lib.common.serde.deserialize.PandasDeserializer`\ ], :py:obj:`Generic`\ [\ :py:obj:`PostgreSQLLinkedServiceType`\ , :py:obj:`PostgreSQLDatasetSettingsType`\ ] Tabular dataset object which identifies data within a data store, such as table/csv/json/parquet/parquetdataset/ and other documents. The input of the dataset is a pandas DataFrame. The output of the dataset is a pandas DataFrame. .. py:attribute:: linked_service :type: PostgreSQLLinkedServiceType .. py:attribute:: settings :type: PostgreSQLDatasetSettingsType .. py:attribute:: serializer :type: ds_resource_plugin_py_lib.common.serde.serialize.PandasSerializer | None .. py:attribute:: deserializer :type: ds_resource_plugin_py_lib.common.serde.deserialize.PandasDeserializer | None .. py:property:: type :type: ds_provider_postgresql_py_lib.enums.ResourceType Get the type of the dataset. :returns: The dataset resource type. :rtype: ResourceType .. py:method:: create(**_kwargs: Any) -> None Create/write data to the configured table. :param _kwargs: Additional keyword arguments for interface compatibility. :returns: None :raises CreateError: If writing data fails. .. py:method:: read(**_kwargs: Any) -> None Read rows from the configured table into `self.output`. :param _kwargs: Additional keyword arguments for interface compatibility. :returns: None :raises ReadError: If reading data fails. .. py:method:: delete(**_kwargs: Any) -> None Delete rows matching configured identity columns. :param _kwargs: Additional keyword arguments for interface compatibility. :returns: None :raises DeleteError: If deleting rows fails. .. py:method:: update(**_kwargs: Any) -> None Update rows matching configured identity columns. :param _kwargs: Additional keyword arguments for interface compatibility. :returns: None :raises UpdateError: If updating rows fails. .. py:method:: upsert(**_kwargs: Any) -> None Insert or update rows using PostgreSQL ON CONFLICT semantics. :param _kwargs: Additional keyword arguments for interface compatibility. :returns: None :raises UpsertError: If upserting rows fails. .. py:method:: purge(**_kwargs: Any) -> None Purge table contents or drop the table. :param _kwargs: Additional keyword arguments for interface compatibility. :returns: None :raises PurgeError: If purging table data fails. .. py:method:: list(**_kwargs: Any) -> None List operation is not supported for this provider. :param _kwargs: Additional keyword arguments for interface compatibility. :returns: None :raises NotSupportedError: Always, as list is not supported. .. py:method:: rename(**_kwargs: Any) -> None Rename operation is not supported for this provider. :param _kwargs: Additional keyword arguments for interface compatibility. :returns: None :raises NotSupportedError: Always, as rename is not supported. .. py:method:: close() -> None Close the dataset and underlying linked service. :returns: None .. py:method:: _output_from_empty_input() -> pandas.DataFrame Build a consistent empty-operation output while preserving input schema. :returns: Empty dataframe or a schema-preserving input copy. :rtype: pd.DataFrame .. py:method:: _get_table() -> sqlalchemy.Table Get the reflected SQLAlchemy table for configured schema and table. :returns: Reflected table object. :rtype: Table .. py:method:: _build_table_from_input(content: pandas.DataFrame) -> sqlalchemy.Table Build a SQLAlchemy Table definition from input DataFrame dtypes. :param content: Input DataFrame to build the table from. :returns: SQLAlchemy Table definition. :rtype: Table .. py:method:: _resolve_create_primary_key_columns(content: pandas.DataFrame) -> collections.abc.Sequence[str] | None Resolve and validate create-time primary key columns. :param content: Input DataFrame used for table creation. :returns: Primary key columns for new table creation. :rtype: Sequence[str] | None :raises ValidationError: If `primary_key` is enabled but columns are invalid. .. py:method:: _copy_into_table(conn: Any, table: sqlalchemy.Table, content: pandas.DataFrame) -> None Insert rows using PostgreSQL COPY. .. py:method:: _validate_columns(table: sqlalchemy.Table, column_names: collections.abc.Sequence[str]) -> None Validate that all requested columns exist in the reflected table. :param table: Reflected SQLAlchemy table. :param column_names: Column names to validate. :returns: None :raises ValidationError: If one or more columns do not exist in the table. .. py:method:: _validate_read_settings() -> None Validate read settings before query construction. :returns: None :raises ValidationError: If limit or order direction is invalid. .. py:method:: _build_select_columns(table: sqlalchemy.Table) -> sqlalchemy.sql.Select[Any] Build a SELECT statement for configured columns or all columns. :param table: Reflected SQLAlchemy table. :returns: SELECT statement with chosen columns. :rtype: Select[Any] :raises ValidationError: If any selected column does not exist. .. py:method:: _build_filters(stmt: sqlalchemy.sql.Select[Any], table: sqlalchemy.Table) -> sqlalchemy.sql.Select[Any] Apply equality filters from read settings to the SELECT statement. :param stmt: Current SELECT statement. :param table: Reflected SQLAlchemy table. :returns: SELECT statement with WHERE conditions applied. :rtype: Select[Any] :raises ValidationError: If any filter column does not exist. .. py:method:: _build_order_by(stmt: sqlalchemy.sql.Select[Any], table: sqlalchemy.Table) -> sqlalchemy.sql.Select[Any] Apply ORDER BY clauses from read settings to the SELECT statement. :param stmt: Current SELECT statement. :param table: Reflected SQLAlchemy table. :returns: SELECT statement with ORDER BY applied. :rtype: Select[Any] :raises ValidationError: If any order-by column does not exist. .. py:class:: PostgreSQLDatasetSettings Bases: :py:obj:`ds_resource_plugin_py_lib.common.resource.dataset.DatasetSettings` Settings for PostgreSQL dataset operations. The `read` settings contains read-specific configuration that only applies to the read() operation, not to create(), delete(), update(), etc. .. py:attribute:: schema :type: str :value: 'public' Schema for dataset operations. .. py:attribute:: table :type: str Table for dataset operations. .. py:attribute:: read :type: ReadSettings Settings for read(). .. py:attribute:: create :type: CreateSettings Settings for create(). .. py:attribute:: update :type: UpdateSettings | None :value: None Settings for update(). .. py:attribute:: upsert :type: UpsertSettings | None :value: None Settings for upsert(). .. py:attribute:: delete :type: DeleteSettings | None :value: None Settings for delete(). .. py:attribute:: purge :type: PurgeSettings Settings for purge(). .. py:class:: PostgreSQLLinkedService Bases: :py:obj:`ds_resource_plugin_py_lib.common.resource.linked_service.LinkedService`\ [\ :py:obj:`PostgreSQLLinkedServiceSettingsType`\ ], :py:obj:`Generic`\ [\ :py:obj:`PostgreSQLLinkedServiceSettingsType`\ ] The class is used to connect with PostgreSQL database. .. py:attribute:: settings :type: PostgreSQLLinkedServiceSettingsType .. py:attribute:: _engine :type: sqlalchemy.Engine | None :value: None The SQLAlchemy engine instance with connection pool. .. py:property:: type :type: ds_provider_postgresql_py_lib.enums.ResourceType Get the type of the linked service. :returns: ResourceType .. py:property:: engine :type: sqlalchemy.Engine | None Get the SQLAlchemy engine instance. :returns: The engine if initialized, None otherwise. :rtype: Engine | None .. py:property:: connection :type: sqlalchemy.Engine Get the established backend connection object. :returns: The initialized SQLAlchemy engine. :rtype: Engine :raises ConnectionError: If connect() has not been called successfully. .. py:property:: pool :type: sqlalchemy.pool.Pool | None Get the connection pool from the engine. The pool is automatically created by SQLAlchemy when create_engine() is called with pool parameters. All connections (via engine.connect(), engine.begin(), etc.) automatically use this pool. :returns: The connection pool if the engine is initialized, None otherwise. :rtype: Pool | None .. py:method:: connect() -> None Connect to the PostgreSQL database and create a connection pool. :returns: None .. py:method:: test_connection() -> tuple[bool, str] Test the connection to the PostgreSQL database. :returns: A tuple containing a boolean indicating success and a string message. :rtype: tuple[bool, str] .. py:method:: _is_authentication_error(error: Exception) -> bool Detect authentication failures from SQLAlchemy/driver exceptions. :param error: The exception to check. :returns: True if the error is an authentication error, False otherwise. :rtype: bool .. py:method:: close() -> None Close the linked service. .. py:class:: PostgreSQLLinkedServiceSettings Bases: :py:obj:`ds_resource_plugin_py_lib.common.resource.linked_service.LinkedServiceSettings` The object containing the PostgreSQL linked service settings. .. py:attribute:: uri :type: str PostgreSQL connection URI. Format: postgresql://[user[:password]@][host][:port][/database][?param1=value1¶m2=value2] .. rubric:: Examples postgresql://user:password@localhost:5432/mydb postgresql://user:password@localhost:5432/mydb?sslmode=require postgresql://user@localhost/mydb postgresql://localhost/mydb .. py:attribute:: pool_size :type: int :value: 5 The size of the connection pool. Defaults to 5. .. py:attribute:: max_overflow :type: int :value: 10 The maximum overflow connections allowed. Defaults to 10. .. py:attribute:: pool_timeout :type: int :value: 30 The timeout in seconds for getting a connection from the pool. Defaults to 30. .. py:attribute:: pool_recycle :type: int :value: 3600 The time in seconds after which a connection is recycled. Defaults to 3600. .. py:data:: __version__