adamreeve commented on code in PR #50419: URL: https://github.com/apache/arrow/pull/50419#discussion_r3575171603
########## docs/source/python/parquet/parquet.rst: ########## @@ -0,0 +1,605 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. currentmodule:: pyarrow +.. _parquet_single_file: + + +Reading and Writing Single Files +================================ + +The functions :func:`~.parquet.read_table` and :func:`~.parquet.write_table` +read and write the :ref:`pyarrow.Table <data.table>` object, respectively. + +Let's look at a simple table: + +.. code-block:: python + + >>> import numpy as np + >>> import pandas as pd Review Comment: These examples use Pandas where plain PyArrow could be easily used, which might give the impression to users that they need to use Pandas when reading or writing Parquet. I can see these are existing examples that have been moved though, so this isn't really related to this PR. Maybe as a follow up we should look at using pure PyArrow in more places, except where we are explicitly documenting Pandas integration? ########## docs/source/python/parquet/parquet_encryption.rst: ########## @@ -0,0 +1,280 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. currentmodule:: pyarrow +.. _parquet_encryption: + + +Parquet Modular Encryption (Columnar Encryption) +------------------------------------------------ + +Columnar encryption is supported for Parquet files in C++ starting from +Apache Arrow 4.0.0 and in PyArrow starting from Apache Arrow 6.0.0. + +Parquet uses the envelope encryption practice, where file parts are encrypted +with "data encryption keys" (DEKs), and the DEKs are encrypted with "master +encryption keys" (MEKs). The DEKs are randomly generated by Parquet for each +encrypted file/column. The MEKs are generated, stored and managed in a Key +Management Service (KMS) of user’s choice. + +Reading and writing encrypted Parquet files involves passing file encryption +and decryption properties to :class:`~pyarrow.parquet.ParquetWriter` and to +:class:`~.ParquetFile`, respectively. + +Writing an encrypted Parquet file: + +.. code-block:: python + + >>> import pyarrow.parquet as pq + >>> encryption_properties = crypto_factory.file_encryption_properties( # doctest: +SKIP + ... kms_connection_config, encryption_config) + >>> with pq.ParquetWriter(filename, schema, # doctest: +SKIP + ... encryption_properties=encryption_properties) as writer: + ... writer.write_table(table) + +Reading an encrypted Parquet file: + +.. code-block:: python + + >>> decryption_properties = crypto_factory.file_decryption_properties( # doctest: +SKIP + ... kms_connection_config) + >>> parquet_file = pq.ParquetFile(filename, # doctest: +SKIP + ... decryption_properties=decryption_properties) + + +In order to create the encryption and decryption properties, a +:class:`pyarrow.parquet.encryption.CryptoFactory` should be created and +initialized with KMS Client details, as described below. + + +KMS Client +~~~~~~~~~~ + +The master encryption keys should be kept and managed in a production-grade +Key Management System (KMS), deployed in the user's organization. Using Parquet +encryption requires implementation of a client class for the KMS server. +Any KmsClient implementation should implement the informal interface +defined by :class:`pyarrow.parquet.encryption.KmsClient` as following: + +.. code-block:: python + + >>> import pyarrow.parquet.encryption as pe + >>> class MyKmsClient(pe.KmsClient): + ... + ... """An example KmsClient implementation skeleton""" + ... def __init__(self, kms_connection_configuration): + ... pe.KmsClient.__init__(self) + ... # Any KMS-specific initialization based on + ... # kms_connection_configuration comes here + ... + ... def wrap_key(self, key_bytes, master_key_identifier): + ... wrapped_key = ... # call KMS to wrap key_bytes with key specified by + ... # master_key_identifier + ... return wrapped_key + ... + ... def unwrap_key(self, wrapped_key, master_key_identifier): + ... key_bytes = ... # call KMS to unwrap wrapped_key with key specified by + ... # master_key_identifier + ... return key_bytes + +The concrete implementation will be loaded at runtime by a factory function +provided by the user. This factory function will be used to initialize the +:class:`pyarrow.parquet.encryption.CryptoFactory` for creating file encryption +and decryption properties. + +For example, in order to use the ``MyKmsClient`` defined above: + +.. code-block:: python + + >>> def kms_client_factory(kms_connection_configuration): + ... return MyKmsClient(kms_connection_configuration) + + >>> crypto_factory = pe.CryptoFactory(kms_client_factory) + +An :download:`example <../../../python/examples/parquet_encryption/sample_vault_kms_client.py>` +of such a class for an open source +`KMS <https://www.vaultproject.io/api/secret/transit>`_ can be found in the Apache +Arrow GitHub repository. The production KMS client should be designed in +cooperation with an organization's security administrators, and built by +developers with experience in access control management. Once such a class is +created, it can be passed to applications via a factory method and leveraged +by general PyArrow users as shown in the encrypted parquet write/read sample +above. + +KMS connection configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Configuration of connection to KMS (:class:`pyarrow.parquet.encryption.KmsConnectionConfig` +used when creating file encryption and decryption properties) includes the +following options: + +* ``kms_instance_url``, URL of the KMS instance. +* ``kms_instance_id``, ID of the KMS instance that will be used for encryption + (if multiple KMS instances are available). +* ``key_access_token``, authorization token that will be passed to KMS. +* ``custom_kms_conf``, a string dictionary with KMS-type-specific configuration. + +Encryption configuration +~~~~~~~~~~~~~~~~~~~~~~~~ + +:class:`pyarrow.parquet.encryption.EncryptionConfiguration` (used when +creating file encryption properties) includes the following options: + +* ``footer_key``, the ID of the master key for footer encryption/signing. +* ``column_keys``, which columns to encrypt with which key. Dictionary with + master key IDs as the keys, and column name lists as the values, + e.g. ``{key1: [col1, col2], key2: [col3]}``. See notes on nested fields below. +* ``uniform_encryption``, whether to encrypt the footer and all columns with + the same ``footer_key``, instead of specifying ``column_keys`` + individually. Cannot be used together with ``column_keys``. +* ``encryption_algorithm``, the Parquet encryption algorithm. + Can be ``AES_GCM_V1`` (default) or ``AES_GCM_CTR_V1``. +* ``plaintext_footer``, whether to write the file footer in plain text (otherwise it is encrypted). +* ``double_wrapping``, whether to use double wrapping - where data encryption keys (DEKs) + are encrypted with key encryption keys (KEKs), which in turn are encrypted + with master encryption keys (MEKs). If set to ``false``, single wrapping is + used - where DEKs are encrypted directly with MEKs. +* ``cache_lifetime``, the lifetime of cached entities (key encryption keys, + local wrapping keys, KMS client objects) represented as a ``datetime.timedelta``. +* ``internal_key_material``, whether to store key material inside Parquet file footers; + this mode doesn’t produce additional files. If set to ``false``, key material is + stored in separate files in the same folder, which enables key rotation for + immutable Parquet files. +* ``data_key_length_bits``, the length of data encryption keys (DEKs), randomly + generated by Parquet key management tools. Can be 128, 192 or 256 bits. + +.. note:: + When ``double_wrapping`` is true, Parquet implements a "double envelope + encryption" mode that minimizes the interaction of the program with a KMS + server. In this mode, the DEKs are encrypted with "key encryption keys" + (KEKs, randomly generated by Parquet). The KEKs are encrypted with "master + encryption keys" (MEKs) in the KMS; the result and the KEK itself are + cached in the process memory. + +An example encryption configuration: + +.. code-block:: python + + >>> encryption_config = pe.EncryptionConfiguration( + ... footer_key="footer_key_name", + ... column_keys={ + ... "column_key_name": ["Column1", "Column2"], + ... }, + ... ) + +.. note:: + + Columns with nested fields (struct or map data types) can be encrypted as a whole, or only + individual fields. Configure an encryption key for the root column name to encrypt all nested + fields with this key, or configure a key for individual leaf nested fields. + + Conventionally, the key and value fields of a map column ``m`` have the names + ``m.key_value.key`` and ``m.key_value.value``, respectively. + An inner field ``f`` of a struct column ``s`` has the name ``s.f``. + + With above example, *all* inner fields are encrypted with the same key by configuring that key + for column ``m`` and ``s``, respectively. + +An example encryption configuration for columns with nested fields, where +all columns are encrypted with the same key identified by ``column_key_id``: + +.. code-block:: python + + >>> import pyarrow as pa + >>> schema = pa.schema([ + ... ("MapColumn", pa.map_(pa.string(), pa.int32())), + ... ("StructColumn", pa.struct([("f1", pa.int32()), ("f2", pa.string())])), + ... ]) + + >>> encryption_config = pe.EncryptionConfiguration( + ... footer_key="footer_key_name", + ... column_keys={ + ... "column_key_id": [ "MapColumn", "StructColumn" ], + ... }, + ... ) + +An example encryption configuration for columns with nested fields, where +some inner fields are encrypted with the same key identified by ``column_key_id``: + +.. code-block:: python + + >>> schema = pa.schema([ + ... ("MapColumn", pa.map_(pa.string(), pa.int32())), + ... ("StructColumn", pa.struct([("f1", pa.int32()), ("f2", pa.string())])), + ... ]) + + >>> encryption_config = pe.EncryptionConfiguration( + ... footer_key="footer_key_name", + ... column_keys={ + ... "column_key_id": [ "MapColumn.key_value.value", "StructColumn.f1" ], + ... }, + ... ) + +Decryption configuration +~~~~~~~~~~~~~~~~~~~~~~~~ + +:class:`pyarrow.parquet.encryption.DecryptionConfiguration` (used when creating +file decryption properties) is optional and it includes the following options: + +* ``cache_lifetime``, the lifetime of cached entities (key encryption keys, local + wrapping keys, KMS client objects) represented as a ``datetime.timedelta``. + +External key material and key rotation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When ``internal_key_material=False`` is set on ``EncryptionConfiguration``, +key material is stored in a separate file next to the Parquet file instead of +in its footer. + +Storing key material externally is what enables key rotation: +``crypto_factory.rotate_master_keys()`` re-wraps the data encryption keys of Review Comment: This should be a cross-reference to the method documentation: ```suggestion :meth:`~pyarrow.parquet.encryption.CryptoFactory.rotate_master_keys` re-wraps the data encryption keys of ``` ########## docs/source/python/parquet/parquet_encryption.rst: ########## @@ -0,0 +1,280 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. currentmodule:: pyarrow +.. _parquet_encryption: + + +Parquet Modular Encryption (Columnar Encryption) +------------------------------------------------ + +Columnar encryption is supported for Parquet files in C++ starting from +Apache Arrow 4.0.0 and in PyArrow starting from Apache Arrow 6.0.0. + +Parquet uses the envelope encryption practice, where file parts are encrypted +with "data encryption keys" (DEKs), and the DEKs are encrypted with "master +encryption keys" (MEKs). The DEKs are randomly generated by Parquet for each +encrypted file/column. The MEKs are generated, stored and managed in a Key +Management Service (KMS) of user’s choice. + +Reading and writing encrypted Parquet files involves passing file encryption +and decryption properties to :class:`~pyarrow.parquet.ParquetWriter` and to +:class:`~.ParquetFile`, respectively. + +Writing an encrypted Parquet file: + +.. code-block:: python + + >>> import pyarrow.parquet as pq + >>> encryption_properties = crypto_factory.file_encryption_properties( # doctest: +SKIP + ... kms_connection_config, encryption_config) + >>> with pq.ParquetWriter(filename, schema, # doctest: +SKIP + ... encryption_properties=encryption_properties) as writer: + ... writer.write_table(table) + +Reading an encrypted Parquet file: + +.. code-block:: python + + >>> decryption_properties = crypto_factory.file_decryption_properties( # doctest: +SKIP + ... kms_connection_config) + >>> parquet_file = pq.ParquetFile(filename, # doctest: +SKIP + ... decryption_properties=decryption_properties) + + +In order to create the encryption and decryption properties, a +:class:`pyarrow.parquet.encryption.CryptoFactory` should be created and +initialized with KMS Client details, as described below. + + +KMS Client +~~~~~~~~~~ + +The master encryption keys should be kept and managed in a production-grade +Key Management System (KMS), deployed in the user's organization. Using Parquet +encryption requires implementation of a client class for the KMS server. +Any KmsClient implementation should implement the informal interface +defined by :class:`pyarrow.parquet.encryption.KmsClient` as following: + +.. code-block:: python + + >>> import pyarrow.parquet.encryption as pe + >>> class MyKmsClient(pe.KmsClient): + ... + ... """An example KmsClient implementation skeleton""" + ... def __init__(self, kms_connection_configuration): + ... pe.KmsClient.__init__(self) + ... # Any KMS-specific initialization based on + ... # kms_connection_configuration comes here + ... + ... def wrap_key(self, key_bytes, master_key_identifier): + ... wrapped_key = ... # call KMS to wrap key_bytes with key specified by + ... # master_key_identifier + ... return wrapped_key + ... + ... def unwrap_key(self, wrapped_key, master_key_identifier): + ... key_bytes = ... # call KMS to unwrap wrapped_key with key specified by + ... # master_key_identifier + ... return key_bytes + +The concrete implementation will be loaded at runtime by a factory function +provided by the user. This factory function will be used to initialize the +:class:`pyarrow.parquet.encryption.CryptoFactory` for creating file encryption +and decryption properties. + +For example, in order to use the ``MyKmsClient`` defined above: + +.. code-block:: python + + >>> def kms_client_factory(kms_connection_configuration): + ... return MyKmsClient(kms_connection_configuration) + + >>> crypto_factory = pe.CryptoFactory(kms_client_factory) + +An :download:`example <../../../python/examples/parquet_encryption/sample_vault_kms_client.py>` +of such a class for an open source +`KMS <https://www.vaultproject.io/api/secret/transit>`_ can be found in the Apache +Arrow GitHub repository. The production KMS client should be designed in +cooperation with an organization's security administrators, and built by +developers with experience in access control management. Once such a class is +created, it can be passed to applications via a factory method and leveraged +by general PyArrow users as shown in the encrypted parquet write/read sample +above. + +KMS connection configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Configuration of connection to KMS (:class:`pyarrow.parquet.encryption.KmsConnectionConfig` +used when creating file encryption and decryption properties) includes the +following options: + +* ``kms_instance_url``, URL of the KMS instance. +* ``kms_instance_id``, ID of the KMS instance that will be used for encryption + (if multiple KMS instances are available). +* ``key_access_token``, authorization token that will be passed to KMS. +* ``custom_kms_conf``, a string dictionary with KMS-type-specific configuration. + +Encryption configuration +~~~~~~~~~~~~~~~~~~~~~~~~ + +:class:`pyarrow.parquet.encryption.EncryptionConfiguration` (used when +creating file encryption properties) includes the following options: + +* ``footer_key``, the ID of the master key for footer encryption/signing. +* ``column_keys``, which columns to encrypt with which key. Dictionary with + master key IDs as the keys, and column name lists as the values, + e.g. ``{key1: [col1, col2], key2: [col3]}``. See notes on nested fields below. +* ``uniform_encryption``, whether to encrypt the footer and all columns with + the same ``footer_key``, instead of specifying ``column_keys`` + individually. Cannot be used together with ``column_keys``. +* ``encryption_algorithm``, the Parquet encryption algorithm. + Can be ``AES_GCM_V1`` (default) or ``AES_GCM_CTR_V1``. +* ``plaintext_footer``, whether to write the file footer in plain text (otherwise it is encrypted). +* ``double_wrapping``, whether to use double wrapping - where data encryption keys (DEKs) + are encrypted with key encryption keys (KEKs), which in turn are encrypted + with master encryption keys (MEKs). If set to ``false``, single wrapping is + used - where DEKs are encrypted directly with MEKs. +* ``cache_lifetime``, the lifetime of cached entities (key encryption keys, + local wrapping keys, KMS client objects) represented as a ``datetime.timedelta``. +* ``internal_key_material``, whether to store key material inside Parquet file footers; + this mode doesn’t produce additional files. If set to ``false``, key material is + stored in separate files in the same folder, which enables key rotation for + immutable Parquet files. +* ``data_key_length_bits``, the length of data encryption keys (DEKs), randomly + generated by Parquet key management tools. Can be 128, 192 or 256 bits. + +.. note:: + When ``double_wrapping`` is true, Parquet implements a "double envelope + encryption" mode that minimizes the interaction of the program with a KMS + server. In this mode, the DEKs are encrypted with "key encryption keys" + (KEKs, randomly generated by Parquet). The KEKs are encrypted with "master + encryption keys" (MEKs) in the KMS; the result and the KEK itself are + cached in the process memory. + +An example encryption configuration: + +.. code-block:: python + + >>> encryption_config = pe.EncryptionConfiguration( + ... footer_key="footer_key_name", + ... column_keys={ + ... "column_key_name": ["Column1", "Column2"], + ... }, + ... ) + +.. note:: + + Columns with nested fields (struct or map data types) can be encrypted as a whole, or only + individual fields. Configure an encryption key for the root column name to encrypt all nested + fields with this key, or configure a key for individual leaf nested fields. + + Conventionally, the key and value fields of a map column ``m`` have the names + ``m.key_value.key`` and ``m.key_value.value``, respectively. + An inner field ``f`` of a struct column ``s`` has the name ``s.f``. + + With above example, *all* inner fields are encrypted with the same key by configuring that key + for column ``m`` and ``s``, respectively. + +An example encryption configuration for columns with nested fields, where +all columns are encrypted with the same key identified by ``column_key_id``: + +.. code-block:: python + + >>> import pyarrow as pa + >>> schema = pa.schema([ + ... ("MapColumn", pa.map_(pa.string(), pa.int32())), + ... ("StructColumn", pa.struct([("f1", pa.int32()), ("f2", pa.string())])), + ... ]) + + >>> encryption_config = pe.EncryptionConfiguration( + ... footer_key="footer_key_name", + ... column_keys={ + ... "column_key_id": [ "MapColumn", "StructColumn" ], + ... }, + ... ) + +An example encryption configuration for columns with nested fields, where +some inner fields are encrypted with the same key identified by ``column_key_id``: + +.. code-block:: python + + >>> schema = pa.schema([ + ... ("MapColumn", pa.map_(pa.string(), pa.int32())), + ... ("StructColumn", pa.struct([("f1", pa.int32()), ("f2", pa.string())])), + ... ]) + + >>> encryption_config = pe.EncryptionConfiguration( + ... footer_key="footer_key_name", + ... column_keys={ + ... "column_key_id": [ "MapColumn.key_value.value", "StructColumn.f1" ], + ... }, + ... ) + +Decryption configuration +~~~~~~~~~~~~~~~~~~~~~~~~ + +:class:`pyarrow.parquet.encryption.DecryptionConfiguration` (used when creating +file decryption properties) is optional and it includes the following options: + +* ``cache_lifetime``, the lifetime of cached entities (key encryption keys, local + wrapping keys, KMS client objects) represented as a ``datetime.timedelta``. + +External key material and key rotation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When ``internal_key_material=False`` is set on ``EncryptionConfiguration``, +key material is stored in a separate file next to the Parquet file instead of +in its footer. + +Storing key material externally is what enables key rotation: +``crypto_factory.rotate_master_keys()`` re-wraps the data encryption keys of +a file that uses external key material under new master keys, without rewriting +the file itself: Review Comment: I think this could be clarified a bit: ```suggestion a file that uses external key material using new master keys and overwrites the external key material file, without changing the Parquet file itself: ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
