Re-opening the ZOL crypto commit for final reviews and testing. Original upstream commit is https://github.com/zfsonlinux/zfs/pull/5769
This will replace, and close, PR https://github.com/openzfs/openzfs/pull/124 This commit contains all additional PRs, fixes and corrections from ZOL, in one commit. The only difference to upstream, is userland `ztest` do not test crypto for IllumOS as the crypto framework is not compiled for userland. A decision should perhaps be made here to leave it as it is, or create and compile a library to link `ztest` with, or compile directly into `ztest`. The ZOL message follows: This change incorporates three major pieces: The first change is a keystore that manages wrapping and encryption keys for encrypted datasets. These commands mostly involve manipulating the new DSL Crypto Key ZAP Objects that live in the MOS. Each encrypted dataset has its own DSL Crypto Key that is protected with a user's key. This level of indirection allows users to change their keys without re-encrypting their entire datasets. The change implements the new subcommands "zfs load-key", "zfs unload-key" and "zfs change-key" which allow the user to manage their encryption keys and settings. In addition, several new flags and properties have been added to allow dataset creation and to make mounting and unmounting more convenient. The second piece of this patch provides the ability to encrypt, decyrpt, and authenticate protected datasets. Each object set maintains a Merkel tree of Message Authentication Codes that protect the lower layers, similarly to how checksums are maintained. This part impacts the zio layer, which handles the actual encryption and generation of MACs, as well as the ARC and DMU, which need to be able to handle encrypted buffers and protected data. The last addition is the ability to do raw, encrypted sends and receives. The idea here is to send raw encrypted and compressed data and receive it exactly as is on a backup system. This means that the dataset on the receiving system is protected using the same user key that is in use on the sending side. By doing so, datasets can be efficiently backed up to an untrusted system without fear of data being compromised. Reviewed by: Matthew Ahrens <mahr...@delphix.com> Reviewed-by: Brian Behlendorf <behlendo...@llnl.gov> Reviewed-by: Jorgen Lundman <lund...@lundman.net> Signed-off-by: Tom Caputi <tcap...@datto.com> Send / Recv Fixes following b52563 This patch fixes several issues discovered after the encryption patch was merged: Fixed a bug where encrypted datasets could attempt to receive embedded data records. Fixed a bug where dirty records created by the recv code wasn't properly setting the dr_raw flag. Fixed a typo where a dmu_tx_commit() was changed to dmu_tx_abort() Fixed a few error handling bugs unrelated to the encryption patch in dmu_recv_stream() Signed-off-by: Tom Caputi <tcap...@datto.com> Encryption patch follow-up * HKDF implementation moved to its own file and tests added to ensure correctness. * Ztest can now create and test encrypted datasets. This is currently disabled until issue ZOL #6526 is resolved, but otherwise functions as advertised. * Several small bug fixes discovered after enabling ztest to run on encrypted datasets. * Fixed coverity defects added by the encryption patch. * Updated man pages for encrypted send / receive behavior. * Fixed a bug where encrypted datasets could receive DRR_WRITE_EMBEDDED records. * Minor code cleanups / consolidation. * Includes fix in dmu_free_long_object_impl IllumOS considerations: Disable crypto tests in ztest Unless permission is given to compile the crypto framework in userland the crypto tests in ztest are disabled on IllumOS. You can view, comment on, or merge this pull request online at: https://github.com/openzfs/openzfs/pull/489 -- Commit Summary -- * 8727 Native data and metadata encryption for zfs -- File Changes -- M usr/src/cmd/mdb/common/modules/zfs/zfs.c (21) M usr/src/cmd/mdb/intel/amd64/libzpool/Makefile (3) M usr/src/cmd/mdb/intel/amd64/zfs/Makefile (1) M usr/src/cmd/mdb/intel/ia32/libzpool/Makefile (3) M usr/src/cmd/mdb/intel/ia32/zfs/Makefile (1) M usr/src/cmd/mdb/sparc/v7/libzpool/Makefile (3) M usr/src/cmd/mdb/sparc/v9/libzpool/Makefile (3) M usr/src/cmd/zdb/Makefile.com (3) M usr/src/cmd/zdb/zdb.c (74) M usr/src/cmd/zdb/zdb_il.c (11) M usr/src/cmd/zfs/zfs_main.c (281) M usr/src/cmd/zinject/translate.c (8) M usr/src/cmd/zoneadm/Makefile (2) M usr/src/cmd/zpool/zpool_main.c (64) M usr/src/cmd/zstreamdump/zstreamdump.c (114) M usr/src/cmd/ztest/ztest.c (175) M usr/src/common/zfs/zfeature_common.c (10) M usr/src/common/zfs/zfeature_common.h (1) M usr/src/common/zfs/zfs_deleg.c (2) M usr/src/common/zfs/zfs_deleg.h (2) M usr/src/common/zfs/zfs_prop.c (97) M usr/src/common/zfs/zfs_prop.h (7) M usr/src/lib/libuutil/common/libuutil.h (4) M usr/src/lib/libzfs/Makefile.com (10) M usr/src/lib/libzfs/common/libzfs.h (22) M usr/src/lib/libzfs/common/libzfs_changelist.c (6) A usr/src/lib/libzfs/common/libzfs_crypto.c (1533) M usr/src/lib/libzfs/common/libzfs_dataset.c (134) M usr/src/lib/libzfs/common/libzfs_diff.c (5) M usr/src/lib/libzfs/common/libzfs_mount.c (50) M usr/src/lib/libzfs/common/libzfs_pool.c (28) M usr/src/lib/libzfs/common/libzfs_sendrecv.c (528) M usr/src/lib/libzfs/common/libzfs_util.c (2) M usr/src/lib/libzfs/common/mapfile-vers (5) M usr/src/lib/libzfs_core/common/libzfs_core.c (115) M usr/src/lib/libzfs_core/common/libzfs_core.h (16) M usr/src/lib/libzfs_core/common/mapfile-vers (3) M usr/src/lib/libzpool/Makefile.com (6) M usr/src/lib/libzpool/common/kernel.c (89) M usr/src/man/man1m/zfs.1m (380) M usr/src/man/man1m/zpool.1m (40) M usr/src/man/man5/zpool-features.5 (22) M usr/src/pkg/manifests/system-test-zfstest.mf (131) M usr/src/test/zfs-tests/runfiles/delphix.run (27) M usr/src/test/zfs-tests/runfiles/omnios.run (42) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_change-key/Makefile (21) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_change-key/cleanup.ksh (30) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_change-key/setup.ksh (32) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_change-key/zfs_change-key.ksh (62) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_change-key/zfs_change-key_child.ksh (86) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_change-key/zfs_change-key_format.ksh (71) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_change-key/zfs_change-key_inherit.ksh (78) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_change-key/zfs_change-key_load.ksh (58) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_change-key/zfs_change-key_location.ksh (65) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_change-key/zfs_change-key_pbkdf2iters.ksh (75) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_clone/zfs_clone_encrypted.ksh (83) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_create/zfs_create_014_pos.ksh (59) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_create/zfs_create_crypt_combos.ksh (98) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_create/zfs_create_encrypted.ksh (134) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_load-key/Makefile (21) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_load-key/cleanup.ksh (30) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_load-key/setup.ksh (32) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_load-key/zfs_load-key.cfg (26) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_load-key/zfs_load-key.ksh (85) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_load-key/zfs_load-key_all.ksh (77) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_load-key/zfs_load-key_common.kshlib (102) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_load-key/zfs_load-key_file.ksh (58) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_load-key/zfs_load-key_location.ksh (73) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_load-key/zfs_load-key_noop.ksh (54) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_load-key/zfs_load-key_recursive.ksh (66) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_mount/zfs_mount_encrypted.ksh (62) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_promote/zfs_promote_encryptionroot.ksh (80) M usr/src/test/zfs-tests/tests/functional/cli_root/zfs_receive/setup.ksh (1) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_receive/zfs_receive_from_encrypted.ksh (83) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_receive/zfs_receive_raw.ksh (93) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_receive/zfs_receive_raw_incremental.ksh (75) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_receive/zfs_receive_to_encrypted.ksh (75) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_rename/zfs_rename_encrypted_child.ksh (78) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_rename/zfs_rename_to_encrypted.ksh (51) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_send/zfs_send_encrypted.ksh (76) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_send/zfs_send_encrypted_unloaded.ksh (59) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_send/zfs_send_raw.ksh (79) M usr/src/test/zfs-tests/tests/functional/cli_root/zfs_set/setup.ksh (1) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_set/zfs_set_keylocation.ksh (93) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_unload-key/Makefile (21) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_unload-key/cleanup.ksh (30) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_unload-key/setup.ksh (32) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_unload-key/zfs_unload-key.ksh (69) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_unload-key/zfs_unload-key_all.ksh (76) A usr/src/test/zfs-tests/tests/functional/cli_root/zfs_unload-key/zfs_unload-key_recursive.ksh (72) M usr/src/test/zfs-tests/tests/functional/cli_root/zpool_create/zpool_create_005_pos.ksh (0) A usr/src/test/zfs-tests/tests/functional/cli_root/zpool_create/zpool_create_024_pos.ksh (111) A usr/src/test/zfs-tests/tests/functional/cli_root/zpool_create/zpool_create_crypt_combos.ksh (89) A usr/src/test/zfs-tests/tests/functional/cli_root/zpool_create/zpool_create_encrypted.ksh (95) M usr/src/test/zfs-tests/tests/functional/cli_root/zpool_get/zpool_get.cfg (1) A usr/src/test/zfs-tests/tests/functional/cli_root/zpool_import/zpool_import_encrypted.ksh (59) A usr/src/test/zfs-tests/tests/functional/cli_root/zpool_import/zpool_import_encrypted_load.ksh (59) A usr/src/test/zfs-tests/tests/functional/cli_root/zpool_scrub/zpool_scrub_encrypted_unloaded.ksh (71) M usr/src/test/zfs-tests/tests/functional/rsend/rsend.kshlib (2) A usr/src/test/zfs-tests/tests/functional/rsend/send_encrypted_heirarchy.ksh (96) M usr/src/uts/common/Makefile.files (3) M usr/src/uts/common/crypto/core/kcf_prov_lib.c (4) M usr/src/uts/common/fs/zfs/abd.c (24) M usr/src/uts/common/fs/zfs/arc.c (1715) M usr/src/uts/common/fs/zfs/bptree.c (3) M usr/src/uts/common/fs/zfs/dbuf.c (241) M usr/src/uts/common/fs/zfs/ddt.c (23) M usr/src/uts/common/fs/zfs/dmu.c (371) M usr/src/uts/common/fs/zfs/dmu_objset.c (303) M usr/src/uts/common/fs/zfs/dmu_send.c (914) M usr/src/uts/common/fs/zfs/dmu_traverse.c (46) M usr/src/uts/common/fs/zfs/dnode.c (111) M usr/src/uts/common/fs/zfs/dnode_sync.c (15) A usr/src/uts/common/fs/zfs/dsl_crypt.c (2640) M usr/src/uts/common/fs/zfs/dsl_dataset.c (131) M usr/src/uts/common/fs/zfs/dsl_destroy.c (14) M usr/src/uts/common/fs/zfs/dsl_dir.c (55) M usr/src/uts/common/fs/zfs/dsl_pool.c (19) M usr/src/uts/common/fs/zfs/dsl_prop.c (3) M usr/src/uts/common/fs/zfs/dsl_scan.c (17) A usr/src/uts/common/fs/zfs/hkdf.c (173) M usr/src/uts/common/fs/zfs/metaslab.c (3) M usr/src/uts/common/fs/zfs/spa.c (82) M usr/src/uts/common/fs/zfs/spa_config.c (2) M usr/src/uts/common/fs/zfs/spa_errlog.c (3) M usr/src/uts/common/fs/zfs/spa_history.c (0) M usr/src/uts/common/fs/zfs/spa_misc.c (0) M usr/src/uts/common/fs/zfs/sys/abd.h (0) M usr/src/uts/common/fs/zfs/sys/arc.h (0) M usr/src/uts/common/fs/zfs/sys/dbuf.h (0) M usr/src/uts/common/fs/zfs/sys/ddt.h (0) M usr/src/uts/common/fs/zfs/sys/dmu.h (0) M usr/src/uts/common/fs/zfs/sys/dmu_objset.h (0) M usr/src/uts/common/fs/zfs/sys/dmu_send.h (0) M usr/src/uts/common/fs/zfs/sys/dmu_traverse.h (0) M usr/src/uts/common/fs/zfs/sys/dnode.h (0) M usr/src/uts/common/fs/zfs/sys/dsl_crypt.h (0) M usr/src/uts/common/fs/zfs/sys/dsl_dataset.h (0) M usr/src/uts/common/fs/zfs/sys/dsl_deleg.h (0) M usr/src/uts/common/fs/zfs/sys/dsl_dir.h (0) M usr/src/uts/common/fs/zfs/sys/dsl_pool.h (0) M usr/src/uts/common/fs/zfs/sys/hkdf.h (0) M usr/src/uts/common/fs/zfs/sys/spa.h (0) M usr/src/uts/common/fs/zfs/sys/spa_impl.h (0) M usr/src/uts/common/fs/zfs/sys/zfs_ioctl.h (0) M usr/src/uts/common/fs/zfs/sys/zil.h (0) M usr/src/uts/common/fs/zfs/sys/zio.h (0) M usr/src/uts/common/fs/zfs/sys/zio_checksum.h (0) M usr/src/uts/common/fs/zfs/sys/zio_crypt.h (0) M usr/src/uts/common/fs/zfs/sys/zio_impl.h (0) M usr/src/uts/common/fs/zfs/vdev.c (0) M usr/src/uts/common/fs/zfs/vdev_raidz.c (0) M usr/src/uts/common/fs/zfs/zfeature.c (0) M usr/src/uts/common/fs/zfs/zfs_fm.c (0) M usr/src/uts/common/fs/zfs/zfs_ioctl.c (0) M usr/src/uts/common/fs/zfs/zfs_vfsops.c (0) M usr/src/uts/common/fs/zfs/zfs_vnops.c (0) M usr/src/uts/common/fs/zfs/zil.c (0) M usr/src/uts/common/fs/zfs/zio.c (0) M usr/src/uts/common/fs/zfs/zio_checksum.c (0) M usr/src/uts/common/fs/zfs/zio_crypt.c (0) M usr/src/uts/common/fs/zfs/zvol.c (0) M usr/src/uts/common/sys/fm/fs/zfs.h (0) M usr/src/uts/common/sys/fs/zfs.h (0) M usr/src/uts/common/sys/mount.h (0) -- Patch Links -- https://github.com/openzfs/openzfs/pull/489.patch https://github.com/openzfs/openzfs/pull/489.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489 ------------------------------------------ openzfs-developer Archives: https://openzfs.topicbox.com/groups/developer/discussions/T60f1bdcc3afc4bfc-Mf102642d319540ee3696d23a Powered by Topicbox: https://topicbox.com
