[developer] Re: December OpenZFS Leadership Meeting

[Matthew Ahrens via openzfs-developer]

Thanks to everyone who participated.  The video recording is now available:
https://www.youtube.com/watch?v=_097QvCwT4Y

Meeting notes (thanks Christian!):


   -

   ZFS Interface for Accelerators (“Z.I.A.”); Jason Lee (Los Alamos
   National Laboratory)
   -

  Slides
  
  -

  Summary: make use of hardware accelerators for compression, checksum,
  raidz
  -

  Main idea:
  -

 Data is owned by either the Host CPU or the accelerator.
 -

 ZIO pipeline refers to data owned by the accelerator through a
 “handle”.
 -

 If the accelerator doesn’t support one step of the pipeline, data
 is moved back to the host for that step. But as long as the
next step is
 supported, we can avoid copying back-and-forth.
 -

 Discussion about what safeguards should be put in place to keep
 track of ownership / source-of-truth of the data. E.g.,
during ABD copy.
 -

  Software architecture provider/consumer of accelerators:
  -

 “Data Processing Unit Services Module” decouples consumers from
 providers
 -

 Non-Restrictive License towards consumers; GPLv2 towards providers
 -

  QA / Discussion
  -

 Watch out for byte-for-byte compatibility, otherwise incompatible
 with “uncompressed ARC + L2ARC” use case.
 -

 Maintainability / Testing: at least a software provider should be
 part of the OpenZFS repo so that we can prevent accidental
breakage (e.g.
 when changing ZIO pipeline implementation)
 -

 Public availability of actual hardware & corresponding providers
 would be critical for upstreaming.
 -

 Discussion on resource contention (e.g. two zpools sharing one
 accelerator)
 -

   Native encryption needs some work (Rich Ercolani)
   -

  Spreadsheet of encryption bugs
  -

  Biggest and scariest: incorrect dnode refcounting
  -

 Panic stacks look very different, trapping at different ASSERTs
 -

 Not reproducible on x86(_64), but on SPARC and on PowerPC
 -

 Reproduces within 24 hours on a PowerPC 64 KVM VM (but not
 qemu-emulated)
 -

  Earliest occurrence dates all the way back to introduction of native
  encryption
  -

  Other scary category: encrypted send/recv corrupts receive-side
  dataset
  -

 One can be triggered by zfs change-key => will use wrong wrapping
 key
 -

  Problem: the community is not aware of a company that could fund
  fixing this.
  -

  Strategy: bring it up at future meetings to raise awareness.
  -

   Request for reviews:
   -

  Allan: calculate a more realistic allocation inflation multiplier:
  https://github.com/openzfs/zfs/pull/12773

Alek: healing recv patch - https://github.com/openzfs/zfs/pull/9372

On Mon, Dec 6, 2021 at 9:11 AM Matthew Ahrens  wrote:

> The next OpenZFS Leadership meeting will be held tomorrow, December 7, 9am
> Pacific time.
> Everyone is welcome to attend and participate, and we will try to keep the
> meeting on agenda and on time.  The meetings will be held online via Zoom,
> and recorded and posted to the website and YouTube after the meeting.
>
> For more information and details on how to attend, as well as notes and
> video from the previous meeting, please see the agenda document:
>
>
> https://docs.google.com/document/d/1w2jv2XVYFmBVvG1EGf-9A5HBVsjAYoLIFZAnWHhV-BM/edit
>
> --matt
>

--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/T24bdaa2886c6cbf5-Mc039a11c3f1507ea0664817b
Delivery options: https://openzfs.topicbox.com/groups/developer/subscription

[developer] Re: December OpenZFS Leadership Meeting

[Matthew Ahrens via openzfs-developer]

Apologies for the delay, here are the video recording and topics from last
month's meeting:

https://www.youtube.com/watch?v=ZUOBo8X5VY4=youtu.be

   -

   OpenZFS 2.0 released!
   -

   DRAID integrated to master!
   -

   Further review requested for forced-export
   https://github.com/openzfs/zfs/pull/11082
   -

   RAIDZ expansion performance
   -

   Visibility of .zfs/snapshot inside a FreeBSD jail


On Mon, Dec 7, 2020 at 1:03 PM Matthew Ahrens  wrote:

> The next OpenZFS Leadership meeting will be held tomorrow, December 8,
> 1pm-2pm Pacific time.  We don't have many topics on the agenda for
> tomorrow's meeting, so let me know if you'd like to add anything.
>
> Everyone is welcome to attend and participate, and we will try to keep the
> meeting on agenda and on time.  The meetings will be held online via Zoom,
> and recorded and posted to the website and YouTube after the meeting.
>
> For more information and details on how to attend, as well as notes and
> video from the previous meeting, please see the agenda document:
>
>
> https://docs.google.com/document/d/1w2jv2XVYFmBVvG1EGf-9A5HBVsjAYoLIFZAnWHhV-BM/edit
>
> --matt
>

--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/T24bdaa2886c6cbf5-M6215b720b46f19b08d818d58
Delivery options: https://openzfs.topicbox.com/groups/developer/subscription

[developer] Re: December OpenZFS Leadership Meeting

[Matthew Ahrens via openzfs-developer]

The next OpenZFS Leadership meeting will be held tomorrow, December 8,
1pm-2pm Pacific time.  We don't have many topics on the agenda for
tomorrow's meeting, so let me know if you'd like to add anything.

Everyone is welcome to attend and participate, and we will try to keep the
meeting on agenda and on time.  The meetings will be held online via Zoom,
and recorded and posted to the website and YouTube after the meeting.

For more information and details on how to attend, as well as notes and
video from the previous meeting, please see the agenda document:

https://docs.google.com/document/d/1w2jv2XVYFmBVvG1EGf-9A5HBVsjAYoLIFZAnWHhV-BM/edit

--matt

--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/T24bdaa2886c6cbf5-M0331e42275385aa79616d7d9
Delivery options: https://openzfs.topicbox.com/groups/developer/subscription

[developer] Re: December OpenZFS Leadership Meeting

[Matthew Ahrens]

Thanks to everyone who participated at this month's meeting.  We discussed:

   - saved send feature
   - ZSTD
   - Encrypted dedup

The recording is now available: https://www.youtube.com/watch?v=Wq64VoZhDMY

Thanks Serapheim for taking notes:

   -

   Saved send feature (Tom)
   -

  Be able to get a partially received dataset from one machine to
  another - do a ZFS “send of a partially received dataset”
  -

  Add-on functionality: ability to resume a saved receive from a
  bookmark
  -

  Status: Testing is almost done and Datto will soon start using it in
  production
  -

  Looking for extra reviewers for the PR -
  https://github.com/zfsonlinux/zfs/pull/9007
  -

 Paul Dagnelie and Matt Ahrens will take a look
 -

  Action Item: Send a heads up in the mailing list about the feature
  for other platforms
  -

   ZSTD rework (WIP PR )
   (Kjeld)
   -

  The goal of this new PR is to get the best of all the existing forks
  of this feature
  -

  Status: Sebastien Gottschall (Brainslayer) have completed the
  majority of the design and thorough testing has been done. Code is being
  cleaned up and restructured.
  -

  Action Item: Sync with Allan who has also done some work with this
  -

  Action item: Get reviewers and feedback for the change
  -

   Feature Request: Encryption to work with dedup across multiple datasets
   - Tom:
   -

  Today different “clone families” have different master keys (the key
  actually used to encrypt the blocks), so blocks with the same plaintext
  will have different cyphertext if they are in the same clone
family - even
  if they have the same wrapping (user) key (i.e. same/inherited keysource
  property)
  -

  Want to add a mechanism to have the same master key for different
  clone families
  -

  Need to design user interface to make it clear what’s going on.
  -

  Suggestion: use a property to indicate that all children have the
  same master key
  -

 Need to work out the details of how this would interact with
 things like rename (into / out of the “same master key hierarchy”)
 -

   Pull Request Open for Persistent L2ARC in ZoL - Brian Behlendorf:
   -

  This is a port of Sasso’s/Nexenta’s work (PR 9582)
  -

  Reviews requested
  -

   ZoF update:
   -

  Getting closer! - Bulk of refactoring is done ~ approximately 4 files
  left to go



On Mon, Dec 9, 2019 at 1:23 PM Matthew Ahrens  wrote:

> The next OpenZFS Leadership meeting will be held tomorrow, December 10,
> 1pm-2pm Pacific time.  The agenda is pretty light this month, so if you
> have any topics you'd like to discuss with the group, tomorrow will be a
> good opportunity :-)
>
> Everyone is welcome to attend and participate, and we will try to keep the
> meeting on agenda and on time.  The meetings will be held online via Zoom,
> and recorded and posted to the website and YouTube after the meeting.
>
> The agenda for the meeting will be a discussion of the projects listed in
> the agenda doc.
>
> For more information and details on how to attend, as well as notes and
> video from the previous meeting, please see the agenda document:
>
>
> https://docs.google.com/document/d/1w2jv2XVYFmBVvG1EGf-9A5HBVsjAYoLIFZAnWHhV-BM/edit
>
> --matt
>

--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/T24bdaa2886c6cbf5-Me33e70ccb5f325ac00e088e4
Delivery options: https://openzfs.topicbox.com/groups/developer/subscription