[developer] Re: February OpenZFS Leadership Meeting

[Matthew Ahrens via openzfs-developer]

>
> Thanks to everyone who participated.

Video recording 
agenda:

   - Update ZFSd to match recent new features added to ZED (vdev properties
   for io/checksum/slow counts that cause device replacement):
   https://reviews.freebsd.org/D44043
   - Fast Dedup is up for review:
   https://github.com/openzfs/zfs/discussions/15896
   - Don’t stall MMP thread on slow I/O:
   https://github.com/openzfs/zfs/pull/15839
   - Support snapdir=disabled: https://github.com/openzfs/zfs/pull/15891
   - Klara has started work on improving arc_evict() for large memory
   systems
   - FYI, (0) days since last block cloning bug (
   https://github.com/openzfs/zfs/issues/15933)


On Mon, Feb 26, 2024 at 8:21 AM Matthew Ahrens  wrote:

> The next OpenZFS Leadership meeting will be held tomorrow, February 27th
> at 1pm Pacific time.
> Everyone is welcome to attend and participate, and we will try to keep the
> meeting on agenda and on time.  The meetings will be held online via Zoom,
> and recorded and posted to the website and YouTube after the meeting.
>
> For more information and details on how to attend, as well as notes and
> video from the previous meeting, please see the agenda document:
>
>
> https://docs.google.com/document/d/1w2jv2XVYFmBVvG1EGf-9A5HBVsjAYoLIFZAnWHhV-BM/edit
>
> --matt
>

--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/T7ea98f99753036e8-M935278348cc07044437b83c7
Delivery options: https://openzfs.topicbox.com/groups/developer/subscription

[developer] Re: February OpenZFS Leadership Meeting

[Matthew Ahrens via openzfs-developer]

Thanks to everyone who participated.  Sorry for the delay in getting this
email out!
Meeting recording is posted: https://youtu.be/hij7PGGjevc

Notes (thanks Christian):

   -

   Update on BRT (Block Reference Table) (Pawel)
   -

  Pawel claims 98% ready.
  -

  Design changes since last update
  -

 If a block is already in dedup table, we now bump the dedup
 refcount instead of starting a second reference counter in the BRT.
 -

 ZIL: now implemented. Log records just use block pointers. No use
 of object IDs, since ZIL is per-objset, but BRT operation
logged in the ZIL
 depends on state of source and dest objset.
 -

  Further potential design changes discussed in the meeting:
  -

 Have a per-VDEV BRT to avoid repeating the VDEV ID in each entry.
 -

=> on-disk & memory space savings
-

  Discussion about generality of the implementation
  -

 Current implementation defines two new FreeBSD-specific syscalls
 -

 Having the Linux copy-reflink IOCTL would make the mechanism “just
 work” with existing Linux tools.
 -

 Also, there’s Linux copy_file_range, which “gives filesystems an
 opportunity to implement "copy acceleration" techniques, such
as the use of
 reflinks.
 -

 The MacOS syscall is path-based, but Lundman says in chat that the
 filesystem implements VNOP_CLONEFILE, which operates on vnodes.
 -

 Ideas about a command line tool / daemon to trigger dedup/BRT
 offline (i.e., after duplicates were written).
 -

  Timeline
  -

 Pawel first needs https://github.com/openzfs/zfs/pull/13027 to be
 merged.
 -

 Then he can make a PR.
 -

 Code will need more reviewers.
 -

   Encryption Bugs (Rich Ercolani)
   -

  (Was not able to make it to the meeting, summary written ahead of
  time.)
  -

  One set of bugs (#12981 
  et al) got a workaround from George Amanakis - thanks George!
  -

  Another bug (#12720 )
  manifested as an error during raw send/recv. The underlying
cause is faulty
  on-disk dnodes with contradicting bonuslength/spill pointer flag
in < 0.6.4
  versions of OpenZFS. Related PR #13014
  .
  -

  WIP PR #12943  for issue
  #11679  had an issue
  reported, going to try and ameliorate with even more locking,
but could use
  someone with familiarity with send/recv a/o the ARC code to help, as I’m
  pretty sure this is just papering over something being done incorrectly.
  -

 Issue #11679 has drawn lots of attention. Someone familiar with
 the DMU should take a look.
 -

  WIP I should circulate for extending FORCE_INHERIT/FORCE_NEW_KEY to
  allow you to escape situations like #12614
  , need to write more
  tests, feel free to ping me if you’re in this situation and want
to try it.
  (Also trying to figure out what a reasonable thing to do in most
cases when
  you receive a change-key in an incremental send is - so far, all of the
  options seem to violate POLP sometimes.)
  -

 Downside: not insignificantly sized foot-gun to allow you to
 change-key -f
 -

 Storing last N copies of the wrapped key and trying them all would
 help you avoid that, but then you have N ways to unlock the key…
 -

  Someone reported issues with receiving unencrypted under an encrypted
  and not unlocked parent (#13033
  ; not data loss or
  anything, just mentioning for completeness)
  -

   Add Blake3 Checksum (PR #12918
   ) (Tino)
   -

  Are there open issues before inclusion?
  -

  Benchmarks
  
  (Unit is MiB/s, more is better)
  -

  Is Blake3 a “strong” checksum, in the ZFS sense (= is it dedup-safe)?
  -

 It is a cryptographic checksum
  .
 -

 The patch adds blake3 to the list of supported dedup checksums
 -

   Looking for additional reviews:
   -

  https://github.com/openzfs/zfs/pull/12773 (reduced spa_inflation)
  -

  https://github.com/openzfs/zfs/pull/12868 (write smoothing)
  -

  https://github.com/openzfs/zfs/pull/12263 (linux user namespace
  support)
  -

  https://github.com/openzfs/zfs/pull/12789 (log spacemap load 

[developer] Re: February OpenZFS Leadership Meeting

[Matthew Ahrens via openzfs-developer]

Thanks to everyone who participated last week.

The video is posted: https://youtu.be/VmXoHO2PDP4

meeting notes:

   -

   Seagate command priority call Feb 3
   -

  Mentioned in the last OpenZFS meeting
  -

  Get in contact with Muhammad if interested
  -

  WIll take place 2/3/2021 11:30 PST
  -

   OpenZFS Versioning Scheme and what it means
   -

  Motivation: Folks were interested in learning when would dRAID be
  included in an official release
  -

  Brian will put something together for the next meeting to be discussed
  -

   Status of specific PRs:
   -

  DirectIO
  -

 Brian Atkinson is still working on it. We expect it to be updated
 soon, at which point we’ll need reviewers
 -

  Pool Feature Compatibility Options during pool creation
  -

 Still ongoing, needs reviewers
 -

  Force Pool Export
  -

 Should be good to go soon. We’d like to get more reviewers as the
 PR touches many different parts of the codebase.


On Mon, Feb 1, 2021 at 9:48 AM Matthew Ahrens  wrote:

> The next OpenZFS Leadership meeting will be held tomorrow, February 2,
> 1pm-2pm Pacific time.  We don't have many topics on the agenda for
> tomorrow's meeting, so let me know if you'd like to add anything.
>
> Everyone is welcome to attend and participate, and we will try to keep the
> meeting on agenda and on time.  The meetings will be held online via Zoom,
> and recorded and posted to the website and YouTube after the meeting.
>
> For more information and details on how to attend, as well as notes and
> video from the previous meeting, please see the agenda document:
>
>
> https://docs.google.com/document/d/1w2jv2XVYFmBVvG1EGf-9A5HBVsjAYoLIFZAnWHhV-BM/edit
>
> --matt
>

--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/T7ea98f99753036e8-Mbb1cecd282a40cb19b2dd843
Delivery options: https://openzfs.topicbox.com/groups/developer/subscription

[developer] Re: February OpenZFS Leadership Meeting

[Matthew Ahrens]

Thanks to everyone who participated at this month's meeting.  We discussed:

   - Thread priorities on linux
   - Feature flag activation bug
   - Moving of the OpenZFS website to University of Washington
   - FOSDEM, Scale, and other conferences


The recording is now available: https://www.youtube.com/watch?v=upUOXBhx-us

Thanks Serapheim for taking notes:

   -

   Thread priorities on linux (Paul D)
   -

  Context: Performance analysis of ZFS send on ZFSonLinux found
  discrepancies with illumos
  -

  Root cause: Linux the threads are minimum priority, lower than user
  threads (as opposed to Illumos)
  -

  Workaround: Increased the priority for now
  -

  Question: How do we want to deal with this among different OS’s?
  -

 FreeBSD has the same priority scheme as illumos (Linux being the
 outlier of the 3)
 -

 In Linux there were performance issues related with thread
 priorities in the past but there hasn’t been any recent investigations
 -

 There is no silver bullet for this
 -

  Consensus:
  -

 There are not many thread cases whose performance we don’t care
 about but we should at least break down threads into
different groups and
 decide on the priority of each group
 -

 This issue is to be added to the list of issues that we need an
 open PR on GIthub for discussion
 -

   Do we need to enhance the feature flag activation code. Setting
   checksum=sha512 or compress=zstd but not writing any blocks, can cause the
   pool to panic on older systems
   -

  Summary: Setting the compression to zstd but you don’t write any
  blocks, and then re-import that pool in a ZFS version that doesn’t
  understand zstd triggers an assertion failure
  -

  Consensus:
  -

 Bump the counter when the feature gets activated AND for each
 block created with it (refcount never goes to zero until the
dataset is
 destroyed).
 -

 Although seemingly difficult to implement we should at least pay a
 close look at the code paths as it may not be as intractable
as it seems.
 -

   Support for ignoring (not being able to mount) datasets with unsupported
   feature flags
   -

  Consensus: This is a good idea in general but it should not be
  considered a solution for the aforementioned zstd problem.
  -

   Moving of the OpenZFS website from Joyent to University of Washington is
   in-progress
   -

  No problems with Joyent, just planning for the long-term
  -

  No actual website changes, just the DNS endpoint will change
  -

  Still would be happy to update any contributions to update the
  website’s content :)
  -

   Changes that need reviewers:
   -

  Persistent L2ARC - https://github.com/zfsonlinux/zfs/pull/9582
  -

  Performance Optimization for Encryption
  -

 Includes new algorithms and changing the default algorithm used
 -

 Action Item: Send an email to the mailing list as a heads up about
 changing the default before going ahead an applying that change
 -

   FOSDEM, Scale, and other conferences
   -

  There doesn’t seem to be any talks related to ZFS in the
  aforementioned conferences but people attending can submit BoF discussions
  -

 iXsystems will potentially be at Scale this year
 -

  Allan Jude has done a ZFS-related talk in the main track of FOSDEM in
  the past
  -

  There will be a ZFS BoF in this year’s BSDCan and potentially one
  ZFS-related talk too.
  -

  The BSDNow podcast is open to interview people doing interesting work
  on ZFS
  -

  General Idea: It would be good to be active in conferences and groups
  outside of the OpenZFS summit and BSD
  -

 On this note, OpenZFS is considering spinning up funds for people
 that want to work on that (e.g. setting up a booth on those
conferences and
 getting the word out).


On Mon, Feb 3, 2020 at 1:26 PM Matthew Ahrens  wrote:

> The next OpenZFS Leadership meeting will be held tomorrow, February 4,
> 1pm-2pm Pacific time.  The agenda is pretty light this month, so if you
> have any topics you'd like to discuss with the group, tomorrow will be a
> good opportunity :-)
>
> Everyone is welcome to attend and participate, and we will try to keep the
> meeting on agenda and on time.  The meetings will be held online via Zoom,
> and recorded and posted to the website and YouTube after the meeting.
>
> The agenda for the meeting will be a discussion of the projects listed in
> the agenda doc.
>
> For more information and details on how to attend, as well as notes and
> video from the previous meeting, please see the agenda document:
>
>
> 

[developer] Re: February OpenZFS Leadership Meeting

[Matthew Ahrens]

At this meeting we discussed:

   - who will review Fast Clone Deletion
   - FIPS 140-2 certification
   - making compressed ARC mandatory
   - platform-specific 'sharenfs' property


video recording available: https://www.youtube.com/watch?v=EXstK9ckcZQ

meeting notes (thanks Karyn!):

   -

   Reviewers for fast clone deletion (ZoL PR
   ; illumos PR
   ) (Sara)
   -

  There is a feature flag change that Sara sent email out about.
  -

  Sara is seeking reviewers.
  -

 Brian volunteered to review, initial pass looks great
 -

  How much review is needed?
  -

  Will have some conflicts with bpobj iteration work.
  -

  BB: Wasn’t planning to get this in before 0.8, but if it would be
  useful it is possible.
  -

 Let’s get it in right after 0.8 (which is due out in March)
 -

 There are a few fixes that are pending for 0.8 to go out. Matt
 will look at them!
 -

   FIPS 140-2 certification (Luke)
   -

  Defense contractors could use ZFS for many things, but require FIPS.
  Other industries (e.g., healthcare) also have this requirement.
  -

 JC: Can you get certification for a source or is it a specific
 binary build?
 -

 Rainbow: It is for specific binary builds, and she does a lot of
 compliance and can help here. You can do source code level
certification.
 -

 sef: It is really expensive and time consuming. Level of
 configuration for testing and certification is super specific.
 -

 BB: We do have binaries from companies like RHEL, but they aren’t
 official builds.“FIPS verified” rather than certified? We’re
already using
 the appropriate crypto algorithm.
 -

 PD: Certifying at the source level makes it easier for a vendor to
 get certified. There are some additional components that
would probably
 need to be looked at (like hash algorithms).
 -

 MA: Certification applies to the crypto algorithm. Does that help
 us since it is a separate module.
 -

 AJ: Different Linux distros will have different binaries that
 would need to be certified separately.
 -

  Luke can connect with Rainbow and sef to see what would need to be
  done to see if it is viable.
  -

   Should compressed ARC be mandatory? (Issue
   ) (Allan J)
   -

  Does anyone turn off compressed ARC? If not, we can avoid special
  cases for this.
  -

  Please let Allan know right away if you do turn off compressed ARC.
  Else that functionality may just be taken out.
  -

 MA: Seems like there are some cases on Linux where we can be
 confident that people aren’t using this combination (i.e., it doesn’t
 work), but that doesn’t cover all cases.
 -

 AJ: The crypto changes definitely made it different than what was
 in FreeBSD.
 -

 JC: FWIW, compressed ARC makes ARC better in many different ways
 in Postgres (at least). They haven’t noticed any latency increases or
 memory overhead that has been called out.
 -

 AM: It is pretty pointless to disable compressed ARC. The
 difference when you disable it and use other mechanisms
(e.g., bcopy), is
 negligible and there are other benefits to keep it on.
 -

  Please comment in github!
  -

  There was no significant negative feedback, so we plan to move
  forward with making compressed ARC mandatory.
  -

   Platform-specific sharenfs (George)
   -

  Sent this out the proposal last night.
  -

  Create platform-specific properties. These platform-specific
  properties won’t take effect when importing the pool on a different
  platform.
  -

 MA: ‘sharenfs’ is a system property because ZFS takes action based
 on it (e.g. share/unshare when you do ‘zfs rename’).  It should be
 platform-specific because the value of the property isn’t
 verified/interpreted by ZFS - it’s passed to the OS-specific
share utility
 without modification.
 -

 AJ: Cross-OS import is a feature I’d like to keep as a 1st class
 citizen.
 -

 MA: JC provided feedback on the proposal. Please talk about your
 counter-proposal.
 -

 JC: Biggest difference was to keep the functionality the same if
 people really want it, but there would be a “veneer”
interface that would
 allow platform-specific properties. Generally people would just use
 whatever properties on their OS. Garrett’s comments are an
 appliance-centric view. The key bit of the proposal is to
ensure