[developer] Re: May OpenZFS Leadership Meeting

[Matthew Ahrens via openzfs-developer]

Thanks to everyone who participated.  The video recording is now available:
https://youtu.be/GKusXS8T_KY

meeting notes:

   -

   New maintainers (Matt)
   -

  Brian has been the sole maintainer for ZoL -> OpenZFS for almost a
  decade now.
  -

  The volume of contribution and users has been increasing and we need
  more people
  -

  Matt and Tony Hutter will be joining him as maintainers
  -

  Tony Nguyen, John Kennedy, and Mark Maybee will also be helping out
  by ushering incoming PRs through the review process (e.g. finding
  reviewers, applying the appropriate tags, and merging them).
  -

   Settings vdev properties at pool create time (Allan)
   -

  This is not a pressing matter for getting the vdev properties PR
  merged, but if anyone comes up with a good idea on how the
argument parsing
  should be done for this, they should definitely send an email to Allan or
  open a discussion in the mailing list.
  -

   Getting zpool status details via pool properties (scrub progress,
etc) (related
   issue ) (Allan)
   -

  Some of these properties you can get with the vdev properties PR as
  it is right now but certain pool properties like a scan estimate or other
  things that admins see from commands like zpool-status are harder to get.
  -

  It would be useful to expose some of these properties through
  zpool-get as-is or at least parts of them.
  -

   If we add a bunch more pool and dataset properties, do we need a new
   ‘verbose’ type or something, where a lot of these new (not for human
   consumption) properties are not displayed by default?
   -

  zpool/zfs get all has been pretty verbose in general as more
  properties are piling up on it and its output is not very
friendly in terms
  of parsing it programmatically
  -

  Allan and Matt are considering a potential alternative command option
  that’s more human readable and outputs some important properties (not all
  of them).
  -

   Userland zfs (PR ) (Mayank)
   -

  The above PR needs reviewers
  -

  There is a lot more work to be scoped and done. Ideally the
  upstreaming work from the cStor folks would involve incremental
PRs adding
  more IOCTLs or other minimal functionality that can be tested
independently
  in the userland and slowly become part of the existing testing
  -

 Allan brought up an old idea of the userland send/receive tests
 -

 Other options include but are not limited to Matt’s old
 socket-based userland ZFS testing.
 -

  Keeping in mind FUSE and laying the groundwork for it during this
  effort should also provide some long-term benefits and flexibility as we
  discover more ways on how to bring the kernel code to the userland.
  -

  There is a lot of ground to be covered but given the excitement there
  should be volunteers to help. If you are interested please reach out to
  Mayank, Matt, or Serapheim.
  -

   Corrective Recv PR  (Alek)
   -

  The above PR seems to be passing our tests and needs
  reviewers/reviews.
  -

   FreeBSD Summit
   -

  Matt will be giving a talk about RAIDZ expansion
  -

  There will be other storage topics covered outside of ZFS for folks
  that are interested
  -

   ZFS support for LXD and similar container technologies (Allan)
   -

  A PR should be out soon (probably next week)
  -

  There will be a need for both reviewers and testers
  -

   ZFS Forced Export (Allan)
   -

  Tests are passing and a PR should be open soon
  -

   OSX port (Jorgen)
   -

  Around 71% of the tests currently pass and we’d like to start opening
  PRs soon
  -

  Besides the above, one thing that is pending in terms of
  infrastructure is figuring out which solution to use for incorporating
  testing for the new OS in our CI/CD. Github Action seems to provide some
  MacOS instances but there may be a timeout that’s not enough for
us to run
  ZTS. AWS is another option.
  -

   Add Blake3 Checksum (PR )
   (Tino)
   -

  Sorry, did not have time for this currently (will do it in the next 4
  weeks)
  -

  I got an account on the gcc farm project, so assembler routines for
  sha2 will also come from me (aarch64, ppc, x64) … together with
blake3 and
  benchmarks
  -

  (Moving as an agenda item for next time)


On Mon, May 24, 2021 at 10:22 PM Matthew Ahrens  wrote:

> The next OpenZFS Leadership meeting will be held tomorrow, May 25, 1pm
> Pacific time.  Topics on the agenda include:
>
>- Settings vdev properties at pool create time
>- Getting zpool status details via 

[developer] Re: May OpenZFS Leadership Meeting

[Matthew Ahrens via openzfs-developer]

Thanks to everyone who participated at this month's meeting.  The video
recording is now available:
https://www.youtube.com/watch?v=SyNn26YbRZY

Meeting notes (thanks Serapheim):

   -

   OpenZFS conference
   -

  Will be held virtually/online this year
  -

  Currently planned at the end of September
  -

 there is flexibility as people won’t be travelling
 -

 higher attendance is expected for the same reason
 -

  Talk Submissions is opening next week
  -

  Format will be roughly the same as other years
  -

  We still haven’t decided on the platform to host it
  -

 We want to hear from others: what conference tools and formats
 worked on other conferences (or didn’t) - contact Karyn or Matt
 -

   Rebuild PR (Brian)
   -

  Work came out as preparatory work for the DRAID projects but for
  mirrors
  -

  The PR as it stands introduces a new option (-R) for the zpool
  replace and zpool attach commands that performs the rebuild and provides
  redundancy
  -

  The caveat is that no checksums are verified but for that same reason
  it can be a lot faster (especially for fragmented pools and/or
small block
  sizes)
  -

  Each vdev can be rebuilt independently, so you can have multiple
  operations going on, unlike resilver which is pool-wide.
  -

  The code used to be part of resilver but then it was reworked to have
  its own standalone infrastructure.
  -

  Looking for feedback: https://github.com/openzfs/zfs/pull/10349
  -

   Documentation system (George Melikov)
   -

  https://openzfs.github.io/openzfs-docs/
  -

  Short-term goal: Move Linux Wiki to the above repo and add automation
  -

  Long-term goal: Have all documentation at one place (e.g. man pages,
  content from the media wiki, etc..)
  -

  For folks interested in migrating the media wiki content feel free to
  ping Karyn, Michael Dexter, or Matt to see what content is still relevant
  and what is stale.
  -

  Open to feedback on what format and tools to use or other ideas
  -

  Several concerns with using RST as the source for manpages.
  -

   POSIX AIO for async dmu? (Matt Macy)
   -

  “wondering if it's worth integrating async dmu with [POSIX AIO] as
  opposed to just using [async dmu] for zvol”
  -

  Some work was started by SpectraLogic years ago
  -

  The goal is to expose the asynchronous ZIO interfaces at the dmu
  level and integrate it to ZVOLs or the POSIX AIO interface (standard
  syscalls like POSIX reads cannot take advantage of this).
  -

  Regarding POSIX AIO the questions are:
  -

 Is anyone already using these interfaces? If so, what’s their
 experience?
 -

 Would people be interested in using it if it existed? If so, for
 what use-case?
 -

   Upstreaming “forced export” feature (Allan)
   -

  Work for this project will be starting soon
  -

  Use-Case: Problems in connectivity with remote vdevs would suspend
  the pool (and potentially block other pools when higher-level locks are
  grabbed)
  -

  Proposal: zpool export -F - force the export of that pool so
  everything goes back to normal.
  -

   Formalizing pad2, bootonce/nextboot/rescue counter (Allan)
   -

  An interface introduced recently regarding this functionality only
  accepts a character array as an argument (in libZFS_Core). The proposal
  here is to either change this existing call or add a new one that accepts
  an nvlist instead to provide more flexibility for new features.
  -

   OS-specific properties (Sean)
   -

  PR is out: https://github.com/openzfs/zfs/pull/10111
  -

  Deals with the issue of os-specific properties varying wildly in
  their behavior between platforms while there is room for normalization.
  -

  Needs Reviewers/Feedback


On Mon, May 25, 2020 at 3:44 PM Matthew Ahrens  wrote:

> The next OpenZFS Leadership meeting will be held tomorrow, May 26, 1pm-2pm
> Pacific time.  We have several topics on the agenda for tomorrow's meeting,
> let me know if you'd like to add anything:
>
>
>-
>
>Documentation system
>-
>
>POSIX AIO for async dmu?
>-
>
>Formalizing pad2, bootonce/nextboot/rescue counter (Allan)
>-
>
>Remove uncompressed ARC? (Matt)
>-
>
>   See previous discussion
>   
>   -
>
>Rebuild PR (Brian)
>-
>
>Send toggle -L fix (Matt)
>
>
> Everyone is welcome to attend and participate, and we will try to keep the
> meeting on agenda and on time.  The meetings will be held online via Zoom,
> and recorded and posted to the website and YouTube after the meeting.
>
> The agenda for the 

[developer] Re: May OpenZFS Leadership Meeting

[Matthew Ahrens]

Thanks everyone who participated in yesterday's meeting.  The video
recording is now up: https://youtu.be/EzyPr3BuePo

The next meeting will be June 25, at 1pm pacific time.

The meeting notes follow:

   -

   Happy ZoL 0.8!
   -

  Lots of downloads and excited people. Successful so far!
  -

   Illumos encryption status (Jerry/Jason/Jörgen)
   -

  Jerry has all of Jörgen’s work pulled over and building on illumos.
  Resolving some test failures and then will put it out for an illumos code
  review.
  -

  The code review will go out to the illumos developer mailing list and
  ask that they identify issues with the platform-dependent changes in the
  port. People will have ample time to comment and raise issues before the
  RTI is submitted.
  -

   RAIDZ Expansion status (Matt)
   -

  Will have an Alpha out this week or next. Pools will need to be
  destroyed if you use it.
  -

  The change lets you add a device to an existing pool and reflows the
  data between disks.
  -

  All of the caveats and details of what does and doesn’t work will be
  included in the PR.
  -

  Call for testing! Once the code is up, any manual testing under
  different workloads and setups will be greatly appreciated. It would also
  be great if people could help write systemic tests.
  -

   BSDCAN update
   -

  The future of ZFS on FreeBSD
   (Allan)
  -

 One of the goals of the talk was to motivate the ZFS repo of
 FreeBSD rebasing over ZoL. The advantages of having the
consumers of ZFS
 closer together was highlighted.
 -

 There was some initial pushback people thought that the rebase
 will make ZFS use the Linux Compatibility layer of FreeBSD,
which is not
 the case. Once this was clear, there were no further pushbacks
 -

 Matt brought up that once the rebase is done and stable,
 discussions may be initiated for potentially renaming the ZoL repo.
 -

  snapshot scalability
   (Matt)
  -

 Slides are posted and video will be up eventually. Please check
 them out if you’re interested in learning more about snapshots.
 -

 Matt will post a link to #openzfs once he has one.
 -

   DRAID meeting
   

   recap (Richard Elling)
   -

  There were about 11 participants
  -

  There is a design doc (linked above) with all the history of its
  edits based on the discussions that took place. Please add
comments to the
  doc to further the discussion.
  -

  There is still work to be done but since the code changes are
  sizeable, we’ll try to split them into individual commits when it makes
  sense.
  -

 Sef says he is available to do reviews at any time, just reach out.
 -

   CoC update (Matt/Karyn)
   -

  Code of Conduct was put into place since the last meeting. We made
  several changes based on community feedback. Thanks for participating!
  -

  We will have a call for nominees for the Working Group in the fall,
  and then will review the CoC at that point and have another community
  comment period. The goal is to complete this before the Dev Summit.
  -

   The Dev Summit is Nov 4-5 in SF. See open-zfs.org
    for upcoming
   dates (e.g., call for presentations).
   -

  Registration opens July 8th
  -

  Send presentations to Matt by August 19th
  -

  Thanks to Datto, Delphix, and Intel for already sponsoring! Reach out
  to @kritter if you want to sponsor and she can give you the details.
  -

   Request from sef! The On-Disk Format
    document is out of date,
   and doesn’t reflect recent changes (e.g., encryption). It would be really
   helpful to update this and keep it up-to-date going forward (Perhaps in a
   semi-automated way?)
   -

  Tom: The info about how things look today exists in various places,
  so it is just a question of pulling it together into a single place.
  -

  Matt: We only have a pdf for that, and we probably can’t just
  copy-paste it somewhere else. It would be good to have a
document with this
  info. Let’s put out a call for a volunteer to do this.
  -

 Paul pointed out that it is licensed under a Berkeley License
 

 -

   Compressed arc performance issue - single threaded sequential read
   -

  Sequential reads are decompressed by the (one) user’s thread.