Re: [ima-evm-utils: PATCH v1 1/1] Change license to LGPL-2.0-or-later and GPL-2.0-or-later

[George Wilson]

On Wed, Feb 21, 2024 at 10:11:34AM +0200, Dmitry Kasatkin wrote:
> Currently libimaevm provided by this project is used by the tool evmctl,
> which is also provided by this project.
> 
> An issue was reported about using libimaevm with other software. Its
> GPL2-only license makes it incompatible to use with other licenses, in
> particular GPL3-only.
> 
> To address this issue, change the project license to GPL-2.0-or-later
> and libimaevm to LGPL 2.0 or later.
> 
> Signed-off-by: Dmitry Kasatkin 

Acked-by: George Wilson 

Re: Fwd: [PowerPC] GCM optimization

[George Wilson]

On Tue, Dec 01, 2020 at 07:55:05PM +0200, Maamoun TK wrote:
> Hi George,
> I'll start writing a white paper called "Optimizing Galois-Counter-Mode on
> PowerPC Architecture Processors". Once I finish the first draft I'll send
> it to Neils to review it.
> 
> 
> > What do you need from the IBM side?  I may be able to help.  We'd
> > definitely
> > like to support you and Niels in publishing your results.
> >
> 
> I have a couple of questions:
> Should we send the paper to you when we make sure everything is ready for
> publishing?

Yes, we'd love to review it.  I may be able to get someone from IBM Research
interested in participating as well to hopefully get deeper crypto and math
perspectives.

> Can you participate as a supervisor to make some decisions like mentioning
> arm implementation results of the research or making comparison with intel
> white papers which used for x86, ARM, and PowerPC GCM implementations in
> OpenSSL library?

Gladly.

> 
> regards,
> Mamone

-- 
George Wilson
IBM Linux Technology Center
Security Development
___
nettle-bugs mailing list
nettle-bugs@lists.lysator.liu.se
http://lists.lysator.liu.se/mailman/listinfo/nettle-bugs

Re: Fwd: [PowerPC] GCM optimization

[George Wilson]

On Thu, Nov 12, 2020 at 07:45:14PM +0200, Maamoun TK wrote:
> -- Forwarded message -
> From: Maamoun TK 
> Date: Thu, Nov 12, 2020 at 7:42 PM
> Subject: Re: [PowerPC] GCM optimization
> To: Niels Möller 
> 
> 
> On Thu, Nov 12, 2020 at 6:40 PM Niels Möller  wrote:
> 
> > I gave it a test run on gcc112 in the gcc compile farm, and speedup of
> > gcm update seems to be 26 times(!) compared to the C version.
> >
> 
> That's reasonable, I got similar speedup on more stable POWER instances
> than gcc compile farm.
> 
> 
> > Where would that documentation be published? In the Nettle manual, as
> > some IBM white paper, or as a more-or-less academic paper, e.g., on
> > arxiv? I will not be able to spend much time on writing, but I'd be
> > happy to review.
> >
> 
> I'll start writing the papers once I got more details from IBM, similar to
> intel documents, the document will be academic and practical at the same

Hi Mamone,

What do you need from the IBM side?  I may be able to help.  We'd definitely
like to support you and Niels in publishing your results.

> time, I'll dive into finite field equations to demonstrate how we get there
> as well as I'll add a practical example to clarify the preference of this
> method in addition to the expected speedup of this method. My
> intention that other crypto libraries could take advantage of this document
> or maybe be a starting point for further improvements to the algorithm so
> I'm checking if IBM would publish or approve such a document the same as
> intel.
> 
> 
> > I have a sketch of ARM Neon code doing the equivalent of two vpmsumd,
> > with reasonable parallelism. Quite a lot of instructions needed.
> >
> 
> If you don't have much time, you can send it here and I'll continue from
> that point. I'm planning to compare the new method with the usual method
> with and without the karatsuba algorithm.
> 
> > +C Alignment of gcm_key table elements, which is declared in gcm.h
> > > +define(`TableElemAlign', `0x100')
> >
> > I still find this large constant puzzling. If I try
> >
> >   struct gcm_key key;
> >   printf("sizeof (key): %zd, sizeof(key.h[0]): %zd\n", sizeof(key),
> > sizeof(key.h[0]));
> >
> > (I added it to the start of test_main in gcm-test.c) and run on the
> > gcc112 machine, I get
> >
> >   sizeof (key): 4096, sizeof(key.h[0]): 16
> >
> > Which is what I'd expect, with elements of size 16 bytes, not 256 bytes.
> >
> > I haven't yet had the time to read the code carefully.
> >
> 
> You see, the alignment of each element is 0x100 (256). The table has 16
> elements and you got the size of the table 4096 which is reasonable because
> 16*256=4096
> 
> regards,
> Mamone
> ___
> nettle-bugs mailing list
> nettle-bugs@lists.lysator.liu.se
> http://lists.lysator.liu.se/mailman/listinfo/nettle-bugs

-- 
George Wilson
IBM Linux Technology Center
Security Development
___
nettle-bugs mailing list
nettle-bugs@lists.lysator.liu.se
http://lists.lysator.liu.se/mailman/listinfo/nettle-bugs

Re: [Haskell-community] Haskell.org committee proposals & GRC

[George Wilson]

I'm voting:

Proposal 0001: Yes
Proposal 0002: Yes

On Sat, 14 Nov 2020 at 03:10, Rebecca Skinner
 wrote:
>
> I'm voting:
>
> Proposal 0001: Yes
> Proposal 0002: Yes
>
> On Thu, Nov 12, 2020 at 4:00 PM Tikhon Jelvis  wrote:
>>
>> I'm voting:
>>
>> Proposal 0001: Yes
>> Proposal 0002: Yes
>>
>> On Thu, Oct 29, 2020 at 8:23 AM Jasper Van der Jeugt  
>> wrote:
>>>
>>> Hi all,
>>>
>>> The Haskell.org committee would like to adopt a new proposal format,
>>> inspired by the GHC Steering Committee.  We have created a repository
>>> with proposals here:
>>>
>>>https://github.com/haskell-org/committee
>>>
>>> I would like to call the members of the committee to vote on these
>>> proposals, but we also value community feedback.  The two proposals
>>> are:
>>>
>>>  -  0001: Establish the proposal process
>>> https://github.com/haskell-org/committee/pull/1
>>>  -  0002: Adopt the GRC code of conduct
>>> https://github.com/haskell-org/committee/pull/2
>>>
>>> For posterity and archival, I am including the full text of the
>>> proposals here.  For the record, I am in favor of accepting both
>>> proposals.
>>>
>>> # Proposal 0001: Haskell.org proposals
>>>
>>> In an effort to make the work of Haskell.org committee more
>>> transparent, we would like to adopt a proposal process similar to the
>>> [GHC Steering Committee].
>>>
>>> The proposed process is fairly light:
>>>
>>> 1.  New proposals are created as pull requests with a single file,
>>> following a `proposals/XYZW-some-title.extention` naming scheme.
>>> A template is available in `proposals/-template.md`.
>>>
>>> 2.  Proposals must have an author set who is responsible for driving
>>> the discussion.  If the author is not a member of the Haskell.org
>>> committee, we may additionally appoint a shepherd from the
>>> committee to help with this.  It is the responsibility of the
>>> author and the shepherd to notify any communities that may be
>>> interested in the proposal, so we can gather community feedback.
>>>
>>> 3.  The proposal must have a link to the PR discussion, so readers can
>>> easily find the full discussion once the PR is merged.
>>>
>>> 4.  We strive to make unanimous decisions, but we can use a majority
>>> vote (the committee has an odd number of members) to move forward.
>>>
>>> 5.  What happens next depends on whether or not the proposal is
>>> accepted:
>>>
>>>  -  If the proposal is accepted, `date-accepted` is set and the
>>> proposal is merged into the repository.  A summary with a link
>>> to the full PR discussion is sent out to
>>> `commun...@haskell.org`.
>>>
>>>  -  If the proposal is not accepted, the proposal is also merged
>>> for posterity, but a section is ammended to explain why it was
>>> rejected.
>>>
>>> # Proposal 0002: Guidelines for Respectful Communication
>>>
>>> The Haskell.org committee adopts the [Guidelines for Respectful
>>> Communication][grc].
>>>
>>> This applies only to members of the board, in in all our public
>>> interactions in the Haskell sphere, including email, social media,
>>> discussion forums, and so on.
>>>
>>> We may later adopt a stricter Code of Conduct, or set a Code of
>>> Conduct for platforms we manage (e.g. Discourse, mailing lists), but
>>> that is out of scope for this proposal.
>>>
>>> [grc]: https://github.com/ghc-proposals/ghc-proposals/blob/master/GRC.rst
>>> [GHC Steering Committee]: https://github.com/ghc-proposals/ghc-proposals
>>>
>>> Warm regards
>>> Jasper
>>
>> ___
>> Haskell-community mailing list
>> Haskell-community@haskell.org
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community
>
> ___
> Haskell-community mailing list
> Haskell-community@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community
___
Haskell-community mailing list
Haskell-community@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community

Re: [PowerPC] GCM optimization

[George Wilson]

vpmsumdF2,H3L,C1
> > +vpmsumdR2,H3M,C1
> > +vpmsumdF3,H2L,C2
> > +vpmsumdR3,H2M,C2
> > +vpmsumdF4,H1L,C3
> > +vpmsumdR4,H1M,C3
> > +vpmsumdF,H4L,C0
> > +vpmsumdR,H4M,C0
> > +
> > +C deferred recombination of partial products
> > +vxor   F3,F3,F4
> > +vxor   R3,R3,R4
> > +vxor   F,F,F2
> > +vxor   R,R,R2
> > +vxor   F,F,F3
> > +vxor   R,R,R3
> > +
> > +C reduction
> > +vpmsumdT,F,POLY_L
> > +xxswapdVSR(D),VSR(F)
> > +vxor   R,R,T
> > +vxor   D,R,D
> > +
> > +addi   DATA,DATA,0x40
> > +bdnz   L4x_loop
> > +
> > +C restore non-volatile vector registers
> > +addi   r8,SP,-64
> > +lvx20,0,r8
> > +addi   r8,r8,16
> > +lvx21,0,r8
> > +addi   r8,r8,16
> > +lvx22,0,r8
> > +addi   r8,r8,16
> > +lvx23,0,r8
> > +
> > +clrldi LENGTH,LENGTH,58  C 'set the high-order 58
> > bits to zeros'
> > +L2x:
> > +C --- process 2 blocks ---
> > +
> > +srdi   r7,LENGTH,5   C 'LENGTH / (2 * 16)'
> > +cmpldi r7,0
> > +beqL1x
> > +
> > +C load table elements
> > +li r8,1*TableElemAlign
> > +li r9,2*TableElemAlign
> > +li r10,3*TableElemAlign
> > +lxvd2x VSR(H1M),0,TABLE
> > +lxvd2x VSR(H1L),r8,TABLE
> > +lxvd2x VSR(H2M),r9,TABLE
> > +lxvd2x VSR(H2L),r10,TABLE
> > +
> > +C input loading
> > +li r10,0x10
> > +lxvd2x VSR(C0),0,DATAC load C0
> > +lxvd2x VSR(C1),r10,DATA  C load C1
> > +
> > +IF_LE(`
> > +vperm  C0,C0,C0,LE_MASK
> > +vperm  C1,C1,C1,LE_MASK
> > +')
> > +
> > +C previous digest combining
> > +vxor   C0,C0,D
> > +
> > +C polynomial multiplication
> > +vpmsumdF2,H1L,C1
> > +vpmsumdR2,H1M,C1
> > +vpmsumd    F,H2L,C0
> > +vpmsumdR,H2M,C0
> > +
> > +C deferred recombination of partial products
> > +vxor   F,F,F2
> > +vxor   R,R,R2
> > +
> > +C reduction
> > +vpmsumdT,F,POLY_L
> > +xxswapdVSR(D),VSR(F)
> > +vxor   R,R,T
> > +vxor   D,R,D
> > +
> > +addi   DATA,DATA,0x20
> > +clrldi LENGTH,LENGTH,59  C 'set the high-order 59
> > bits to zeros'
> > +L1x:
> > +C --- process 1 block ---
> > +
> > +srdi   r7,LENGTH,4   C 'LENGTH / (1 * 16)'
> > +cmpldi r7,0
> > +beqLmod
> > +
> > +C load table elements
> > +li r8,1*TableElemAlign
> > +lxvd2x VSR(H1M),0,TABLE
> > +lxvd2x VSR(H1L),r8,TABLE
> > +
> > +C input loading
> > +lxvd2x VSR(C0),0,DATAC load C0
> > +
> > +IF_LE(`
> > +vperm  C0,C0,C0,LE_MASK
> > +')
> > +
> > +C previous digest combining
> > +vxor   C0,C0,D
> > +
> > +C polynomial multiplication
> > +vpmsumdF,H1L,C0
> > +vpmsumdR,H1M,C0
> > +
> > +C reduction
> > +vpmsumdT,F,POLY_L
> > +xxswapdVSR(D),VSR(F)
> > +vxor   R,R,T
> > +vxor   D,R,D
> > +
> > +addi   DATA,DATA,0x10
> > +clrldi LENGTH,LENGTH,60  C 'set the high-order 60
> > bits to zeros'
> > +Lmod:
> > +C --- process the modulo bytes, padding the low-order bytes with
> > zeros ---
> > +
> > +cmpldi LENGTH,0
> > +beqLdone
> > +
> > +C load table elements
> > +li r8,1*TableElemAlign
> > +lxvd2x VSR(H1M),0,TABLE
> > +lxvd2x VSR(H1L),r8,TABLE
> > +
> > +C push every modulo byte to the stack and load them with padding into
> > vector register
> > +vxor   ZERO,ZERO,ZERO
> > +addi   r8,SP,-16
> > +stvx   ZERO,0,r8
> > +Lstb_loop:
> > +subic. LENGTH,LENGTH,1
> > +lbzx   r7,LENGTH,DATA
> > +stbx   r7,LENGTH,r8
> > +bneLstb_loop
> > +lxvd2x VSR(C0),0,r8
> > +
> > +IF_LE(`
> > +vperm  C0,C0,C0,LE_MASK
> > +')
> > +
> > +C previous digest combining
> > +vxor   C0,C0,D
> > +
> > +C polynomial multiplication
> > +vpmsumdF,H1L,C0
> > +vpmsumdR,H1M,C0
> > +
> > +C reduction
> > +vpmsumdT,F,POLY_L
> > +xxswapdVSR(D),VSR(F)
> > +vxor   R,R,T
> > +vxor   D,R,D
> > +
> > +Ldone:
> > +C byte-reverse of each doubleword permuting on little-endian mode
> > +IF_LE(`
> > +vperm  D,D,D,LE_MASK
> > +')
> > +stxvd2xVSR(D),0,XC store digest 'D'
> > +
> > +blr
> > +EPILOGUE(_nettle_gcm_hash)
> > +
> > +.data
> > +C 0xC201
> > +.polynomial:
> > +.align 4
> > +IF_BE(`
> > +.byte 0xC2
> > +.rept 14
> > +.byte 0x00
> > +.endr
> > +.byte 0x01
> > +',`
> > +.byte 0x01
> > +.rept 14
> > +.byte 0x00
> > +.endr
> > +.byte 0xC2
> > +')
> >
> > --
> > 2.17.1
> >
> ___
> nettle-bugs mailing list
> nettle-bugs@lists.lysator.liu.se
> http://lists.lysator.liu.se/mailman/listinfo/nettle-bugs

-- 
George Wilson
IBM Linux Technology Center
Security Development
___
nettle-bugs mailing list
nettle-bugs@lists.lysator.liu.se
http://lists.lysator.liu.se/mailman/listinfo/nettle-bugs

Re: [developer] Proposal: Platform-specific ZFS properties

[George Wilson]

Certainly going with user-defined properties would solve the problem of
safety but it comes at the cost of the administrative model. The
administrators would now be required to know the correct
vendor/distribution combination for each platform in order to change a
given property. Any existing scripts would now break with this model too.
It also introduces namespaces to avoid property name collisions.
Distributions would have to define their namespaces which could then be
changed by vendors that consume and modify them. This isn't a new problem,
but would be a new problem for what were once native properties that are
now moved to user-defined properties. I don't have a sense for what the
impact would be if we go down this path since I'm not sure how many people
rely on such properties as "sharenfs" and "sharesmb". I suspect that
illumos uses them the most, followed by freebsd, and then linux, but that's
just a guess.

The goal of the proposal was not to put that burden on the admins but
rather try to preserve the native properties while still providing cross
platform safety. It also seems to me that if vendors and distributions are
already going down the path of creating incompatibilities in their
implementations, then they don't care about or haven't thought about cross
platform safety. And if we can provide automatic conversion of a native
property to a platform-specific property then we haven't made it worse for
them.

I guess the question we should first answer is whether or not we want to
keep these native properties as native properties. Should they stay or
should they go? Effectively moving them from native properties mean
deprecating them and allowing  distributions/vendor to move down the
user-defined path if they so wish.


On Mon, Feb 25, 2019 at 10:54 PM Garrett D'Amore  wrote:

> I suspect having the platform “resolve” which property to use is going to
> be problematic at the very best.  The semantics are platform specific, and
> to make matters worse, I suspect that even on some platforms, you will find
> distributions that have imparted their own meanings to these properties.
> For example, I know of several different distributions built on illumos,
> and they don’t necessarily handle the properties in a perfectly compatible
> way.  I suspect the situation may actually be worse on Linux, where some
> distributions are likely to ship with **wildly** different userland and
> system layers (e.g. systemd vs. legacy rc scripts), and may thus wind up
> having quite different supported properties.
>
>
>
> Probably, the best solution here is for distributions to use their own
> “user property” for this, and that could apply to operating systems as
> well.  For example, one can imagine “openindiana:sharenfs” as a property.
> The upshot of that is that these properties might not convey automatically
> when moving file systems between distributions, but as the semantics (and
> possibly even syntax) of the contents are assumed not be 100% compatible,
> probably this is for the very best.  It is to be hoped that the NAS vendors
> building on top of ZFS are already making decent use of user properties
> (and namespace prefixes) to minimize problems.  Certainly that is the case
> for RackTop.
>
>
>
> Automatic conversion (or simply using a single shared value) only makes
> sense when there is broad agreement about the property contents.  Absent
> that, namespace separation, **without** naïve attempts to convert, is
> probably the next best option.
>
>
>
> Garrett
>
>
>
> *From: *George Wilson 
> *Sent: *Monday, February 25, 2019 9:43 PM
> *To: *openzfs-developer 
> *Subject: *[developer] Proposal: Platform-specific ZFS properties
>
>
>
> A couple of months ago we discussed the issues surrounding zfs properties
> which have platform specific implementations. The example given was
> “sharenfs” which has platform-specific syntax and platform-specific
> implementations. Having platform-specific syntax for these types of
> properties means that importing pools across platforms can lead to
> unexpected behavior. In order to ensure cross platform safety, I would like
> to propose the concept of platform-specific properties.
>
>
>
> Here are some high level goals:
>
>
>
> 1. Platform-specific properties are only processed on their respective
> platform. If a platform-specific property does not exist for the platform
> where the pool is imported then it’s treated as an unset property and the
> default setting is used.
>
> 2. Converting a property to a platform-specific property should not break
> existing scripts.
>
> Administrators may rely on property names, like sharenfs, so
> platform-specific properties should continue to use the same naming
> convention
>
> 3. Platforms should

[developer] Proposal: Platform-specific ZFS properties

[George Wilson]

A couple of months ago we discussed the issues surrounding zfs properties
which have platform specific implementations. The example given was
“sharenfs” which has platform-specific syntax and platform-specific
implementations. Having platform-specific syntax for these types of
properties means that importing pools across platforms can lead to
unexpected behavior. In order to ensure cross platform safety, I would like
to propose the concept of platform-specific properties.


Here are some high level goals:


1. Platform-specific properties are only processed on their respective
platform. If a platform-specific property does not exist for the platform
where the pool is imported then it’s treated as an unset property and the
default setting is used.

2. Converting a property to a platform-specific property should not break
existing scripts.

Administrators may rely on property names, like sharenfs, so
platform-specific properties should continue to use the same naming
convention

3. Platforms should be able to read and set platform-specific properties
for other platforms


Proposed Implementation:


To accomplish this, I propose that we introduce a property alias to our
current property scheme. The “alias” would only exist if a property is a
platform-specific property. Platform-specific properties would append the
operating system name to the property. For example, “sharenfs_Linux”,
“sharenfs_SunOS”, or “sharenfs_FreeBSD”.


In the case of the “sharenfs” property, the alias would remain as
“sharenfs” and the real property name would be “sharenfs_Linux”, for
example. When a user wants to “get” the value of “sharenfs” the system will
automatically map the property name to the platform-specific property.
Administrators can use the alias or the real property name when setting or
getting the property:


# zfs set sharenfs=on tank (preferred)
# zfs set sharenfs_Linux=on tank


Both result in the “sharenfs_Linux” property being set to “on” on-disk.
Reading a property does the reverse mapping, allowing users to get
“sharenfs” without having to know about the underlying platform-specific
name:


# zfs get sharenfs tank

NAME  PROPERTY  VALUE SOURCE

tank  sharenfs  on   default


Having this mapping allows pools to safely import across platforms since
only properties that exist for that platform will be read and evaluated.


Administrators may also want the ability to set another platform’s
property, so in those cases setting the real property name should be
allowed:


ZFS running on a Linux system:

# zfs set sharenfs_SunOS=off tank
# zfs get sharenfs tank

NAME  PROPERTY  VALUE SOURCE

tank  sharenfs  on   default


#zfs get sharenfs_SunOS tank

NAME  PROPERTY  VALUE SOURCE

tank  sharenfs_SunOS  on   default


How do people feel about this approach? If we are to adopt this model, then
how should we handle converting existing properties to platform-specific
properties? Feedback and suggestions are welcome.


Thanks,

George

--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/T6b50d98b3cf62715-Mfaa3006a379f2c07eed0d10a
Delivery options: https://openzfs.topicbox.com/groups/developer/subscription

[Haskell] ANN: New Haskell.org Committee Member

[George Wilson]

Following the nomination period and discussion, the Haskell.org
committee has selected the following member for a new three-year term,
expiring 2021:

  * Emily Pillmore

As per the rules of the committee, this discussion was held among the
current members of the committee, and the outgoing member of the
committee who was not seeking reappointment.

Thank you to all the many candidates who submitted a self-nomination.
We received a number of strong nominations. We would encourage all
those who nominated themselves to consider self-nominating again in
the future.

The outgoing member is Gershom Bazerman. He served not one but two
terms for the Haskell.org committee, and he was chair during his last
term. During this long period, Gershom has undeniably made the
community a better place with his leadership and efforts. Thank you
for your service!

Since Gershom was our chair, the committee, including the new member,
will hold a discussion and elect a new chair from amongst ourselves.

Regards,
George Wilson
___
Haskell mailing list
Haskell@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell

[Haskell-community] ANN: New Haskell.org Committee Member

[George Wilson]

Following the nomination period and discussion, the Haskell.org
committee has selected the following member for a new three-year term,
expiring 2021:

  * Emily Pillmore

As per the rules of the committee, this discussion was held among the
current members of the committee, and the outgoing member of the
committee who was not seeking reappointment.

Thank you to all the many candidates who submitted a self-nomination.
We received a number of strong nominations. We would encourage all
those who nominated themselves to consider self-nominating again in
the future.

The outgoing member is Gershom Bazerman. He served not one but two
terms for the Haskell.org committee, and he was chair during his last
term. During this long period, Gershom has undeniably made the
community a better place with his leadership and efforts. Thank you
for your service!

Since Gershom was our chair, the committee, including the new member,
will hold a discussion and elect a new chair from amongst ourselves.

Regards,
George Wilson
___
Haskell-community mailing list
Haskell-community@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community

[Haskell] Call for Haskell.org Committee Nominations

[George Wilson]

Dear Haskellers,

It is time to put out a call for new nominations (typically but not
necessarily self-nominations) to the haskell.org committee. We have
one member of our committee due for retirement -- our chair
Gershom Bazerman.

To nominate yourself, please send an email to committee at haskell.org
by the 7th of January, 2019. Retiring members are eligible to re-nominate
themselves. Please feel free to include any information about yourself
that you think will help us to make a decision.

The Haskell.org committee serves as a board of directors for
Haskell.org, a 501(c)3 nonprofit which oversees technical and
financial resources related to Haskell community infrastructure.

Being a member of the committee does not necessarily require a
significant amount of time, but committee members should aim to be
responsive during discussions when the committee is called upon to
make a decision. Strong leadership, communication, and judgement are
very important characteristics for committee members. The role is
about setting policy, providing direction/guidance for Haskell.org
infrastructure, planning for the long term, and being fiscally
responsible with the Haskell.org funds (and donations). As overseers
for policy regarding the open source side of Haskell, committee
members must also be able to set aside personal or business related
bias and make decisions with the good of the open source Haskell
community in mind.

We seek a broad representation from different segments of the Haskell
world -- including but not limited to those focused on education,
those focused on industrial applications, those with background in
organizing users-groups, and those focused directly on our technical
infrastructure.

More details about the committee's roles and responsibilities are on

https://wiki.haskell.org/Haskell.org_committee

If you have any questions about the process, please feel free to
e-mail us at committee at haskell.org, or contact one of us
individually.

Regards,
George Wilson
___
Haskell mailing list
Haskell@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell

[Touch-packages] [Bug 1798212] Re: systemd stuck in uninterruptible sleep state

[George Wilson]

We just hit a similar issue on Ubuntu 18.04. From the crash dump, I'm
able to see this stack:

[12929.914040] kworker/u4:5D0 27285  2 0x8080
[12929.914053] Workqueue: events_unbound fsnotify_mark_destroy_workfn
[12929.914055] Call Trace:
[12929.914075]  __schedule+0x291/0x8a0
[12929.914078]  schedule+0x2c/0x80
[12929.914081]  schedule_timeout+0x1cf/0x350
[12929.914086]  ? select_idle_sibling+0x29/0x410
[12929.914089]  ? __enqueue_entity+0x5c/0x60
[12929.914091]  ? enqueue_entity+0x10e/0x6b0
[12929.914093]  wait_for_completion+0xba/0x140
[12929.914094]  ? wake_up_q+0x80/0x80
[12929.914099]  __synchronize_srcu.part.13+0x85/0xb0
[12929.914101]  ? trace_raw_output_rcu_utilization+0x50/0x50
[12929.914103]  ? ttwu_do_activate+0x77/0x80
[12929.914105]  synchronize_srcu+0x66/0xe0
[12929.914107]  ? synchronize_srcu+0x66/0xe0
[12929.914109]  fsnotify_mark_destroy_workfn+0x7b/0xe0
[12929.914113]  process_one_work+0x1de/0x410
[12929.914115]  worker_thread+0x228/0x410
[12929.914118]  kthread+0x121/0x140
[12929.914119]  ? process_one_work+0x410/0x410
[12929.914121]  ? kthread_create_worker_on_cpu+0x70/0x70

Several other systemd processes are also hung. I was able to issue an
NMI to generate a crash dump so I can pull out more information if that
would be helpful.

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1798212

Title:
  systemd stuck in uninterruptible sleep state

Status in systemd package in Ubuntu:
  Confirmed

Bug description:
  Ubuntu 16.04 systemd 229 may get stuck in an uninterruptible sleep
  state. Any subsequent systemctl command times out with "Failed to
  execute operation: Connection timed out" error message. gdb and strace
  is unable to attach to systemd process 1.

  $ systemd --version
  systemd 229
  +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP 
+GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN

  $ cat /proc/1/stack 
  [<0>] flush_work+0x129/0x1e0
  [<0>] flush_delayed_work+0x3f/0x50
  [<0>] fsnotify_wait_marks_destroyed+0x15/0x20
  [<0>] fsnotify_destroy_group+0x48/0xd0
  [<0>] inotify_release+0x1e/0x50
  [<0>] __fput+0xea/0x220
  [<0>] fput+0xe/0x10
  [<0>] task_work_run+0x8a/0xb0
  [<0>] exit_to_usermode_loop+0xc4/0xd0
  [<0>] do_syscall_64+0xf4/0x130
  [<0>] entry_SYSCALL_64_after_hwframe+0x3d/0xa2
  [<0>] 0x

  $ cat /proc/1/status 
  Name: systemd
  Umask:
  State:D (disk sleep)
  Tgid: 1
  Ngid: 0
  Pid:  1
  PPid: 0
  TracerPid:9724
  Uid:  0   0   0   0
  Gid:  0   0   0   0
  FDSize:   128
  Groups:
  NStgid:   1
  NSpid:1
  NSpgid:   1
  NSsid:1
  VmPeak: 251136 kB
  VmSize: 185728 kB
  VmLck:   0 kB
  VmPin:   0 kB
  VmHWM:6620 kB
  VmRSS:6264 kB
  RssAnon:  2380 kB
  RssFile:  3884 kB
  RssShmem:0 kB
  VmData:  18752 kB
  VmStk: 132 kB
  VmExe:1392 kB
  VmLib:3692 kB
  VmPTE: 128 kB
  VmSwap:  0 kB
  HugetlbPages:0 kB
  CoreDumping:  0
  Threads:  1
  SigQ: 1/257098
  SigPnd:   0004
  ShdPnd:   0001
  SigBlk:   7be3c0fe28014a03
  SigIgn:   1000
  SigCgt:   000184ec
  CapInh:   
  CapPrm:   003f
  CapEff:   003f
  CapBnd:   003f
  CapAmb:   
  NoNewPrivs:   0
  Seccomp:  0
  Speculation_Store_Bypass: vulnerable
  Cpus_allowed: 
  Cpus_allowed_list:0-31
  Mems_allowed: 
,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,0003
  Mems_allowed_list:0-1
  voluntary_ctxt_switches:  716968
  nonvoluntary_ctxt_switches:   9565

  $ dmesg
  [...]
  [1255076.993707] INFO: task systemd:1 blocked for more than 120 seconds.
  [1255077.000295]   Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
  [1255077.006804] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
  [1255077.014996] systemd D0 1  0 0x
  [1255077.015000] Call Trace:
  [1255077.015011]  __schedule+0x3d6/0x8b0
  [1255077.015019]  ? enqueue_entity+0x112/0x670
  [1255077.015021]  schedule+0x36/0x80
  [1255077.015025]  schedule_timeout+0x1db/0x370
  [1255077.015028]  ? check_preempt_curr+0x54/0x90
  [1255077.015030]  ? ttwu_do_wakeup+0x1e/0x150
  [1255077.015033]  wait_for_completion+0xb4/0x140
  [1255077.015035]  ? wake_up_q+0x70/0x70
  [1255077.015040]  flush_work+0x129/0x1e0
  [1255077.015043]  ? 

[Bug 1798212] Re: systemd stuck in uninterruptible sleep state

[George Wilson]

We just hit a similar issue on Ubuntu 18.04. From the crash dump, I'm
able to see this stack:

[12929.914040] kworker/u4:5D0 27285  2 0x8080
[12929.914053] Workqueue: events_unbound fsnotify_mark_destroy_workfn
[12929.914055] Call Trace:
[12929.914075]  __schedule+0x291/0x8a0
[12929.914078]  schedule+0x2c/0x80
[12929.914081]  schedule_timeout+0x1cf/0x350
[12929.914086]  ? select_idle_sibling+0x29/0x410
[12929.914089]  ? __enqueue_entity+0x5c/0x60
[12929.914091]  ? enqueue_entity+0x10e/0x6b0
[12929.914093]  wait_for_completion+0xba/0x140
[12929.914094]  ? wake_up_q+0x80/0x80
[12929.914099]  __synchronize_srcu.part.13+0x85/0xb0
[12929.914101]  ? trace_raw_output_rcu_utilization+0x50/0x50
[12929.914103]  ? ttwu_do_activate+0x77/0x80
[12929.914105]  synchronize_srcu+0x66/0xe0
[12929.914107]  ? synchronize_srcu+0x66/0xe0
[12929.914109]  fsnotify_mark_destroy_workfn+0x7b/0xe0
[12929.914113]  process_one_work+0x1de/0x410
[12929.914115]  worker_thread+0x228/0x410
[12929.914118]  kthread+0x121/0x140
[12929.914119]  ? process_one_work+0x410/0x410
[12929.914121]  ? kthread_create_worker_on_cpu+0x70/0x70

Several other systemd processes are also hung. I was able to issue an
NMI to generate a crash dump so I can pull out more information if that
would be helpful.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1798212

Title:
  systemd stuck in uninterruptible sleep state

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1798212/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[developer] Re: [openzfs/openzfs] 9751 Allocation throttling misplacing ditto blocks (#688)

[George Wilson]

grwilson commented on this pull request.



> @@ -1039,6 +1039,13 @@ metaslab_group_allocatable(metaslab_group_t *mg, 
> metaslab_group_t *rotor,
if (mg->mg_no_free_space)
return (B_FALSE);
 
+   /*
+* Relax allocation throttling for ditto blocks.  Due to
+* random imbalances in allocation it tends to push copies
+* to one vdev, that looks a bit better at the moment.
+*/
+   qmax = qmax * (4 + d) / 4;

Out of curiosity, did you find that it made a difference to increase the qmax 
value between the 2nd and 3rd DVAs? Or is equally efficient to say any DVA > 1 
gets to go over the qmax value by 25%?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/688#discussion_r221943428
--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/T5b2a8092d211827d-M6fd39e132f4291f80b4382a6
Delivery options: https://openzfs.topicbox.com/groups/developer/subscription

[developer] Re: [openzfs/openzfs] 9738 Fix third block copy allocations, broken at 9112. (#687)

[George Wilson]

grwilson approved this pull request.





-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/687#pullrequestreview-160374250
--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/T8d1d942c9a2fd9eb-M61cec72ac90abd6986b3e741
Delivery options: https://openzfs.topicbox.com/groups/developer/subscription

[developer] Re: [openzfs/openzfs] 9102 zfs should be able to initialize storage devices (#586)

[George Wilson]

@citrus-it in the past we've seen storage arrays that have zeroed out blocks 
behind the scenes leading to ZFS reported checksums. We wanted to have a 
default value that would not be confused with that type of corruption. We did 
make this a global parameter (`zfs_initialize_value`) so that it could be 
changed via mdb or /etc/system (or other mechanisms). Maybe a future 
enhancement would be to make this a true parameter to the pool. 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/586#issuecomment-384632473
--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/discussions/T6777d2c2033b2134-Mf79eda027a0ee1e1dcf88407
Delivery options: https://openzfs.topicbox.com/groups

Re: [Haskell-community] New Videos for Haskell.org Homepage

[George Wilson]

Thumbs up. It looks like a good selection to me.

On 9 April 2018 at 05:52, Gershom B  wrote:
> I wanted to call people's attention to this PR:
>
> https://github.com/haskell-infra/hl/pull/229
>
> The videos are overdue for updating, and the suggestions look good to
> me, but a bit more feedback (even if just a flood of thumbs ups :-))
> wouldn't hurt before pulling the trigger.
>
> -g
> ___
> Haskell-community mailing list
> Haskell-community@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community
___
Haskell-community mailing list
Haskell-community@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community

Re: [Haskell-community] Requesting guidance for GSoC Project

[George Wilson]

Hi Khilan,

It sounds like you have everything you need to write up a proposal and
apply at https://summerofcode.withgoogle.com
Google has a guide for writing GSOC proposals here:
https://google.github.io/gsocguides/student/writing-a-proposal

Cheers,
George

On 20 March 2018 at 04:15, Khilan Ravani <khilanrav...@gmail.com> wrote:
> Sir/Mam,
>
> I am a sophomore student pursuing Computer Science and Engineering (C.S.E)
> from Indian Institute of Information Technology, Vadodara. I am well adapt
> with concepts of Deep Learning, Image Processing & AI and am currently
> working on a few research topics in the same domain. I have adequate
> experience in machine learning and Signal / Image processing (completed a
> few deep learning based projects ) and am currently pursuing some research
> problems in Satellite Imaging (Hyper-spectral data analysis) and processing
> SAR images.
>
> Firstly, I would like to sincerely apologize to you for contacting you via
> the mailing list this late. Please forgive me for misinterpreting the
> process for new project ideas for participating in GSoC 2018 for Haskell
> organization. Actually, I tried to communicate my interest for 'Summer of
> Haskell' program (I was unaware at that time that it was only for years when
> Haskell didn't participate in GSoC) to the general committee
> (commit...@haskell.org) from where George Wilson sir directed me towards Alp
> and Jared sir and then I got in touch with Alexey sir.
>
> The project that I aim to accomplish during the GSoC period is quite
> straightforward. I plan to enhance the existing code base of Haskell Image
> Processing (HIP) package with some new contemporary algorithms (with a few
> state of the art ones). The current version of HIP package has a limited
> scope and I plan to manifold the coverage of this library by adding various
> algorithms from different classes of image processing in it. I have
> discussed this project with Alp, Jared and Alexey sir of whom Alexey sir is
> the author of the HIP package. Some algorithms which I plan to implement
> include :
>
> 1) Adaptive Histogram Equalization (since Histogram is already implemented)
> 2) Canny Edge Detection, Contour Extraction
> 3) Hough transform
> 4) Gabor filter
> 5) Deconvolution
> 6) Otsu Thresholding method
> 7) Dithering (Floyd - Steinberg)
> 8) Un - implemented filters (HOG, LOG etc.)
> 9) Fingerprint (i.e Biometric) Detection.
>
> I feel that this project is doable within the summer period and that it can
> be a great addition to the Haskell organization since firstly, It would
> expand the scope of the library and increase the user base of Haskell and
> secondly, no other functional programming languages (including Clojure, Elm
> etc.) have a decent image processing library and hence it can be a very good
> chance for Haskell to be an obvious choice of developers looking forward to
> implement Computer vision problems with some functional programming
> language. I have had an discussion with  Alexey and Jared sir and they seem
> to agree that this project can be a really good addition for our Haskell
> organization.
>
> Also, for my technical skills : I do possess the required programming skills
> (C++, OpenCV, Python, Haskell (Basics), Tensorflow, Keras (CUDA framework),
> Watson API, Matlab) as well as the exposure to open source platforms such as
> github. Also, I can provide you with my CV if and when you say.
>
> I would be really delighted to know how you feel about this project. Once
> again, I am very sorry for contacting this late via the mailing lists and
> would like to sincerely request you to go through the project that I am
> suggesting and kindly provide your valuable opinions on the same.
>
> I will happily provide you with further clarifications on the project if you
> need any.
> Looking forward to hear from you.
>
> Thank you,
> Yours sincerely,
> Khilan Ravani.
>
>
>
> ___
> Haskell-community mailing list
> Haskell-community@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community
>
___
Haskell-community mailing list
Haskell-community@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community

[developer] Re: [openzfs/openzfs] 9102 zfs should be able to initialize storage devices (#586)

[George Wilson]

You could go through and `dd` to every disk before adding it to the pool and 
that is the common practice used by many. The problem that this is trying to 
solve is to be able to avoid the delay of having to wait for every disk to be 
pre-warmed before using it. Also, people often forget to "warm" a disk before 
adding it to a pool and then find themselves stuck with the performance 
penalty. This command allows you to "warm" the disk at anytime. It also means 
that you could "warm" the disk while you immediately start using it. During 
that time there would be an I/O penalty since you're obviously doing more I/O 
than just writes but it allows those operations to run concurrently. 
Furthermore, you could extend this command to implement some form of secure 
delete where you want to write a pattern over any free space. Or even manually 
TRIM the data by issuing TRIM commands on any free region of the disk. This 
functionality has a very specific use case for us at Delphix but we felt that 
others might be interested in something similar (especially since people today 
are already using `dd` for this purpose).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/586#issuecomment-372033793
--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/discussions/T6777d2c2033b2134-Ma0706c51ef1ecbff5498e4c9
Delivery options: https://openzfs.topicbox.com/groups

[developer] [openzfs/openzfs] 9102 zfs should be able to initialize storage devices (#586)

[George Wilson]

Reviewed by: John Wren Kennedy 
Reviewed by: Matthew Ahrens 
Reviewed by: Pavel Zakharov 
Reviewed by: Prakash Surya 

PROBLEM
The first access to a block incurs a performance penalty on some platforms
(e.g. AWS's EBS, VMware VMDKs). Therefore we recommend that volumes are "thick
provisioned", where supported by the platform (VMware). This can create a large
delay in getting a new virtual machines up and running (or adding storage to
an existing Engine). If the thick provision step is omitted, write  performance
will be suboptimal until all blocks on the LUN have been written.

SOLUTION
This feature introduces a way to 'initialize' the disks at install or in the
background to make sure we don't incur this first read penalty.

When an entire LUN is added to ZFS, we make all space available immediately,
and allow ZFS to find unallocated space and zero it out. This works with
concurrent writes to arbitrary offsets, ensuring that we don't zero out
something that has been (or is in the middle of being) written. This scheme
can also be applied to existing pools (affecting only free regions on the
vdev). Detailed design:
- new subcommand:zpool initialize [-cs]  [ ...]
- start, suspend, or cancel initialization
- Creates new open-context thread for each vdev
- Thread iterates through all metaslabs in this vdev
- Each metaslab:
- select a metaslab
- load the metaslab
- mark the metaslab as being zeroed
- walk all free ranges within that metaslab and translate them
  to ranges on the leaf vdev
- issue a "zeroing" I/O on the leaf vdev that corresponds to a
  free range on the metaslab we're working on
- continue until all free ranges for this metaslab have been
  "zeroed"
- reset/unmark the metaslab being zeroed
- if more metaslabs exist, then repeat above tasks.
- if no more metaslabs, then we're done.

- progress for the initialization is stored on-disk in the vdev’s leaf
  zap object. The following information is stored:
- the last offset that has been initialized
- the state of the initialization process (i.e. active,
  suspended, or canceled)
- the start time for the initialization

- progress is reported via the zpool status command and shows
  information for each of the videos that are initializing
You can view, comment on, or merge this pull request online at:

  https://github.com/openzfs/openzfs/pull/586

-- Commit Summary --

  * 9102 zfs should be able to initialize storage devices

-- File Changes --

M usr/src/cmd/truss/codes.c (4)
M usr/src/cmd/zpool/zpool_main.c (156)
M usr/src/cmd/ztest/ztest.c (98)
M usr/src/lib/libzfs/common/libzfs.h (5)
M usr/src/lib/libzfs/common/libzfs_pool.c (94)
M usr/src/lib/libzfs/common/libzfs_util.c (9)
M usr/src/lib/libzfs/common/mapfile-vers (1)
M usr/src/lib/libzfs_core/common/libzfs_core.c (37)
M usr/src/lib/libzfs_core/common/libzfs_core.h (4)
M usr/src/lib/libzfs_core/common/mapfile-vers (6)
M usr/src/lib/libzpool/common/llib-lzpool (3)
M usr/src/man/man1m/zpool.1m (31)
M usr/src/pkg/manifests/system-test-zfstest.mf (39)
M usr/src/test/zfs-tests/include/commands.cfg (1)
M usr/src/test/zfs-tests/runfiles/delphix.run (13)
A 
usr/src/test/zfs-tests/tests/functional/cli_root/zpool_initialize/Makefile (21)
A 
usr/src/test/zfs-tests/tests/functional/cli_root/zpool_initialize/cleanup.ksh 
(31)
A 
usr/src/test/zfs-tests/tests/functional/cli_root/zpool_initialize/zpool_initialize.kshlib
 (43)
A 
usr/src/test/zfs-tests/tests/functional/cli_root/zpool_initialize/zpool_initialize_attach_detach_add_remove.ksh
 (68)
A 
usr/src/test/zfs-tests/tests/functional/cli_root/zpool_initialize/zpool_initialize_import_export.ksh
 (78)
A 
usr/src/test/zfs-tests/tests/functional/cli_root/zpool_initialize/zpool_initialize_offline_export_import_online.ksh
 (66)
A 
usr/src/test/zfs-tests/tests/functional/cli_root/zpool_initialize/zpool_initialize_online_offline.ksh
 (74)
A 
usr/src/test/zfs-tests/tests/functional/cli_root/zpool_initialize/zpool_initialize_split.ksh
 (64)
A 
usr/src/test/zfs-tests/tests/functional/cli_root/zpool_initialize/zpool_initialize_start_and_cancel_neg.ksh
 (60)
A 
usr/src/test/zfs-tests/tests/functional/cli_root/zpool_initialize/zpool_initialize_start_and_cancel_pos.ksh
 (52)
A 
usr/src/test/zfs-tests/tests/functional/cli_root/zpool_initialize/zpool_initialize_suspend_resume.ksh
 (63)
A 
usr/src/test/zfs-tests/tests/functional/cli_root/zpool_initialize/zpool_initialize_unsupported_vdevs.ksh
 (74)
A 

[developer] Re: [openzfs/openzfs] arc_loan_compressed_buf() can increment arc_loaned_bytes by the wrong value (#558)

[George Wilson]

grwilson approved this pull request.





-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/558#pullrequestreview-99102002
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T396cc9871881f0d5-M279a791a0241b69ede4f9bfe
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] arc_loan_compressed_buf() can increment arc_loaned_bytes by the wrong value (#558)

[George Wilson]

grwilson requested changes on this pull request.

This change looks good, can you also change `arc_loan_buf` to mimic this 
change. I think we should always rely on `arc_buf_size` to determine the 
correct value to pass to `arc_loaned_bytes_update`



-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/558#pullrequestreview-99065653
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T396cc9871881f0d5-Meb38a8a3ec5b5179e7015416
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] 8857 zio_remove_child() panic due to already destroyed parent zio (#505)

[George Wilson]

@grwilson pushed 1 commit.

1d7e2b0  all bits


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/505/files/445e55128ca0a1cf9b0a7013d996edd50da3a0f4..1d7e2b0d0cb2416d9838f0bd25fc174f31c4be09

--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/Td0adf1bfacf39ae0-M94874194d6bc4b41eff311de
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] 8857 zio_remove_child() panic due to already destroyed parent zio (#505)

[George Wilson]

@grwilson pushed 1 commit.

445e551  predefine child bits


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/505/files/b1342b226af1aa8a39cbcbe3b59a9270765c8ef2..445e55128ca0a1cf9b0a7013d996edd50da3a0f4

--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/Td0adf1bfacf39ae0-M70d1f4a1e1285c1d085c2cc6
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] 7584 Improve 'zpool labelclear' command (#424)

[George Wilson]

One of the guiding principles for zfs is simple administration and it seems 
like we're exposing way too many knobs to the administrator. These knobs expose 
the internals of the product. For example, if I want to clear the labels, why 
wouldn't I just run `zpool labelclear` and have the command figure out if we 
need to clear label 2 and 3 or 0-3 or any combination? In other words if the 
user wants to clear the labels then clear whatever valid labels we find. If the 
user specifies -f then clear them all. I like that this change is trying to 
protect the user but it seems like we can accomplish this without having to 
make the user figure out the internal details of the product.

This is less of a concern but one that I want to make sure we can discuss -- I 
still struggle with invalidating the nvlist encoding vs setting txg = 0. Yes, 
it's 1 byte vs 8 bytes so we have a smaller chance of impacting any software 
that is using that disk but the chance is not 0% in either case. I would argue 
that setting the value of txg=0 at least provides some diagnostics with 
existing tools and possibly some recovery opportunities that are not available 
with the nvlist invalidate case. For example,  if invalidating the nvlist 
impacts the software running on that disk, how would you ever know that the 
disk was once used by zfs and has been invalidated? You would end up with 
corruption with no way of diagnosing what caused it. We could enhance the tools 
to look for the invalid encoding making this less of an issue. I recognize this 
is not a new problem since the current implementation "wipes" the labels. This 
is why I wonder if we ever need to "wipe" the labels or do we just want zfs to 
forget about this device? Do we know of cases where we have needed to "wipe" 
the label?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/424#issuecomment-364462555
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T59e810f5492df392-Mdb7922ecbe8217cb2bbd1d04
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] 8857 zio_remove_child() panic due to already destroyed parent zio (#505)

[George Wilson]

@youzhongyang @avg-I can you take another look to make sure I've address all of 
your concerns?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/505#issuecomment-364239237
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/Td0adf1bfacf39ae0-M53d2269b8fffa3144053b079
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] 8857 zio_remove_child() panic due to already destroyed parent zio (#505)

[George Wilson]

Your comments are considered and I will be pushing a new commit with the 
updated logic. I've tried several variations to the suggestions to find which 
looks the best and has the least amount of impact. The suggested 
`ZIO_CHILD_BIT` option looks the best so I've gone with that option.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/505#issuecomment-364237275
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/Td0adf1bfacf39ae0-Me6489fa6afb526dca5007381
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] 7584 Improve 'zpool labelclear' command (#424)

[George Wilson]

grwilson commented on this pull request.

I have concerns about the ultimate goal for this command. It's unclear if we 
really want to "wipe" or just "forget" about this device.

> @@ -709,8 +751,12 @@ zpool_do_labelclear(int argc, char **argv)
}
 
if (zpool_read_label(fd, ) != 0) {
-   (void) fprintf(stderr,
-   gettext("failed to read label from %s\n"), vdev);
+   if (force)

Shouldn't this check force_invalid since it's possible for zpool_read_label to 
return -1 for reasons other than the inability to read the label. If just force 
is used then we could wipe out the label of a legit pool.

> + if (pread64(fd, , sizeof (vdev_label_t),
+   label_offset(size, l)) != sizeof (vdev_label_t))
+   return (-1);
+
+   if (!force) {
+   nvlist_t *config = NULL;
+   if (nvlist_unpack(buf, buflen, , 0) != 0)
+   return (-1);
+   nvlist_free(config);
+   }
+   }
+
+   if (wipe) {
+   (void) memset(, 0, sizeof (vdev_label_t));
+   } else {
+   if (nvlist_invalidate(buf, buflen) != 0)

If the goal is to prevent zpool import from showing the pool, then we should 
set the TXG value to 0 which is how ZFS clears a label while still leaving some 
information around for debugging.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/424#pullrequestreview-94746416
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T59e810f5492df392-Mb71b9d84a4c53f6d4b400daf
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] 8857 zio_remove_child() panic due to already destroyed parent zio (#505)

[George Wilson]

grwilson commented on this pull request.



> @@ -209,12 +209,13 @@ enum zio_flag {
ZIO_FLAG_CANFAIL)
 
 enum zio_child {
-   ZIO_CHILD_VDEV = 0,
-   ZIO_CHILD_GANG,
-   ZIO_CHILD_DDT,
-   ZIO_CHILD_LOGICAL,
-   ZIO_CHILD_TYPES
+   ZIO_CHILD_VDEV  = 1 << 0,
+   ZIO_CHILD_GANG  = 1 << 1,
+   ZIO_CHILD_DDT   = 1 << 2,
+   ZIO_CHILD_LOGICAL   = 1 << 3

Yeah, I looked at this option but didn't like how it looked specifically for 
this reason. I've looked at using variable arguments too but it has drawback 
too. I'm going to consider this option again but using a variable to hold the 
bits we plan to pass into `zio_wait_for_children`.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/505#discussion_r158716835
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/Td0adf1bfacf39ae0-Ma54010b2908fc3d5b0c3ea0a
Powered by Topicbox: https://topicbox.com

[developer] [openzfs/openzfs] 8857 zio_remove_child() panic due to already destroyed parent zio (#505)

[George Wilson]

PROBLEM

It's possible for a parent zio to complete even though it has children
which have not completed. This can result in the following panic:

> $C
ff01809128c0 vpanic()
ff01809128e0 mutex_panic+0x58(fb94c904, ff597dde7f80)
ff0180912950 mutex_vector_enter+0x347(ff597dde7f80)
ff01809129b0 zio_remove_child+0x50(ff597dde7c58, ff32bd901ac0,
ff3373370908)
ff0180912a40 zio_done+0x390(ff32bd901ac0)
ff0180912a70 zio_execute+0x78(ff32bd901ac0)
ff0180912b30 taskq_thread+0x2d0(ff33bae44140)
ff0180912b40 thread_start+8()
> ::status
debugging crash dump vmcore.2 (64-bit) from batfs0390
operating system: 5.11 joyent_20170911T171900Z (i86pc)
image uuid: (not set)
panic message: mutex_enter: bad mutex, lp=ff597dde7f80
owner=ff3c59b39480 thread=ff0180912c40
dump content: kernel pages only

The problem is that dbuf_prefetch along with l2arc can create a zio tree
which confuses the parent zio and allows it to complete with while children
still exist. Here's the scenario:

zio tree:
pio
 |--- lio

The parent zio, pio, has entered the zio_done stage and begins to check its
children to see there are still some that have not completed. In zio_done(),
the children are checked in the following order:

zio_wait_for_children(zio, ZIO_CHILD_VDEV, ZIO_WAIT_DONE)
zio_wait_for_children(zio, ZIO_CHILD_GANG, ZIO_WAIT_DONE)
zio_wait_for_children(zio, ZIO_CHILD_DDT, ZIO_WAIT_DONE)
zio_wait_for_children(zio, ZIO_CHILD_LOGICAL, ZIO_WAIT_DONE)

If pio, finds any child which has not completed then it stops executing and
goes to sleep. Each call to zio_wait_for_children() will grab the io_lock
while checking the particular child.

In this scenario, the pio has completed the first call to
zio_wait_for_children() to check for any ZIO_CHILD_VDEV children. Since
the only zio in the zio tree right now is the logical zio, lio, then it
completes that call and prepares to check the next child type.

In the meantime, the lio completes and in its callback creates a child vdev
zio, cio. The zio tree looks like this:

zio tree:
pio
 |--- lio
 |--- cio

The lio then grabs the parent's io_lock and removes itself.

zio tree:
pio
 |--- cio

The pio continues to run but has already completed its check for ZIO_CHILD_VDEV
and will erroneously complete. When the child zio, cio, completes it will panic
the system trying to reference the parent zio which has been destroyed.

SOLUTION
The fix is to rework the zio_wait_for_children() logic to accept an enum
bitmask for all the children types that it's interested in checking. The
io_lock will be held the entire time we check all the children types. Since
the function now accepts an enum, a simple ZIO_CHILD() macro is provided
to allow the conversion between the enum and the array index used by the
io children logic.
You can view, comment on, or merge this pull request online at:

  https://github.com/openzfs/openzfs/pull/505

-- Commit Summary --

  * 8857 zio_remove_child() panic due to already destroyed parent zio

-- File Changes --

M usr/src/uts/common/fs/zfs/sys/zio.h (11)
M usr/src/uts/common/fs/zfs/zio.c (77)

-- Patch Links --

https://github.com/openzfs/openzfs/pull/505.patch
https://github.com/openzfs/openzfs/pull/505.diff

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/505

--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/Td0adf1bfacf39ae0-M415e285ea79a91f21ea0c4e1
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] fixed arc_cksum_is_equal() that doesn't take into account ABD-logic (#498)

[George Wilson]

grwilson commented on this pull request.



> @@ -1567,7 +1569,7 @@ arc_cksum_is_equal(arc_buf_hdr_t *hdr, zio_t *zio)
bzero((char *)cbuf + csize, HDR_GET_PSIZE(hdr) - csize);

BTW, this wasn't part of your change but this also looks broken. I believe that 
we should be using `abd_zero_off` here rather than bzero.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/498#pullrequestreview-78441520
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/Te84e226aa4103950-M65cbb43b19c24b7023572006
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] fixed arc_cksum_is_equal() that doesn't take into account ABD-logic (#498)

[George Wilson]

grwilson approved this pull request.

Changes look good. Make sure you open up an illumos bug for this this.



-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/498#pullrequestreview-78438208
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/Te84e226aa4103950-Mce57e684ea8700b41cd62bb2
Powered by Topicbox: https://topicbox.com

Re: [developer] read-only pool, remove vdev

[George Wilson]

Andriy,

I'm not familiar with spa_async_thread_vd() (I don't believe that is
upstreamed yet) so I can't comment on what is the correct thing to do
there. As for spa_async_thread(), I think we can ignore all tasks when the
pool is read-only. You'll probably need to update the call to vdev_probe()
in zio_vdev_io_done() so that we only call it if spa_writeable().

Thanks,
George

On Tue, Oct 17, 2017 at 1:40 PM Andriy Gapon  wrote:

> 
> Both spa_async_thread() and spa_async_thread_vd() assert that spa_sync_on
> is
> true.  I think that the logic to ensure that is sound except for one case.
> Specifically, if a pool is imported read-only, then spa_sync_on is never
> set to
> true (which is correct), but spa_async_suspended is not incremented either.
> So, spa_async_request() ==> spa_async_dispatch_vd() does not bail out and
> spa_async_thread_vd gets executed.
> 
> I am actually not sure what's the best thing to do here for a read-only
> pool.
> On the one hand, we should not touch its on-disk configuration, on the
> other
> hand it seems that we should update the in-memory state of vdevs.
> 
> Maybe the assertion simply can be dropped?
> I am not sure if spa_async_thread_vd() really depends on spa_sync being
> active.
> What do you think?
> 
> --
> Andriy Gapon

--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/Tbc5170d4f71ec486-M4666ec33d56c533e901ac0d4
Powered by Topicbox: https://topicbox.com

Re: [PATCH 0/8] KEYS: keyctl operations for asymmetric keys [ver #2]

[George Wilson]

On Thu, Jun 23, 2016 at 02:47:34PM +0100, David Howells wrote:
> 
> Here's a set of patches that provides keyctl access for asymmetric keys,
> including a query function, and functions to do encryption, decryption,
> signature creation and signature verification.
> 
> I've added a PKCS#8 asymmetric key parser so that you can load an RSA private
> key into the kernel.  Currently only DER-encoded and unencrypted PKCS#8 is
> supported.  Encryption and verification can use a public key from an X.509
> cert, but signing and decryption require a private key, though encryption and
> verification can use that too.
> 
> Example usage:
> 
>   j=`openssl pkcs8 -in private_key.pem -topk8 -nocrypt -outform DER | \
>   keyctl padd asymmetric foo @s`
>   echo -n abcdefghijklmnopqrst >/tmp/data
>   keyctl pkey_encrypt $j /tmp/data enc=pkcs1 >/tmp/enc
>   keyctl pkey_decrypt $j /tmp/enc enc=pkcs1 >/tmp/dec
>   cmp /tmp/data /tmp/dec
>   keyctl pkey_sign $j /tmp/data enc=pkcs1 hash=sha1 >/tmp/sig
>   keyctl pkey_verify $j /tmp/data /tmp/sig enc=pkcs1 hash=sha1
> 
> Changes:
> 
>  (*) I've taken out the password parameters for the moment as there isn't
>  yet a subtype that uses them.  I have, however, left space in the
>  keyctl UAPI to add them back later.
> 
> The kernel patches can be found here also:
> 
>   
> http://git.kernel.org/cgit/linux/kernel/git/dhowells/linux-fs.git/log/?h=keys-asym-keyctl
> 
> The keyutils changes needed can be found here:
> 
>   
> http://git.kernel.org/cgit/linux/kernel/git/dhowells/keyutils.git/log/?h=pkey
> 
> David
> ---
> David Howells (8):
>   KEYS: Provide key type operations for asymmetric key ops
>   KEYS: Provide keyctls to drive the new key type ops for asymmetric keys
>   KEYS: Provide missing asymmetric key subops for new key type ops
>   KEYS: Make the X.509 and PKCS7 parsers supply the sig encoding type
>   KEYS: Provide software public key query function
>   KEYS: Allow the public_key struct to hold a private key
>   KEYS: Implement encrypt, decrypt and sign for software asymmetric key
>   KEYS: Implement PKCS#8 RSA Private Key parser
> 
> 
>  Documentation/crypto/asymmetric-keys.txt  |   26 ++
>  Documentation/security/keys.txt   |  217 +++
>  crypto/asymmetric_keys/Kconfig|   10 +
>  crypto/asymmetric_keys/Makefile   |   13 +
>  crypto/asymmetric_keys/asymmetric_keys.h  |3 
>  crypto/asymmetric_keys/asymmetric_type.c  |   59 +
>  crypto/asymmetric_keys/pkcs7_parser.c |1 
>  crypto/asymmetric_keys/pkcs8.asn1 |   24 ++
>  crypto/asymmetric_keys/pkcs8_parser.c |  184 +
>  crypto/asymmetric_keys/public_key.c   |  195 --
>  crypto/asymmetric_keys/signature.c|   95 +
>  crypto/asymmetric_keys/x509_cert_parser.c |   21 +-
>  include/crypto/public_key.h   |   14 +
>  include/keys/asymmetric-subtype.h |9 +
>  include/linux/key-type.h  |   11 +
>  include/linux/keyctl.h|   46 
>  include/uapi/linux/keyctl.h   |   30 +++
>  security/keys/Makefile|1 
>  security/keys/compat.c|   18 ++
>  security/keys/internal.h  |   39 
>  security/keys/keyctl.c|   24 ++
>  security/keys/keyctl_pkey.c   |  323 
> +
>  22 files changed, 1319 insertions(+), 44 deletions(-)
>  create mode 100644 crypto/asymmetric_keys/pkcs8.asn1
>  create mode 100644 crypto/asymmetric_keys/pkcs8_parser.c
>  create mode 100644 include/linux/keyctl.h
>  create mode 100644 security/keys/keyctl_pkey.c
> 

Hi David,

To lend support for this patch series, we have a compelling use case in 
OpenPOWER
firmware.  We need to verify that our key management command queue elements are
properly signed.  However, we have limited flash storage space and want to avoid
dragging in a userspace library.  Sub Swaminathan (on copy) has prototyped a
solution using these patches.  We'd very much like to see them applied.

-- 
George Wilson
IBM Linux Technology Center
Security Development

Re: [PATCH 0/8] KEYS: keyctl operations for asymmetric keys [ver #2]

[George Wilson]

On Thu, Jun 23, 2016 at 02:47:34PM +0100, David Howells wrote:
> 
> Here's a set of patches that provides keyctl access for asymmetric keys,
> including a query function, and functions to do encryption, decryption,
> signature creation and signature verification.
> 
> I've added a PKCS#8 asymmetric key parser so that you can load an RSA private
> key into the kernel.  Currently only DER-encoded and unencrypted PKCS#8 is
> supported.  Encryption and verification can use a public key from an X.509
> cert, but signing and decryption require a private key, though encryption and
> verification can use that too.
> 
> Example usage:
> 
>   j=`openssl pkcs8 -in private_key.pem -topk8 -nocrypt -outform DER | \
>   keyctl padd asymmetric foo @s`
>   echo -n abcdefghijklmnopqrst >/tmp/data
>   keyctl pkey_encrypt $j /tmp/data enc=pkcs1 >/tmp/enc
>   keyctl pkey_decrypt $j /tmp/enc enc=pkcs1 >/tmp/dec
>   cmp /tmp/data /tmp/dec
>   keyctl pkey_sign $j /tmp/data enc=pkcs1 hash=sha1 >/tmp/sig
>   keyctl pkey_verify $j /tmp/data /tmp/sig enc=pkcs1 hash=sha1
> 
> Changes:
> 
>  (*) I've taken out the password parameters for the moment as there isn't
>  yet a subtype that uses them.  I have, however, left space in the
>  keyctl UAPI to add them back later.
> 
> The kernel patches can be found here also:
> 
>   
> http://git.kernel.org/cgit/linux/kernel/git/dhowells/linux-fs.git/log/?h=keys-asym-keyctl
> 
> The keyutils changes needed can be found here:
> 
>   
> http://git.kernel.org/cgit/linux/kernel/git/dhowells/keyutils.git/log/?h=pkey
> 
> David
> ---
> David Howells (8):
>   KEYS: Provide key type operations for asymmetric key ops
>   KEYS: Provide keyctls to drive the new key type ops for asymmetric keys
>   KEYS: Provide missing asymmetric key subops for new key type ops
>   KEYS: Make the X.509 and PKCS7 parsers supply the sig encoding type
>   KEYS: Provide software public key query function
>   KEYS: Allow the public_key struct to hold a private key
>   KEYS: Implement encrypt, decrypt and sign for software asymmetric key
>   KEYS: Implement PKCS#8 RSA Private Key parser
> 
> 
>  Documentation/crypto/asymmetric-keys.txt  |   26 ++
>  Documentation/security/keys.txt   |  217 +++
>  crypto/asymmetric_keys/Kconfig|   10 +
>  crypto/asymmetric_keys/Makefile   |   13 +
>  crypto/asymmetric_keys/asymmetric_keys.h  |3 
>  crypto/asymmetric_keys/asymmetric_type.c  |   59 +
>  crypto/asymmetric_keys/pkcs7_parser.c |1 
>  crypto/asymmetric_keys/pkcs8.asn1 |   24 ++
>  crypto/asymmetric_keys/pkcs8_parser.c |  184 +
>  crypto/asymmetric_keys/public_key.c   |  195 --
>  crypto/asymmetric_keys/signature.c|   95 +
>  crypto/asymmetric_keys/x509_cert_parser.c |   21 +-
>  include/crypto/public_key.h   |   14 +
>  include/keys/asymmetric-subtype.h |9 +
>  include/linux/key-type.h  |   11 +
>  include/linux/keyctl.h|   46 
>  include/uapi/linux/keyctl.h   |   30 +++
>  security/keys/Makefile|1 
>  security/keys/compat.c|   18 ++
>  security/keys/internal.h  |   39 
>  security/keys/keyctl.c|   24 ++
>  security/keys/keyctl_pkey.c   |  323 
> +
>  22 files changed, 1319 insertions(+), 44 deletions(-)
>  create mode 100644 crypto/asymmetric_keys/pkcs8.asn1
>  create mode 100644 crypto/asymmetric_keys/pkcs8_parser.c
>  create mode 100644 include/linux/keyctl.h
>  create mode 100644 security/keys/keyctl_pkey.c
> 

Hi David,

To lend support for this patch series, we have a compelling use case in 
OpenPOWER
firmware.  We need to verify that our key management command queue elements are
properly signed.  However, we have limited flash storage space and want to avoid
dragging in a userspace library.  Sub Swaminathan (on copy) has prototyped a
solution using these patches.  We'd very much like to see them applied.

-- 
George Wilson
IBM Linux Technology Center
Security Development

Re: [PATCH 0/8] KEYS: keyctl operations for asymmetric keys [ver #2]

[George Wilson]

On Thu, Jun 23, 2016 at 02:47:34PM +0100, David Howells wrote:
> 
> Here's a set of patches that provides keyctl access for asymmetric keys,
> including a query function, and functions to do encryption, decryption,
> signature creation and signature verification.
> 
> I've added a PKCS#8 asymmetric key parser so that you can load an RSA private
> key into the kernel.  Currently only DER-encoded and unencrypted PKCS#8 is
> supported.  Encryption and verification can use a public key from an X.509
> cert, but signing and decryption require a private key, though encryption and
> verification can use that too.
> 
> Example usage:
> 
>   j=`openssl pkcs8 -in private_key.pem -topk8 -nocrypt -outform DER | \
>   keyctl padd asymmetric foo @s`
>   echo -n abcdefghijklmnopqrst >/tmp/data
>   keyctl pkey_encrypt $j /tmp/data enc=pkcs1 >/tmp/enc
>   keyctl pkey_decrypt $j /tmp/enc enc=pkcs1 >/tmp/dec
>   cmp /tmp/data /tmp/dec
>   keyctl pkey_sign $j /tmp/data enc=pkcs1 hash=sha1 >/tmp/sig
>   keyctl pkey_verify $j /tmp/data /tmp/sig enc=pkcs1 hash=sha1
> 
> Changes:
> 
>  (*) I've taken out the password parameters for the moment as there isn't
>  yet a subtype that uses them.  I have, however, left space in the
>  keyctl UAPI to add them back later.
> 
> The kernel patches can be found here also:
> 
>   
> http://git.kernel.org/cgit/linux/kernel/git/dhowells/linux-fs.git/log/?h=keys-asym-keyctl
> 
> The keyutils changes needed can be found here:
> 
>   
> http://git.kernel.org/cgit/linux/kernel/git/dhowells/keyutils.git/log/?h=pkey
> 
> David
> ---
> David Howells (8):
>   KEYS: Provide key type operations for asymmetric key ops
>   KEYS: Provide keyctls to drive the new key type ops for asymmetric keys
>   KEYS: Provide missing asymmetric key subops for new key type ops
>   KEYS: Make the X.509 and PKCS7 parsers supply the sig encoding type
>   KEYS: Provide software public key query function
>   KEYS: Allow the public_key struct to hold a private key
>   KEYS: Implement encrypt, decrypt and sign for software asymmetric key
>   KEYS: Implement PKCS#8 RSA Private Key parser
> 
> 
>  Documentation/crypto/asymmetric-keys.txt  |   26 ++
>  Documentation/security/keys.txt   |  217 +++
>  crypto/asymmetric_keys/Kconfig|   10 +
>  crypto/asymmetric_keys/Makefile   |   13 +
>  crypto/asymmetric_keys/asymmetric_keys.h  |3 
>  crypto/asymmetric_keys/asymmetric_type.c  |   59 +
>  crypto/asymmetric_keys/pkcs7_parser.c |1 
>  crypto/asymmetric_keys/pkcs8.asn1 |   24 ++
>  crypto/asymmetric_keys/pkcs8_parser.c |  184 +
>  crypto/asymmetric_keys/public_key.c   |  195 --
>  crypto/asymmetric_keys/signature.c|   95 +
>  crypto/asymmetric_keys/x509_cert_parser.c |   21 +-
>  include/crypto/public_key.h   |   14 +
>  include/keys/asymmetric-subtype.h |9 +
>  include/linux/key-type.h  |   11 +
>  include/linux/keyctl.h|   46 
>  include/uapi/linux/keyctl.h   |   30 +++
>  security/keys/Makefile|1 
>  security/keys/compat.c|   18 ++
>  security/keys/internal.h  |   39 
>  security/keys/keyctl.c|   24 ++
>  security/keys/keyctl_pkey.c   |  323 
> +
>  22 files changed, 1319 insertions(+), 44 deletions(-)
>  create mode 100644 crypto/asymmetric_keys/pkcs8.asn1
>  create mode 100644 crypto/asymmetric_keys/pkcs8_parser.c
>  create mode 100644 include/linux/keyctl.h
>  create mode 100644 security/keys/keyctl_pkey.c
> 

Hi David,

To lend support for this patch series, we have a compelling use case in 
OpenPOWER
firmware.  We need to verify that our key management command queue elements are
properly signed.  However, we have limited flash storage space and want to avoid
dragging in a userspace library.  Sub Swaminathan (on copy) has prototyped a
solution using these patches.  We'd very much like to see them applied.

-- 
George Wilson
IBM Linux Technology Center
Security Development

[developer] Re: [zfs] ZIO_IOCTL_PIPELINE excludes ZIO_STAGE_VDEV_IO_DONE

[George Wilson]

Andriy,

Assuming this is disk specific, you could look at doing this in the IOCTL
callback -- "vdev_disk_ioctl_done".

Thanks,
George

On Fri, Sep 15, 2017 at 11:31 AM Andriy Gapon <a...@freebsd.org> wrote:

>
> In FreeBSD we create a struct bio object for each I/O request that goes to
> a
> disk.  So, we create bio-s for all vdev zio-s that are to be passed down
> including read, write, ioctl (disk cache flush) zio-s.  Previously, we
> freed
> those bio-s in a callback from the I/O subsystem.  But since the ABD
> change we
> had to move that to the io_done stage, because the callback's context is
> too
> restrictive for abd_return_buf / abd_return_buf_copy (at least for the
> current
> ABD implementation which is taken as is from illumos).  So, currently we
> are
> leaking bio-s created for ioctl zio-s because the io_done stage is not
> called
> for them.
>
> There is probably another way to fix this problem as the ioctl zio-s have
> nothing to do with ABD.  So, we could free their bio-s in the callback as
> we did
> before and handle the regular i/o bio-s in the io_done stage as we do
> now.  This
> should work, but at the cost of the less uniform bio handling code.
>
> So, I wanted to see which of the solutions would be better from the point
> of
> view of the general zio pipeline architecture.
>
> Thank you.
>
> On 14/09/2017 18:48, George Wilson wrote:
> > Andriy,
> >
> > The ZIO_STAGE_VDEV_IO_DONE is not necessary for ZIO_IOCTL_PIPELINE and
> that's
> > why it was removed. At least in illumos, there is no functional reason
> for
> > calling it. To add it back would require a small amount of change which
> should
> > not be a big deal. Can you provide some context around the FreeBSD bug?
> >
> > Thanks,
> > George
> >
> >
> > On Thu, Sep 14, 2017 at 12:45 AM Andriy Gapon <a...@freebsd.org
> > <mailto:a...@freebsd.org>> wrote:
> >
> >
> > Does anyone know why ZIO_IOCTL_PIPELINE does not include
> ZIO_STAGE_VDEV_IO_DONE?
> > Or, in other words, why ZIO_IOCTL_PIPELINE actively excludes it by
> not using
> > ZIO_VDEV_IO_STAGES?
> >
> > It seems that ZIO_IOCTL_PIPELINE had ZIO_VDEV_IO_STAGES until this
> commit:
> > > commit e14bb3258d05c1b1077e2db7cf77088924e56919
> > > Author: Jeff Bonwick <jeff.bonw...@sun.com>
> > > Date:   Mon Sep 29 18:13:58 2008 -0700
> > >6754011 SPA 3.0: lock breakup, i/o pipeline refactoring, device
> failure
> > handling
> > >6667208 zfs/zpool commands on failed pool should not hang
> > >6430480 grabbing config lock as writer during I/O load can take
> > excessively long
> >
> > Of course, the commit message does not have any detailed
> explanations and the
> > referenced bugs are not publicly accessible.   And it was almost 9
> years ago.
> >
> > I wonder if the change was because of anything specific to illumos
> vdev_disk.
> > I think that it would be totally okay to enable
> ZIO_STAGE_VDEV_IO_DONE with
> > FreeBSD vdev_geom.  In fact, right now ZFS/FreeBSD has a bug because
> the done
> > stage is not executed.
> >
> > --
> > Andriy Gapon
> >
> > --
> > illumos-zfs
> > Archives:
> >
> https://illumos.topicbox.com/groups/zfs/discussions/T7be335e1da154b96-Mea10ec00d17c40849ece338f
> > Powered by Topicbox: https://topicbox.com
> >
> > *illumos-zfs* | Archives
> > <
> https://illumos.topicbox.com/groups/zfs/discussions/T7be335e1da154b96-M0cc0a74989d1c00edb4b391a
> > | Powered by Topicbox <https://topicbox.com>
>
>
> --
> Andriy Gapon
>

--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T4b8a967c63e22769-M66e2a94be7620f7b67d7965b
Powered by Topicbox: https://topicbox.com

[developer] Re: [zfs] ZIO_IOCTL_PIPELINE excludes ZIO_STAGE_VDEV_IO_DONE

[George Wilson]

Andriy,

The ZIO_STAGE_VDEV_IO_DONE is not necessary for ZIO_IOCTL_PIPELINE and
that's why it was removed. At least in illumos, there is no functional
reason for calling it. To add it back would require a small amount of
change which should not be a big deal. Can you provide some context around
the FreeBSD bug?

Thanks,
George


On Thu, Sep 14, 2017 at 12:45 AM Andriy Gapon  wrote:

>
> Does anyone know why ZIO_IOCTL_PIPELINE does not include
> ZIO_STAGE_VDEV_IO_DONE?
> Or, in other words, why ZIO_IOCTL_PIPELINE actively excludes it by not
> using
> ZIO_VDEV_IO_STAGES?
>
> It seems that ZIO_IOCTL_PIPELINE had ZIO_VDEV_IO_STAGES until this commit:
> > commit e14bb3258d05c1b1077e2db7cf77088924e56919
> > Author: Jeff Bonwick 
> > Date:   Mon Sep 29 18:13:58 2008 -0700
> >6754011 SPA 3.0: lock breakup, i/o pipeline refactoring, device
> failure handling
> >6667208 zfs/zpool commands on failed pool should not hang
> >6430480 grabbing config lock as writer during I/O load can take
> excessively long
> 
> Of course, the commit message does not have any detailed explanations and
> the
> referenced bugs are not publicly accessible.   And it was almost 9 years
> ago.
> 
> I wonder if the change was because of anything specific to illumos
> vdev_disk.
> I think that it would be totally okay to enable ZIO_STAGE_VDEV_IO_DONE with
> FreeBSD vdev_geom.  In fact, right now ZFS/FreeBSD has a bug because the
> done
> stage is not executed.
> 
> --
> Andriy Gapon

--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T4b8a967c63e22769-Mf919ed11d9fcd748d8163047
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] 8473 scrub does not detect errors on active spares (#422)

[George Wilson]

grwilson approved this pull request.





-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/422#pullrequestreview-58790150
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/Te100c9e22da2b1d8-M7220f80d4eda69a462372957
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] 8423 Implement large_dnode pool feature (#409)

[George Wilson]

grwilson approved this pull request.





-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/409#pullrequestreview-58659280
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/Tfefe6318017dd4c7-M880a8d1f8a373d1489e6a948
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] 8423 Implement large_dnode pool feature (#409)

[George Wilson]

grwilson commented on this pull request.

There are lots of good comments in the pull request that aren't referenced in 
the actual code. It would be good to incorporate some of these as comments.

/*
 * All forms of zfs create (create, mkdir, mkxattrdir, symlink)
 * eventually end up in zfs_mknode(), which assigns the object's
-* creation time and generation number.  The generic VOP_CREATE()
-* doesn't have either concept, so we smuggle the values inside
-* the vattr's otherwise unused va_ctime and va_nblocks fields.
+* creation time, generation number, and dnode slot count. The
+* generic VOP_CREATE() has no concept of these attributes, so
+* we smuggle the values inside * the vattr's otherwise unused

Extra "*" -- `inside * the vattr's`

/*
 * All forms of zfs create (create, mkdir, mkxattrdir, symlink)
 * eventually end up in zfs_mknode(), which assigns the object's
-* creation time and generation number.  The generic VOP_CREATE()
-* doesn't have either concept, so we smuggle the values inside
-* the vattr's otherwise unused va_ctime and va_nblocks fields.
+* creation time, generation number, and dnode slot count. The
+* generic VOP_CREATE() has no concept of these attributes, so
+* we smuggle the values inside * the vattr's otherwise unused
+* va_ctime, va_nblocks, and va_nlink fields.

shouldn't this comment include `va_fsid` or should the code below be using 
`va_nlink` instead of `va_fsid`?

> + } else if (ds && ds->ds_feature_inuse[SPA_FEATURE_LARGE_DNODE]) {
+   /*
+* For large_dnode datasets, scan from the beginning of the
+* dnode block to find the starting offset. This is needed
+* because objectp could be part of a large dnode so we can't
+* assume it's a hole even if dmu_object_info() returns ENOENT.
+*/
+   int epb = DNODE_BLOCK_SIZE >> DNODE_SHIFT;
+   int skip;
+   uint64_t i;
+
+   for (i = *objectp & ~(epb - 1); i <= *objectp; i += skip) {
+   dmu_object_info_t doi;
+
+   error = dmu_object_info(os, i, );
+   if (error)

NIT: error != 0

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/409#pullrequestreview-57301052
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/Tfefe6318017dd4c7-Mccb2ecbc27ed810810878cf9
Powered by Topicbox: https://topicbox.com

Re: [tpmdd-devel] tpm: read burstcount from TPM_STS in one 32-bit transaction

[George Wilson]

On Tue, Jul 25, 2017 at 10:36:11AM -0700, James Bottomley wrote:
> On Tue, 2017-07-25 at 15:04 +0200, Michal Suchánek wrote:
> > Hello,
> > 
> > in commit 9754d45e9970 ("tpm: read burstcount from TPM_STS in one
> > 32-bit transaction") you change reading of two 8-bit values to one
> > 32bit read. This is obviously wrong wrt endianess unless the
> > underlying tpm_tis_read32 does endian conversion. 
> 
> Some of the bus read primitives do do endianness conversions.  The
> problem is with the SPI attachment, which has unclear endianness.  A
> standard PCI bus attachment uses ioread32() which automatically
> transforms from a little endian bus to the cpu endianness, however SPI
> is forced to transfer the bytes one at a time over the serial bus and
> then transform.  The assumption seems to be that the TIS TPM is
> replying in little endian format when SPI connected.
> 
> We can probably get the PPC people to confirm this, I believe they have
> a SPI attached TPM.

All the current OpenPOWER hardware designs I'm aware of have the TPM on
I2C.  Trusted Computing support in OpenPOWER firmware depends on it
being on I2C.

> 
> James
> 
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> tpmdd-devel mailing list
> tpmdd-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/tpmdd-devel


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
tpmdd-devel mailing list
tpmdd-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/tpmdd-devel

Re: [tpmdd-devel] tpm: read burstcount from TPM_STS in one 32-bit transaction

[George Wilson]

On Tue, Jul 25, 2017 at 10:36:11AM -0700, James Bottomley wrote:
> On Tue, 2017-07-25 at 15:04 +0200, Michal Suchánek wrote:
> > Hello,
> > 
> > in commit 9754d45e9970 ("tpm: read burstcount from TPM_STS in one
> > 32-bit transaction") you change reading of two 8-bit values to one
> > 32bit read. This is obviously wrong wrt endianess unless the
> > underlying tpm_tis_read32 does endian conversion. 
> 
> Some of the bus read primitives do do endianness conversions.  The
> problem is with the SPI attachment, which has unclear endianness.  A
> standard PCI bus attachment uses ioread32() which automatically
> transforms from a little endian bus to the cpu endianness, however SPI
> is forced to transfer the bytes one at a time over the serial bus and
> then transform.  The assumption seems to be that the TIS TPM is
> replying in little endian format when SPI connected.
> 
> We can probably get the PPC people to confirm this, I believe they have
> a SPI attached TPM.

All the current OpenPOWER hardware designs I'm aware of have the TPM on
I2C.  Trusted Computing support in OpenPOWER firmware depends on it
being on I2C.

> 
> James
> 
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> tpmdd-devel mailing list
> tpmdd-de...@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/tpmdd-devel

Re: [tpmdd-devel] tpm: read burstcount from TPM_STS in one 32-bit transaction

[George Wilson]

On Tue, Jul 25, 2017 at 10:36:11AM -0700, James Bottomley wrote:
> On Tue, 2017-07-25 at 15:04 +0200, Michal Suchánek wrote:
> > Hello,
> > 
> > in commit 9754d45e9970 ("tpm: read burstcount from TPM_STS in one
> > 32-bit transaction") you change reading of two 8-bit values to one
> > 32bit read. This is obviously wrong wrt endianess unless the
> > underlying tpm_tis_read32 does endian conversion. 
> 
> Some of the bus read primitives do do endianness conversions.  The
> problem is with the SPI attachment, which has unclear endianness.  A
> standard PCI bus attachment uses ioread32() which automatically
> transforms from a little endian bus to the cpu endianness, however SPI
> is forced to transfer the bytes one at a time over the serial bus and
> then transform.  The assumption seems to be that the TIS TPM is
> replying in little endian format when SPI connected.
> 
> We can probably get the PPC people to confirm this, I believe they have
> a SPI attached TPM.

All the current OpenPOWER hardware designs I'm aware of have the TPM on
I2C.  Trusted Computing support in OpenPOWER firmware depends on it
being on I2C.

> 
> James
> 
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> tpmdd-devel mailing list
> tpmdd-de...@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/tpmdd-devel

Re: [Bug-wget] Dubious Dash

[Frederick George Wilson]

Glad to help.  And if someone would kindly delete my duplicate post "Dash
Dubious," I would greatly appreciate.

Where would I find the Examples directory?  Do I need to download wget2?

On Thu, Jul 20, 2017 at 1:56 AM, Tim Rühsen <tim.rueh...@gmx.de> wrote:

> Thanks for the pointer.
>
>
> I opened an issue for Wget2 at https://gitlab.com/gnuwget/wget2/issues/234
> .
>
>
> With Best Regards, Tim
>
>
>
> On 07/20/2017 04:44 AM, Frederick George Wilson wrote:
> > Those pesky manifest.mpd URLs -- the noncompliant (unverifiable) ones,
> that
> > is!
> >
> >  I go to great lengths to unchain myself from the browser.  Can anyone
> > recommend a concise wget command line that mimics Mozilla while
> retrieving
> > a 150 to 300 KB file (via "links.txt)" roughly every three (3) seconds
> from
> > a 24/7 live streaming site?
> > Relevant URL:
> > http://vs-dash-ww-rd-live.bbcfmt.hs.llnwd.net/al/
> lossless/client_manifest.mpd
> > Bonus challenge: hear one (1) hour of working, unbroken sound from the
> Beeb
> > (BBC Radio 3 and her experimental FLAC stream) using wget.  (Note: the
> .m4s
> > file segments have a very short shelf life, i.e., < 10 mins).  It may be
> > closer to a < 5 min. shelf life.  I couldn't be sure, as I was too busy
> > clacking out my clunky command line(s).
> > P.S. - If you're on anything more modern than Windows XP or Vista, you
> can
> > simply plunk that .mpd URL into the latest VLC 3.0.0 nightly build, and
> VLC
> > will do all the work -- but that's no fun!
> >
>
>

[developer] Re: [openzfs/openzfs] 7910 l2arc_write_buffers() may write beyond target_sz (#310)

[George Wilson]

grwilson approved this pull request.





-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/310#pullrequestreview-43429326
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/Td414270217188b5f-M85068777e76d3cff5873bebf
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] 7910 l2arc_write_buffers() may write beyond target_sz (#310)

[George Wilson]

grwilson commented on this pull request.



>   ARCSTAT_INCR(arcstat_l2_size, write_sz);
-   ARCSTAT_INCR(arcstat_l2_asize, write_asize);
-   vdev_space_update(dev->l2ad_vdev, write_asize, 0, 0);
+   ARCSTAT_INCR(arcstat_l2_asize, write_psize);

I agree. These variables and their meanings have changed over time with new 
features. This cleanup will help developers. Thank you. We'll tackle asize in 
the future. 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/310#discussion_r121385972
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/Td414270217188b5f-M0c53a8f2f6030b4a46b5e2ed
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] 7303 dynamic metaslab selection (#176)

[George Wilson]

@grwilson pushed 2 commits.

7900469  Merge branch 'master' into metaslab_selection
86d718e  git-zfs-make-Wed Sep 21 15:56:55 EDT 2016


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/176/files/08a54ded12775e93db1d7bb984cd41876d82d5d8..86d718e3a3ae1987734e995451925db0734df98c



---
openzfs-developer
Archives: https://www.listbox.com/member/archive/274414/=now
RSS Feed: https://www.listbox.com/member/archive/rss/274414/28015062-cce53afa
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=28015062_secret=28015062-f966d51c
Powered by Listbox: http://www.listbox.com

Re: [discuss] Removing gendered pronouns from illumos source

[George Wilson]

+1 Thanks guys!

On Mon, Aug 1, 2016 at 4:52 PM, Matthew Ahrens  wrote:

> Thank you for your work on this, Chris and DJ.  I look forward to seeing
> this change upstreamed to illumos.
>
> --matt
>
> On Mon, Aug 1, 2016 at 4:36 PM, Dj Hoffman  wrote:
>
>> Hello all,
>> At Delphix we are working on a patch to remove gendered pronouns
>> (he/she/his/hers/him/her/himself/herself) in favor of gender-neutral
>> singular pronouns (they(1)/them/themself(2)/it/its) when they don't
>> reference specific people. Chris Williamson and I found numerous references
>> where users are referred to as "he", which we replaced with variants on
>> "they". In addition, we found instances where threads, servers, functions,
>> and other inanimate objects were referred to as "he" and we replaced them
>> with variants on "it". If you have any questions about why we feel this
>> change is necessary please do not hesitate to email us directly or reply to
>> this thread. We will be putting out a review on developer once it has gone
>> through our internal review process.
>>
>> [1] https://en.wikipedia.org/wiki/Singular_they
>> [2] http://www.oxforddictionaries.com/words/themselves-or-themself
>>
>> Sincerely,
>> --
>>
>> DJ Hoffman
>>
>> Member of Technical Staff
>>
>> http://www.delphix.com
>>
>>
> *illumos-discuss* | Archives
> 
>  |
> Modify
> 
> Your Subscription 
>



---
illumos-discuss
Archives: https://www.listbox.com/member/archive/182180/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182180/21175430-2e6923be
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=21175430_secret=21175430-6a77cda4
Powered by Listbox: http://www.listbox.com

Re: [tpmdd-devel] Regarding recently Added TPM2.0 support to the Nuvoton i2c driver

[George Wilson]

On Wed, Jul 27, 2016 at 10:31:52AM -0600, Jason Gunthorpe wrote:
> On Wed, Jul 27, 2016 at 11:05:14AM -0500, George Wilson wrote:
> 
> > > Yes, generally Linux expects DT to be set correctly by the boot
> > > firmware. Early firmware needs to know the TPM type anyhow to do the
> > > TPM setup, so this doesn't seem like a realistic scenario.
> > 
> > A reset is required after upgrade/downgrade.  But the version still
> > needs to be detected by the firmware somehow.  It could be configured
> > manually in firmware state after the upgrade/downgrade to properly set
> > the property, which seems much less desirable than a probe.
> 
> Well, the firmware has to take care of this. If the firmware wants to
> support switchable firmware it needs to be able to do probe that works
> with all the firmware versions.
> 
> ACPI and DT both expect the TPM version to be passed to the OS, it is
> up to the firmware to do that correctly..
> 
> The fact you've already seen TPM2 probing failures makes me very
> reluctant to turn it on more broadly, and maybe even think we should
> get rid of it from tis too..

Cool, that clarifies it precisely:  the expectation is that firmware will
do any necessary probing and pass the property up in the device tree.  All
drivers supporting OF must use the data property as a flag for 2.0 for
models capable of 2.0 function.

> 
> Jason
> 

-- 
George Wilson
IBM Linux Technology Center
Security Development


--
___
tpmdd-devel mailing list
tpmdd-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/tpmdd-devel

Re: [tpmdd-devel] Regarding recently Added TPM2.0 support to the Nuvoton i2c driver

[George Wilson]

On Tue, Jul 26, 2016 at 03:03:44PM -0600, Jason Gunthorpe wrote:
> On Tue, Jul 26, 2016 at 03:39:02PM -0500, George Wilson wrote:
> > > Generally speaking probing is somewhat discouraged, currently we only
> > > probe for PC platform tis (and even that might be a mistake), all
> > > other drivers are designed to be explicit.
> > 
> > How should field upgradable/downgradable TPMs be handled since hardcoding
> > the version in the device tree might give the wrong answer?  Would early
> > firmware be expected to probe nonetheless and set the right device tree
> > property?
> 
> Is that a real thing?

Yep, it's a thing.  I know of at least 2 parts from 2 different
suppliers that are field upgradable/downgradable.

> 
> Yes, generally Linux expects DT to be set correctly by the boot
> firmware. Early firmware needs to know the TPM type anyhow to do the
> TPM setup, so this doesn't seem like a realistic scenario.

A reset is required after upgrade/downgrade.  But the version still
needs to be detected by the firmware somehow.  It could be configured
manually in firmware state after the upgrade/downgrade to properly set
the property, which seems much less desirable than a probe.

> 
> For TPM we made a somewhat arbitary choice that TPM2 has to be
> explicit. If there are real systems that benefit from auto-probing it
> could be revisited..

I think it's as necessary - at least at the firmware level - as for
x86_64.

> 
> But, to be honest, I'm not certain how robust our probe technique is,
> and I think we should avoid probing, since TCG didn't design an
> approved detection sequence (??).

We did see issues with some older 1.2 TPMs that hung when probed by the
kernel with the wrong device driver but that hasn't been an issue for
some time.  It looks like UEFI ultimately does probe and likely that's
required for other platforms.  I don't think it's safe to assume a
TPM is always loaded with a particular firmware version across hard
resets.

> 
> Jason
> 

-- 
George Wilson
IBM Linux Technology Center
Security Development


--
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity planning
reports.http://sdm.link/zohodev2dev
___
tpmdd-devel mailing list
tpmdd-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/tpmdd-devel

Re: [tpmdd-devel] Regarding recently Added TPM2.0 support to the Nuvoton i2c driver

[George Wilson]

On Tue, Jul 26, 2016 at 02:17:11PM -0600, Jason Gunthorpe wrote:
> On Tue, Jul 26, 2016 at 11:44:43PM +0530, Nayna wrote:
> > I got these questions while testing some TPM2.0 stuff using the kernel 
> > code from repo having this patch and am using Nuvoton TPM.
> > 
> > #1. It seems that support is added only for following device-ids.
> >  {.compatible = "nuvoton,npct501"},
> >  {.compatible = "winbond,wpct301"},
> >  {.compatible = "nuvoton,npct601", .data = OF_IS_TPM2},
> > 
> > So, was wondering about why device id nuvoton,npct650  wasn't added for 
> > the support ?
> 
> Yes, that was the device ID list for Nuvoton.
> 
> It is convention in device tree to include older device IDs if the
> device is compatible.
> 
> So you might do
> 
>  compatible = "nuvoton,npct650", "nuvoton,npct601"
> 
> Andrew, is 601 even the right name?
> 
> > Was it expected to work with some wild-card type device-id as specified 
> > in the first line of description comment of file i.e. npct6XX. ?
> 
> No.
> 
> > So, why is there hard-coded checking and not using tpm2_probe() method 
> > which is itself based on direct TPM hardware response for setting the 
> > TPM2 flag. ? Is there something I am missing in the design which 
> > mandates to have .data set as OF_IS_TPM2.
> 
> Generally speaking probing is somewhat discouraged, currently we only
> probe for PC platform tis (and even that might be a mistake), all
> other drivers are designed to be explicit.

How should field upgradable/downgradable TPMs be handled since hardcoding
the version in the device tree might give the wrong answer?  Would early
firmware be expected to probe nonetheless and set the right device tree
property?

> 
> Jason
> 
> --
> What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
> patterns at an interface-level. Reveals which users, apps, and protocols are 
> consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
> J-Flow, sFlow and other flows. Make informed decisions using capacity planning
> reports.http://sdm.link/zohodev2dev
> ___
> tpmdd-devel mailing list
> tpmdd-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/tpmdd-devel
> 

-- 
George Wilson
IBM Linux Technology Center
Security Development


--
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity planning
reports.http://sdm.link/zohodev2dev
___
tpmdd-devel mailing list
tpmdd-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/tpmdd-devel

Re: [developer] spa_namespace_lock vs sync thread

[George Wilson]

Andriy,

I think this is basically a rule, although I don't think it's stated
anywhere. We do rely heavily on this locking strategy since there are many
places that will hold the namespace lock to prevent spa config changes but
yet wait for a txg to sync.

Is it honored everywhere? Well, I hope so but there probably hasn't been a
detailed analysis done to see if there might be places where we do
something like a spa_open() from some obscure path in the sync thread. I
would hope that our testing would have discovered that quickly and
hopefully developers will sufficiently exercise their code to make sure
they don't violate this.

Have you seen instances where we deadlock as a result of this?

Thanks,
George

On Fri, Jul 8, 2016 at 9:40 AM, Andriy Gapon  wrote:

> 
> I see a few places in the code that do the following:
> mutex_enter(_namespace_lock);
> dsl_sync_task(...);
> mutex_exit(_namespace_lock);
> One example is spa_change_guid().
> 
> In my understanding this implies that the sync thread must never acquire
> spa_namespace_lock.  Is this a real rule?  Do we always honor it?
> 
> Thank you!
> --
> Andriy Gapon
> 



---
openzfs-developer
Archives: https://www.listbox.com/member/archive/274414/=now
RSS Feed: https://www.listbox.com/member/archive/rss/274414/28015062-cce53afa
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=28015062_secret=28015062-f966d51c
Powered by Listbox: http://www.listbox.com

Re: [developer] panic: solaris assert: ba.ba_phys->bt_bytes == 0 (0x400 == 0x0)

[George Wilson]

Andriy,

Can you give me some details about how you're able to reproduce this panic.
I would like to help debug this. I'm also looking into the range_tree()
 panic, so any details you can provide would be very helpful.

If you can publish the crash dumps, I can also download them and take a
look.

Thanks,
George

On Wed, Jun 22, 2016 at 4:53 PM, Andriy Gapon  wrote:

>
> Igor,
>
> your suggestion was certainly a good one, however I took a path of a
> lesser effort and tested my workload on the latest illumos kernel:
>
> panic[cpu3]/thread=ff000bc56c40: assertion failed:
> ba.ba_phys->bt_bytes == 0 (0x400 == 0x0), file:
> ../../common/fs/zfs/bptree.c, line: 293
>
> ff000bc56890 genunix:process_type+164b75 ()
> ff000bc56a20 zfs:bptree_iterate+4bf ()
> ff000bc56a90 zfs:dsl_scan_sync+17c ()
> ff000bc56b50 zfs:spa_sync+2bb ()
> ff000bc56c20 zfs:txg_sync_thread+260 ()
> ff000bc56c30 unix:thread_start+8 ()
>
> syncing file systems... done
> dumping to /dev/zvol/dsk/rpool/dump, offset 65536, content: kernel
> dumping:  0:34 100% done
> 100% done: 339495 pages dumped, dump succeeded
> rebooting...
>
> So, if anyone is interested I can provide any requested information from
> the crash dump or try your debugging suggestions.
>
> On 22/06/2016 17:45, Igor Kozhukhov wrote:
> > based on your changeset number - it is old update:
> >
> https://github.com/illumos/illumos-gate/commit/26455f9efcf9b1e44937d4d86d1ce37b006f25a9
> > 6052 decouple lzc_create() from the implementation details
> >
> > we have a lot of others changes in illumos tree and i can say - i have
> > no panic on my system with gcc48 build - i have tested by zfs tests.
> >
> > Maybe, as solution, you can try to merge to latest changes and try to
> > check it again?
> > i had panic with gcc48 build, but Matt pointed to some delphix update
> > and we have upstreamed it and i have no panics any more with full list
> > of zfs tests, what availabe on illumos tree.
> >
> > best regards,
> > -Igor
> >
> >
> >> On Jun 22, 2016, at 5:17 PM, Andriy Gapon  >> > wrote:
> >>
> >>
> >> I am not yet convinced that the problem has anything to do with
> >> miscompiled code.  I am using exactly the same optimizations and exactly
> >> the same compiler as the official FreeBSD builds.
> >>
> >> On 22/06/2016 17:03, Igor Kozhukhov wrote:
> >>> Hi Andri,
> >>>
> >>> i have DilOS with gcc-4.8,5 (+ special patches) for illumos builds.
> >>> i had some problems with zdb - found it by zfs tests.
> >>>
> >>> problem has been fixed by disable of optimization :
> >>> -fno-aggressive-loop-optimizations
> >>>
> >>> also, i have added:
> >>> -fno-ipa-sra
> >>>
> >>> but i no remember a story why i have added it ;)
> >>> probabbly it was added with another illumos component and new gcc-4.8
> >>>
> >>> As you know, illumos still is using gcc-4.4.4 and some newer compilers
> >>> can produce new issues with older code :)
> >>>
> >>> I think, you can try to play with your clang optimization flags too.
> >>> i have no experience with clang.
> >>>
> >>> best regards,
> >>> -Igor
> >>>
> >>>
>  On Jun 22, 2016, at 4:21 PM, Andriy Gapon   
>  > wrote:
> 
> 
>  I am getting the following panic using the latest FreeBSD head that is
>  synchronized with OpenZFS code as of
>  illumos/illumos-gate@26455f9efcf9b1e44937d4d86d1ce37b006f25a9.
> 
>  panic: solaris assert: ba.ba_phys->bt_bytes == 0 (0x400 == 0x0), file:
>  /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/bptree.c,
>  line: 292
>  cpuid = 1
>  KDB: stack backtrace:
>  db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
>  0xfe004db9d310
>  vpanic() at vpanic+0x182/frame 0xfe004db9d390
>  panic() at panic+0x43/frame 0xfe004db9d3f0
>  assfail3() at assfail3+0x2c/frame 0xfe004db9d410
>  bptree_iterate() at bptree_iterate+0x35e/frame 0xfe004db9d540
>  dsl_scan_sync() at dsl_scan_sync+0x24f/frame 0xfe004db9d890
>  spa_sync() at spa_sync+0x897/frame 0xfe004db9dad0
>  txg_sync_thread() at txg_sync_thread+0x383/frame 0xfe004db9dbb0
>  fork_exit() at fork_exit+0x84/frame 0xfe004db9dbf0
>  fork_trampoline() at fork_trampoline+0xe/frame 0xfe004db9dbf0
>  --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
> 
>  I have a crash dump, but unfortunately it's hard to work with it,
>  because a lot of useful information got "optimized out" by clang.
> 
>  I can reproduce the panic using a synthetic workload, but I do not
> have
>  a concise reproduction scenario.  Every time the panic happens
> bt_bytes
>  is 0x400, I haven't seen any other number there.
> 
>  Does anyone have an idea what could be causing this?
>  I can try any diagnostic code that might shed more light.
>  Thank you!
> 
>  --
>  

Re: [OpenZFS Developer] [openzfs] Possible access beyond end of string in zpool comment (#47)

[George Wilson]

LGTM

---
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/47#issuecomment-165781640___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: [OpenZFS Developer] [openzfs] ASSERT supported zio_types for file and disk vdevs (#43)

[George Wilson]

LGTM.

---
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/43#issuecomment-161622259___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: [OpenZFS Developer] [openzfs] Fix mutex leak in dmu_objset_find_dp (#44)

[George Wilson]

LGTM

---
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/44#issuecomment-161621968___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: [OpenZFS Developer] [openzfs] 6458 kmem reap thread gets blocked in reclaim callback (#38)

[George Wilson]

> @@ -1508,7 +1509,15 @@ exi_cache_trim(struct exportinfo *exi)
>* used for NFSAUTH_CACHE_TRIM seconds.
>*/
>   for (c = avl_first(tree); c != NULL; c = AVL_NEXT(tree, c)) {
> - rw_enter(>authc_lock, RW_WRITER);
> + /*
> +  * We are being called by the kmem subsystem to reclaim
> +  * memory so don't block if we can't get the lock.
> +  */
> + if (rw_tryenter(>exi_cache_lock, RW_WRITER) == 0) {

Yes, this should be the authc_lock here. This was a result of a mismerge.

---
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/38/files#r45091793___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: [OpenZFS Developer] [openzfs] 6392 zdb: Introduce -V for verbatim import (#22)

[George Wilson]

LGTM

---
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/22#issuecomment-152831278___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: [OpenZFS Developer] [openzfs] 6414 vdev_config_sync could be simpler (#31)

[George Wilson]

Looks good to me

---
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/31#issuecomment-152831553___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: [OpenZFS Developer] [openzfs] 6391 Override default SPA config location via environment (#21)

[George Wilson]

LGTM

---
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/21#issuecomment-152831323___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: [OpenZFS Developer] Review Request 250: DLPX-37488 exporting a pool while an async destroy is running can leave entries in the deferred tree

[George Wilson]

> On Oct. 7, 2015, 5:42 p.m., Justin Gibbs wrote:
> > usr/src/uts/common/fs/zfs/dsl_scan.c, lines 1421-1427
> > <https://reviews.csiden.org/r/250/diff/1/?file=17649#file17649line1421>
> >
> > Wouldn't it be better to merge this logic into dsl_scan_active() so 
> > that the policy is contained in one place?  Something like:
> > 
> > ```c
> > if (!dsl_scan_active(scn, INCLUDE_STALLED_SCANS))
> > return;
> > ```
> 
> Justin Gibbs wrote:
> Grr.  Review board posted a dup and I somehow failed to mark the right 
> lines...

I'm not convinced it would be better since proceeding with a scan when a scan 
is not running is not common policy but I don't have a strong opinion either 
way. I think if we did change the interface I would prefer to see a "force" 
flag or something like that so that it's clear that we're asking for an 
exception. Using "include" make it seem like that's the preferred behavior (at 
least to me).


- George


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.csiden.org/r/250/#review818
---


On Oct. 7, 2015, 3:02 a.m., Matthew Ahrens wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.csiden.org/r/250/
> ---
> 
> (Updated Oct. 7, 2015, 3:02 a.m.)
> 
> 
> Review request for OpenZFS Developer Mailing List and Christopher Siden.
> 
> 
> Bugs: 6292
> https://www.illumos.org/issues/6292
> 
> 
> Repository: illumos-gate
> 
> 
> Description
> ---
> 
> 6292 exporting a pool while an async destroy is running can leave entries in 
> the deferred tree
> Reviewed by: Paul Dagnelie <p...@delphix.com>
> Reviewed by: Matthew Ahrens <mahr...@delphix.com>
> 
> Original author: George Wilson
> 
> 
> Diffs
> -
> 
>   usr/src/uts/common/fs/zfs/dsl_scan.c 
> 6ba5cb6a1c831f681fba16dbc8d25fd8b59b13c9 
> 
> Diff: https://reviews.csiden.org/r/250/diff/
> 
> 
> Testing
> ---
> 
> http://jenkins.delphix.com/job/zfs-precommit/3149/
> 
> 
> Thanks,
> 
> Matthew Ahrens
> 
>

___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: [OpenZFS Developer] Review Request 258: 6319 assertion failed in zio_ddt_write: bp->blk_birth == txg

[George Wilson]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.csiden.org/r/258/#review826
---

Ship it!


Ship It!

- George Wilson


On Oct. 10, 2015, 11:15 p.m., Matthew Ahrens wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.csiden.org/r/258/
> ---
> 
> (Updated Oct. 10, 2015, 11:15 p.m.)
> 
> 
> Review request for OpenZFS Developer Mailing List and Christopher Siden.
> 
> 
> Bugs: 6319
> https://www.illumos.org/issues/6319
> 
> 
> Repository: illumos-gate
> 
> 
> Description
> ---
> 
> 6319 assertion failed in zio_ddt_write: bp->blk_birth == txg
> 
> Revert "5693 ztest fails in dbuf_verify: buf[i] == 0, due to dedup and 
> bp_override"
> 
> This reverts commit 7f7ace370074e350853da254c65688fd43ddc695.
> 
> Original author: Matthew Ahrens
> 
> 
> Diffs
> -
> 
>   usr/src/uts/common/fs/zfs/zio.c 7fa795ea8cadafa0aa7bc241b0735b2fd4ce2593 
> 
> Diff: https://reviews.csiden.org/r/258/diff/
> 
> 
> Testing
> ---
> 
> ztest; zfs test suite
> http://jenkins.delphix.com/job/zfs-precommit/3194/
> 
> 
> Thanks,
> 
> Matthew Ahrens
> 
>

___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: [OpenZFS Developer] cache device sharing

[George Wilson]

I don't think L2ARC can be shared only spare devices can be shared.

- George

On Fri, Sep 11, 2015 at 7:23 AM, Andriy Gapon 
wrote:

>
> I am curious what was the original reason to not allow sharing of cache
> devices
> (L2ARC) between pools?
> After all, the main ARC is shared for all pools.
> If more control is needed, perhaps we could have a pool property like
> shared-cache-devices (but less verbose) which would tell if data from other
> pools can be cached on this pool's cache devices.
>
> --
> Andriy Gapon
>
> ___
> developer mailing list
> developer@open-zfs.org
> http://lists.open-zfs.org/mailman/listinfo/developer
>
___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: [OpenZFS Developer] cache device sharing

[George Wilson]

The ideas was that only one pool could own it so that failover would be
possible and the cache device would follow that pool. It also would be
required for persistent l2arc.

However, it should be possible to create an independent l2arc which never
fails over but can be added to multiple pools. It would not be a trivial
implementation but seems doable.

- George

On Fri, Sep 11, 2015 at 7:40 AM, Andriy Gapon <andriy.ga...@clusterhq.com>
wrote:

> On 11/09/2015 16:36, George Wilson wrote:
> > I don't think L2ARC can be shared only spare devices can be shared.
>
> But why?
> And I do not mean multiple ownership of the cache devices. I mean that if
> a pool
> with a cache device is imported then the cache device could be used by ARC
> for
> the second level caching of data from any pool.
>
> > On Fri, Sep 11, 2015 at 7:23 AM, Andriy Gapon <
> andriy.ga...@clusterhq.com
> > <mailto:andriy.ga...@clusterhq.com>> wrote:
> >
> >
> > I am curious what was the original reason to not allow sharing of
> cache
> > devices
> > (L2ARC) between pools?
> > After all, the main ARC is shared for all pools.
> > If more control is needed, perhaps we could have a pool property like
> > shared-cache-devices (but less verbose) which would tell if data
> from other
> > pools can be cached on this pool's cache devices.
> >
> > --
> > Andriy Gapon
> >
> > ___
> > developer mailing list
> > developer@open-zfs.org <mailto:developer@open-zfs.org>
> > http://lists.open-zfs.org/mailman/listinfo/developer
> >
> >
>
>
> --
> Andriy Gapon
>
>
___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: [OpenZFS Developer] rt-rt_space is not zero. range_tree.c:153

[George Wilson]

Jorgen,

Since you're unloading the pool, there should not be any new allocations or
frees happening. This is actually prevented by the call to txg_sync_stop()
from spa_unload(). Here it should perform the final txg_wait_synced() to
clear out all the ms_freetrees and stop the sync thread.

You could add an ASSERT at the end of txg_stop_sync() to determine that the
trees are actually empty. I would also look at callers of
metaslab_free_dva() after the txg sync thread has been stopped.

Let me know what you find out.

Thanks,
George

On Thu, Aug 27, 2015 at 4:42 AM, Jorgen Lundman lund...@lundman.net wrote:


 Can you provide some details about the stack trace when you hit this
 failure. All of the ms_freetrees should be empty by the time you can
 range_tree_destroy(). So to debug this we need to understand the calling
 stack to determine why that isn't happening.


 The stack at the moment of the VERIFY0() triggered panic is:

 Aug 27 03:26:24 big-vm-mac kernel[0]: ZFS: rt-rt_space is !0 : space 6144
 from metaslab.c:1982
 Aug 27 03:26:24 big-vm-mac kernel[0]: zfs: 2103296 (512)
 Aug 27 03:26:24 big-vm-mac kernel[0]: zfs: 2125824 (1024)
 Aug 27 03:26:24 big-vm-mac kernel[0]: zfs: 2128896 (2560)
 Aug 27 03:26:24 big-vm-mac kernel[0]: zfs: 2144256 (2048)
 Aug 27 03:26:24 big-vm-mac kernel[0]: SPL: backtrace range_tree_destroy
 Aug 27 03:26:24 big-vm-mac kernel[0]: SPL: 0xff88a06738e0 :
 0xff7fa4c0471e net.lundman.zfs : _range_tree_destroy + 0x6e
 Aug 27 03:26:24 big-vm-mac kernel[0]: SPL: 0xff88a0673900 :
 0xff7fa4c00510 net.lundman.zfs : _metaslab_fini + 0x140
 Aug 27 03:26:24 big-vm-mac kernel[0]: SPL: 0xff88a0673940 :
 0xff7fa4c2b804 net.lundman.zfs : _vdev_metaslab_fini + 0x84
 Aug 27 03:26:24 big-vm-mac kernel[0]: SPL: 0xff88a0673970 :
 0xff7fa4c2b453 net.lundman.zfs : _vdev_free + 0x83
 Aug 27 03:26:24 big-vm-mac kernel[0]: SPL: 0xff88a06739a0 :
 0xff7fa4c2b425 net.lundman.zfs : _vdev_free + 0x55
 Aug 27 03:26:24 big-vm-mac kernel[0]: SPL: 0xff88a06739d0 :
 0xff7fa4c1121f net.lundman.zfs : _spa_unload + 0x11f
 Aug 27 03:26:24 big-vm-mac kernel[0]: SPL: 0xff88a06739f0 :
 0xff7fa4c1363a net.lundman.zfs : _spa_export_common + 0x29a
 Aug 27 03:26:24 big-vm-mac kernel[0]: SPL: 0xff88a0673a30 :
 0xff7fa4c13395 net.lundman.zfs : _spa_destroy + 0x25
 Aug 27 03:26:24 big-vm-mac kernel[0]: SPL: 0xff88a0673a50 :
 0xff7fa4c65b2e net.lundman.zfs : _zfs_ioc_pool_destroy + 0x1e
 Aug 27 03:26:24 big-vm-mac kernel[0]: SPL: 0xff88a0673a70 :
 0xff7fa4c622fc net.lundman.zfs : _zfsdev_ioctl + 0x66c



 ___
 developer mailing list
 developer@open-zfs.org
 http://lists.open-zfs.org/mailman/listinfo/developer

___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: [OpenZFS Developer] Review Request 229: account for ashift when choosing buffers to be written to l2arc device

[George Wilson]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.csiden.org/r/229/#review764
---

Ship it!


Ship It!

- George Wilson


On June 25, 2015, 12:34 p.m., Andriy Gapon wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.csiden.org/r/229/
 ---
 
 (Updated June 25, 2015, 12:34 p.m.)
 
 
 Review request for OpenZFS Developer Mailing List, Justin Gibbs, George 
 Wilson, Matthew Ahrens, and Saso Kiselkov.
 
 
 Bugs: 5219
 https://www.illumos.org/issues/5219
 
 
 Repository: illumos-gate
 
 
 Description
 ---
 
 **NOTE: this review request supercedes 
 [#112](https://reviews.csiden.org/r/112/).**
 
 If we don't account for that, then we might end up overwriting disk
 area of buffers that have not been evicted yet, because l2arc_evict
 operates in terms of disk addresses.
 
 The discrepancy between the write size calculation and the actual increment
 to l2ad_hand was introduced in
 commit 
 [aad02571bc59671aa3103bb070ae365f531b0b62](https://github.com/illumos/illumos-gate/commit/aad02571bc59671aa3103bb070ae365f531b0b62)
 
 The change that introduced l2ad_hand alignment was almost correct
 as the write size was accumulated as a sum of rounded buffer sizes.
 See commit 
 [e14bb3258d05c1b1077e2db7cf77088924e56919](https://github.com/illumos/illumos-gate/commit/e14bb3258d05c1b1077e2db7cf77088924e56919)
 
 Also, we now consistently use asize / a_sz for the allocated size and
 psize / p_sz for the physical size.
 The latter accounts for a possible size reduction because of the compression,
 whereas the former accounts for a possible subsequent size expansion
 because of the alignment requirements.
 
 The code still assumes that either underlying storage subsystems or
 hardware is able to do read-modify-write when an L2ARC buffer size is
 not a multiple of a disk's block size.  This is true for 4KB sector disks
 that provide 512B sector emulation, but may not be true in general.
 In other words, we currently do not have any code to make sure that
 an L2ARC buffer, whether compressed or not, which is used for physical I/O
 has a suitable size.
 
 Note that currently the cache device utilization is calculated based
 on the physical size, not the allocated size.  The same applies to l2_asize
 kstat. That is wrong, but this commit does not fix that.
 The accounting problem was introduced partially in
 commit 
 [aad02571bc59671aa3103bb070ae365f531b0b62](https://github.com/illumos/illumos-gate/commit/aad02571bc59671aa3103bb070ae365f531b0b62)
 and partially in 
 [3038a2b421b40dc5ac11cd88423696618584f85a](https://github.com/illumos/illumos-gate/commit/3038a2b421b40dc5ac11cd88423696618584f85a)
 (accounting became consistent but in favour of the wrong size).
 
 
 Diffs
 -
 
   usr/src/uts/common/fs/zfs/arc.c 09d2e1dd8fdf6648d863d4c036c16b3f7981dbf5 
 
 Diff: https://reviews.csiden.org/r/229/diff/
 
 
 Testing
 ---
 
 FreeBSD and Linux.
 
 
 Thanks,
 
 Andriy Gapon
 


___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: [OpenZFS Developer] Review Request 229: account for ashift when choosing buffers to be written to l2arc device

[George Wilson]

Andriy,

Looking now.

Thanks,
George

On Wed, Aug 26, 2015 at 10:05 AM, Andriy Gapon nore...@csiden.org wrote:

 This is an automatically generated e-mail. To reply, visit:
 https://reviews.csiden.org/r/229/

 On June 19th, 2015, 12:09 a.m. EEST, *Matthew Ahrens* wrote:

 Looks good to me.  Would like to see review from Saso and George as well.

 George, Saso, ping.


 - Andriy

 On June 25th, 2015, 3:34 p.m. EEST, Andriy Gapon wrote:
 Review request for OpenZFS Developer Mailing List, Justin Gibbs, George
 Wilson, Matthew Ahrens, and Saso Kiselkov.
 By Andriy Gapon.

 *Updated June 25, 2015, 3:34 p.m.*
 *Bugs: * 5219 https://www.illumos.org/issues/5219
 *Repository: * illumos-gate
 Description

 *NOTE: this review request supercedes #112 
 https://reviews.csiden.org/r/112/.*

 If we don't account for that, then we might end up overwriting disk
 area of buffers that have not been evicted yet, because l2arc_evict
 operates in terms of disk addresses.

 The discrepancy between the write size calculation and the actual increment
 to l2ad_hand was introduced in
 commit aad02571bc59671aa3103bb070ae365f531b0b62 
 https://github.com/illumos/illumos-gate/commit/aad02571bc59671aa3103bb070ae365f531b0b62

 The change that introduced l2ad_hand alignment was almost correct
 as the write size was accumulated as a sum of rounded buffer sizes.
 See commit e14bb3258d05c1b1077e2db7cf77088924e56919 
 https://github.com/illumos/illumos-gate/commit/e14bb3258d05c1b1077e2db7cf77088924e56919

 Also, we now consistently use asize / a_sz for the allocated size and
 psize / p_sz for the physical size.
 The latter accounts for a possible size reduction because of the compression,
 whereas the former accounts for a possible subsequent size expansion
 because of the alignment requirements.

 The code still assumes that either underlying storage subsystems or
 hardware is able to do read-modify-write when an L2ARC buffer size is
 not a multiple of a disk's block size.  This is true for 4KB sector disks
 that provide 512B sector emulation, but may not be true in general.
 In other words, we currently do not have any code to make sure that
 an L2ARC buffer, whether compressed or not, which is used for physical I/O
 has a suitable size.

 Note that currently the cache device utilization is calculated based
 on the physical size, not the allocated size.  The same applies to l2_asize
 kstat. That is wrong, but this commit does not fix that.
 The accounting problem was introduced partially in
 commit aad02571bc59671aa3103bb070ae365f531b0b62 
 https://github.com/illumos/illumos-gate/commit/aad02571bc59671aa3103bb070ae365f531b0b62
 and partially in 3038a2b421b40dc5ac11cd88423696618584f85a 
 https://github.com/illumos/illumos-gate/commit/3038a2b421b40dc5ac11cd88423696618584f85a
 (accounting became consistent but in favour of the wrong size).

 Testing

 FreeBSD and Linux.

 Diffs

- usr/src/uts/common/fs/zfs/arc.c
(09d2e1dd8fdf6648d863d4c036c16b3f7981dbf5)

 View Diff https://reviews.csiden.org/r/229/diff/

___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: [OpenZFS Developer] rt-rt_space is not zero. range_tree.c:153

[George Wilson]

Jorgen,

Can you provide some details about the stack trace when you hit this
failure. All of the ms_freetrees should be empty by the time you can
range_tree_destroy(). So to debug this we need to understand the calling
stack to determine why that isn't happening.

Thanks,
George

On Sat, Aug 22, 2015 at 8:13 PM, Jorgen Lundman lund...@lundman.net wrote:


 Hello list;

 So running the test environment, we can regularly trigger the following
 VERIFY:

 panic(cpu 0 caller 0xff7f8a065e08): VERIFY3( 0   ==  
 rt-rt_space )  failed ( 0   ==   14336
 )\n@range_tree.c:153

 I threw in some code to determine the allocator;

 ```
 range_tree_destroy(range_tree_t *rt)
 {
 if (rt-rt_space) {
 printf(ZFS: rt-rt_space is !0 : space %llu from %s:%u \n,
rt-rt_space, rt-rt_debug_allocator, rt-rt_debug_line);
 range_tree_walk(rt, dumpdump, NULL);
 }
 //VERIFY0(rt-rt_space);
 ```

 and output is similar to:

 22/08/2015 4:27:35.000 PM kernel[0]: ZFS: rt-rt_space is !0 : space 9728
 from metaslab.c:1975
 22/08/2015 4:27:35.000 PM kernel[0]: zfs: 4194304 (1536)
 22/08/2015 4:27:35.000 PM kernel[0]: zfs: 4197888 (1536)
 22/08/2015 4:27:35.000 PM kernel[0]: zfs: 4202496 (1024)
 22/08/2015 4:27:35.000 PM kernel[0]: zfs: 4208640 (512)
 22/08/2015 4:27:35.000 PM kernel[0]: zfs: 4212224 (5120)

 In fact, all of them are from metaslab.c:1975, which corresponds to

 msp-ms_freetree[t] = range_tree_create(NULL, msp,
 msp-ms_lock);

 The tests run at the time are the pool upgrade tests, so possibly
 related to the upgrade code.


 Has anyone come across this already, is it known anywhere? ZOL #3390 and
 #2947 at least mention it.

 Lund

 https://github.com/openzfsonosx/zfs/issues/361



 ___
 developer mailing list
 developer@open-zfs.org
 http://lists.open-zfs.org/mailman/listinfo/developer

___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

[AngularJS] Angular Templates not loading on file update/build in $StateProvider (WebLogic 12)

[George Wilson]

Greetings all,
I have been looking around and am having a hard time finding a definitive 
solution but we are having an issue with an AngularJS based Java 
application deployed to WebLogic. Ideally, any files which are updated 
following a new build would be updated and served to the browser. This 
appears to be the case with all of our Javascript files (We are seeing 304 
responses for assets which have not been updated) but with our partial 
templates this does not appear to be the case. Fiddler is showing those 
assets as being served from Cache. Ideally we would not have to 
completely do away with caching but rather would only update cached files 
if a change has occurred. 

Any ideas? 

-- 
You received this message because you are subscribed to the Google Groups 
AngularJS group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to angular+unsubscr...@googlegroups.com.
To post to this group, send email to angular@googlegroups.com.
Visit this group at http://groups.google.com/group/angular.
For more options, visit https://groups.google.com/d/optout.

[AngularJS] Re: Angular Templates not loading on file update/build in $StateProvider (WebLogic 12)

[George Wilson]

I added the following line in a call to myApp.config():

$httpProvider.defaults.headers.common['Cache-Control'] = 'max-age=0, 
must-revalidate';

This appears to have corrected it but a colleague is verifying.

On Thursday, August 6, 2015 at 9:34:06 AM UTC-7, George Wilson wrote:

 Greetings all,
 I have been looking around and am having a hard time finding a definitive 
 solution but we are having an issue with an AngularJS based Java 
 application deployed to WebLogic. Ideally, any files which are updated 
 following a new build would be updated and served to the browser. This 
 appears to be the case with all of our Javascript files (We are seeing 304 
 responses for assets which have not been updated) but with our partial 
 templates this does not appear to be the case. Fiddler is showing those 
 assets as being served from Cache. Ideally we would not have to 
 completely do away with caching but rather would only update cached files 
 if a change has occurred. 

 Any ideas? 


-- 
You received this message because you are subscribed to the Google Groups 
AngularJS group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to angular+unsubscr...@googlegroups.com.
To post to this group, send email to angular@googlegroups.com.
Visit this group at http://groups.google.com/group/angular.
For more options, visit https://groups.google.com/d/optout.

Re: [OpenZFS Developer] Review Request 219: 5909 ensure that shared snap names don't become too long after promotion

[George Wilson]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.csiden.org/r/219/#review731
---

Ship it!


Would be nice to have a new test for the zfs test suite.

- George Wilson


On May 6, 2015, 12:21 p.m., Andriy Gapon wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.csiden.org/r/219/
 ---
 
 (Updated May 6, 2015, 12:21 p.m.)
 
 
 Review request for OpenZFS Developer Mailing List, Christopher Siden and 
 Matthew Ahrens.
 
 
 Bugs: 5909
 https://www.illumos.org/projects/illumos-gate//issues/5909
 
 
 Repository: illumos-gate
 
 
 Description
 ---
 
 5909 ensure that shared snap names don't become too long after promotion
 
 
 Diffs
 -
 
   usr/src/uts/common/fs/zfs/dsl_dataset.c 
 ef781cbce8ec5e3dfbdd048e459491f8aeced4bf 
 
 Diff: https://reviews.csiden.org/r/219/diff/
 
 
 Testing
 ---
 
 OpenZFS/FreeBSD
 
 
 Thanks,
 
 Andriy Gapon
 


___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: [OpenZFS Developer] Review Request 217: Fix stack overflow and panic on FreeBSD.

[George Wilson]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.csiden.org/r/217/#review728
---

Ship it!


Ship It!

- George Wilson


On May 8, 2015, 3:29 a.m., Gleb Smirnoff wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.csiden.org/r/217/
 ---
 
 (Updated May 8, 2015, 3:29 a.m.)
 
 
 Review request for OpenZFS Developer Mailing List and Matthew Ahrens.
 
 
 Repository: illumos-gate
 
 
 Description
 ---
 
 Do not put zfsvfs_t on the stack. Its size of 7656 bytes consumes too much 
 stack. Allocate it
 temporarily instead.
 
 On FreeBSD the kernel stack size is 16384. Issuing 'zpool create' command 
 builds a kernel
 stack consisting of at least 36 frames, with zfs_create_fs() somewhere in the 
 middle. If
 kernel is compiled with -O0, then stack will be exhausted and kernel panics. 
 The default
 compilation option is -O2, and it doesn't panic yet. But still putting extra 
 7656 bytes
 is risky. If any of the 36 functions is modified to consume a bit more stack, 
 we will panic
 on the default kernel.
 
 
 Diffs
 -
 
   usr/src/uts/common/fs/zfs/zfs_znode.c 
 4664899d13f86f556b71aaca6949448ffd45b0eb 
 
 Diff: https://reviews.csiden.org/r/217/diff/
 
 
 Testing
 ---
 
 With the patch, FreeBSD 11-CURRENT compiled with -O0 doesn't panic on 'zpool 
 create'.
 
 
 Thanks,
 
 Gleb Smirnoff
 


___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: [OpenZFS Developer] Review Request 226: 1778 Assertion failed: rn-rn_nozpool == B_FALSE, file ../common/libzfs_import.c, line 1077, function zpool_open_func

[George Wilson]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.csiden.org/r/226/#review735
---

Ship it!


Ship It!

- George Wilson


On May 27, 2015, 1:51 p.m., Andrew Stormont wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.csiden.org/r/226/
 ---
 
 (Updated May 27, 2015, 1:51 p.m.)
 
 
 Review request for OpenZFS Developer Mailing List, George Wilson and Matthew 
 Ahrens.
 
 
 Bugs: 1778
 https://www.illumos.org/projects/illumos-gate//issues/1778
 
 
 Repository: illumos-gate
 
 
 Description
 ---
 
 Unnecessary assert in libzfs sometimes causes import to blow up.
 
 
 Diffs
 -
 
   usr/src/lib/libzfs/common/libzfs_import.c 
 19f2fbc57e53142863b745cd648eaa4fbaba282c 
 
 Diff: https://reviews.csiden.org/r/226/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Andrew Stormont
 


___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: [OpenZFS Developer] Review Request 172: 5694 traverse_prefetcher does not prefetch enough

[George Wilson]

 On March 9, 2015, 3:36 p.m., Josef 'Jeff' Sipek wrote:
  usr/src/uts/common/fs/zfs/dmu_traverse.c, line 41
  https://reviews.csiden.org/r/172/diff/1/?file=14363#file14363line41
 
  why signed?
  
  Does 50MB have some special meaning?  Or is just just a random value 
  12MB?

I left it as signed since that's what pd_bytes_fetched is.

The 50MB value is actually a conservative number and we saw good performance 
gains with our workload (mostly 8KB blocks). Remember that 12MB used before is 
only true if your stream is mostly 128KB block.


- George


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.csiden.org/r/172/#review566
---


On March 9, 2015, 2:34 p.m., Matthew Ahrens wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.csiden.org/r/172/
 ---
 
 (Updated March 9, 2015, 2:34 p.m.)
 
 
 Review request for OpenZFS Developer Mailing List and Christopher Siden.
 
 
 Bugs: 5694
 https://www.illumos.org/projects/illumos-gate//issues/5694
 
 
 Repository: illumos-gate
 
 
 Description
 ---
 
 5694 traverse_prefetcher does not prefetch enough
 Reviewed by: Matthew Ahrens mahr...@delphix.com
 Reviewed by: Alex Reece a...@delphix.com
 Reviewed by: Christopher Siden christopher.si...@delphix.com
 
 Original author: George Wilson
 
 
 Diffs
 -
 
   usr/src/uts/common/fs/zfs/dmu_traverse.c 
 14eef2e7516c9b5e717ef318321ee137dcd19c9f 
 
 Diff: https://reviews.csiden.org/r/172/diff/
 
 
 Testing
 ---
 
 ztest; zfs test suite
 
 http://jenkins/job/zfs-precommit/1821/
 
 
 Thanks,
 
 Matthew Ahrens
 


___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: [OpenZFS Developer] Review Request 167: 5669 altroot not set in zpool create when specified with -o

[George Wilson]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.csiden.org/r/167/#review543
---

Ship it!


Ship It!

- George Wilson


On Feb. 27, 2015, 7:45 p.m., Xin LI wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.csiden.org/r/167/
 ---
 
 (Updated Feb. 27, 2015, 7:45 p.m.)
 
 
 Review request for OpenZFS Developer Mailing List and Christopher Siden.
 
 
 Bugs: 5669
 https://www.illumos.org/projects/illumos-gate//issues/5669
 
 
 Repository: illumos-gate
 
 
 Description
 ---
 
 Currently, 'altroot' is only set when zpool create sees '-R'. It should be 
 also set if -o altroot= is used.
 
 Without this change, e.g zpool create -o altroot=/install_target usr would 
 fail if /usr is not empty.
 
 
 Diffs
 -
 
   usr/src/cmd/zpool/zpool_main.c 8b992efcb4e4c98581e109dc1f983b6b9d2ef104 
 
 Diff: https://reviews.csiden.org/r/167/diff/
 
 
 Testing
 ---
 
 Tested on FreeBSD.
 
 The change can be tested with zpool create -o altroot=/somedir [an existing 
 non-empty directory in /] [some device] on Illumos.
 
 
 Thanks,
 
 Xin LI
 


___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: [OpenZFS Developer] Review Request 163: 5630 stale bonus buffer in recycled dnode_t leads to data corruption

[George Wilson]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.csiden.org/r/163/#review530
---

Ship it!


Ship It!

- George Wilson


On Feb. 21, 2015, 5:34 a.m., Justin Gibbs wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.csiden.org/r/163/
 ---
 
 (Updated Feb. 21, 2015, 5:34 a.m.)
 
 
 Review request for OpenZFS Developer Mailing List and Christopher Siden.
 
 
 Bugs: 5630
 https://www.illumos.org/projects/illumos-gate//issues/5630
 
 
 Repository: illumos-gate
 
 
 Description
 ---
 
 Fix data corruption in ZFS triggered by rapidly destroying
 and creating datasets while listing datasets in another
 context.
 
 dnode_sync_free() resets the in-core dnode_t of a just
 deleted object so it represents an unallocated object.  The
 dnode_t will only be considered to represent a free dnode
 after its last hold is released.  This last hold may be
 dropped after dnode_sync_free() returns.  For data dbufs
 holding the dnode, this delay can be due to asynchronous
 evictions from the arc coincident with the dnode being
 destroyed.  For bonus buffers, this can happen if the object
 type can be destroyed even when held in another context.
 
 The observed data corruption occurred when a dsl_dir was
 destroyed for a recursive dataset destroy, while a zfs
 list operation also held this dsl_dir.  Although
 dnode_sync_free() calls dnode_evict_dbufs(), the hold on
 the dsl_dir's bonus buffer prevented it from being evicted.
 This left the bonus buffer associated with the dnode_t even
 after the last hold on the dnode was released.
 
 Some time later, this dnode_t was reused for a new dsl_dir.
 Instead of getting a new (and zero filled) bonus buffer,
 the contents from the old dsl_dir were returned.  The dsl_dir
 initialization code assumes the bonus buffer is zero filled
 and so only explicitly initializes fields that have non-zero
 initial values.  This allowed the stale data to be incorporated
 into the new dsl_dir and written to the media.
 
 Bonus buffers are only evicted via dnode_evict_dbufs() when
 there are no holds on the bonus buffer, and via
 dnode_buf_pageout/dnode_destroy().  The fix employed here
 evicts the bonus buffer for a freed dnode when the bonus
 buffer's last hold is dropped, but before the bonus buffer's
 dnode hold is released.  This ensures the dnode_t can only
 be reused after the bonus buffer eviction completes.
 
 sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c:
 sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dnode.h:
 sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:
   Add dnode_rele_and_unlock(), which avoids an extra
   lock drop and acquisition in contexts where the
   dn_mtx is already held.
 
 sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:
   In dbuf_rele_and_unlock(), immediately evict bonus
   buffers associated with freed dnodes.  The data in
   the bonus buffer is no longer relevant and this
   prevents a stale bonus buffer from being associated
   with the dnode_t should the dnode_t be reused prior
   to being destroyed.
 
 In dbuf_rele_and_unlock(), hold the bonus buffer's mutex
 until after DB_DNODE_EXIT().  This prevents another
 thread (e.g. dmu_objset_evict()-dnode_evict_dbufs()),
 from evicting the bonus buffer and invalidating the
 dbuf's dnh before or during the DB_DNODE_ENTER/EXIT()
 calls.
 
 
 Diffs
 -
 
   usr/src/uts/common/fs/zfs/dnode.c 4e376fc4d920fcc3f3429a359ed394952d116b42 
   usr/src/uts/common/fs/zfs/dbuf.c c735c9694ff905586a26d33d6ffd36cfa41ce4cb 
   usr/src/uts/common/fs/zfs/sys/dnode.h 
 406954a65323b8de8aa28b07073f588bc1805062 
 
 Diff: https://reviews.csiden.org/r/163/diff/
 
 
 Testing
 ---
 
 ZFS test suite on FreeBSD. ztest on illumos.
 
 Continuous loop on FreeBSD of recursively receiving and destroying a tree of 
 18 file systems while running management code that responds to ZFS 
 create/destroy events and enumerates file systems.  This test would fail 
 within 1 hour prior to this fix.  It has run now for 2.5 days without failure.
 
 Additional stress test:
 Thread 1:
 ```
 while true; do
 zpool create foo da1
 zpool export foo
 zpool import foo
 zpool destroy foo
 done
 ```
 
 Thread 2:
 ```
 while true; do
 zpool status foo  /dev/null
 done
 ```
 
 
 Thanks,
 
 Justin Gibbs
 


___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: [OpenZFS Developer] Review Request 155: 5531 NULL pointer dereference in dsl_prop_get_ds()

[George Wilson]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.csiden.org/r/155/#review477
---

Ship it!


Ship It!

- George Wilson


On Feb. 4, 2015, 4:59 a.m., Justin Gibbs wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.csiden.org/r/155/
 ---
 
 (Updated Feb. 4, 2015, 4:59 a.m.)
 
 
 Review request for OpenZFS Developer Mailing List.
 
 
 Bugs: 5531
 https://www.illumos.org/projects/illumos-gate//issues/5531
 
 
 Repository: illumos-gate
 
 
 Description
 ---
 
 Follow on to /r/153.
 
 uts/common/fs/zfs/dsl_dataset.c:
   Close race in dsl_dataset_try_add_ref() where the dsl_dataset_t
   still points to a valid bonus buffer, but that bonus buffer is
   held by a new dsl_dataset_t instance for this dataset.  Allow
   the add_ref operation only if the passed in dsl_dataset_t owns
   its referenced bonus buffer - the dbuf user for the bonus
   buffer is this dsl_dataset_t.
 
 
 Diffs
 -
 
   usr/src/uts/common/fs/zfs/dsl_dataset.c 
 ba9c766c65f64289b4e5e8c799a04ab3fde65e8e 
 
 Diff: https://reviews.csiden.org/r/155/diff/
 
 
 Testing
 ---
 
 ztest on illumos.
 
 
 Thanks,
 
 Justin Gibbs
 


___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: [OpenZFS Developer] Review Request 152: 5527 ::spa_space fails with mdb: couldn't find member dd_phys of type struct zfs`dsl_dir'

[George Wilson]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.csiden.org/r/152/#review461
---

Ship it!


Ship It!

- George Wilson


On Jan. 27, 2015, 5:55 a.m., Justin Gibbs wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.csiden.org/r/152/
 ---
 
 (Updated Jan. 27, 2015, 5:55 a.m.)
 
 
 Review request for OpenZFS Developer Mailing List.
 
 
 Bugs: 5527
 https://www.illumos.org/projects/illumos-gate//issues/5527
 
 
 Repository: illumos-gate
 
 
 Description
 ---
 
 5527 ::spa_space fails with mdb: couldn't find member dd_phys of type struct 
 zfs`dsl_dir'
 
 
 Diffs
 -
 
   usr/src/cmd/mdb/common/modules/zfs/zfs.c 
 225cf3dee18e110149ce6c7180fa50a3928893f9 
 
 Diff: https://reviews.csiden.org/r/152/diff/
 
 
 Testing
 ---
 
 ```
 % mdb /dev/ksyms /dev/kmem
 Loading modules: [ unix genunix specfs mac cpu.generic uppc apix scsi_vhci 
 zfs mpt sd ip hook neti sockfs arp usba stmf stmf_sbd fctl lofs random idm 
 sppp ptm nfs cpc fcip ufs logindmux ]
  ::walk spa | ::spa_space
 dd_space_towrite = 0M 0M 0M 0M
 dd_phys.dd_used_bytes = 41582M
 dd_phys.dd_compressed_bytes = 41166M
 dd_phys.dd_uncompressed_bytes = 41166M
 ms_allocmap = 0M 0M 0M 0M
 ms_freemap = 0M 0M 0M 0M
 ms_tree = 964M
 last synced avail = 9528M
 current syncing avail = 9528M
 ```
 
 
 Thanks,
 
 Justin Gibbs
 


___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: [OpenZFS Developer] Review Request 147: 5438 zfs_blkptr_verify should continue after zfs_panic_recover

[George Wilson]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.csiden.org/r/147/#review407
---

Ship it!


Ship It!

- George Wilson


On Dec. 16, 2014, 6:39 a.m., Xin LI wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.csiden.org/r/147/
 ---
 
 (Updated Dec. 16, 2014, 6:39 a.m.)
 
 
 Review request for OpenZFS Developer Mailing List and Christopher Siden.
 
 
 Repository: illumos-gate
 
 
 Description
 ---
 
 zfs_blkptr_verify should use 'continue' after zfs_panic_recover or fail more 
 gracefully.  Without this the code would do e.g. test if vd is NULL then 
 access vd-vdev_ops, which defeats the purpose of using zfs_panic_recover.
 
 
 Diffs
 -
 
   usr/src/uts/common/fs/zfs/zio.c d0a42e9af13b7012ac48d4073ebe3520b069c4d5 
 
 Diff: https://reviews.csiden.org/r/147/diff/
 
 
 Testing
 ---
 
 On FreeBSD only.
 
 
 Thanks,
 
 Xin LI
 


___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: [OpenZFS Developer] Review Request 128: 5313 Allow I/Os to be aggregated across ZIO priority classes

[George Wilson]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.csiden.org/r/128/#review408
---

Ship it!


Ship It!

- George Wilson


On Dec. 2, 2014, 12:22 p.m., Andriy Gapon wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.csiden.org/r/128/
 ---
 
 (Updated Dec. 2, 2014, 12:22 p.m.)
 
 
 Review request for OpenZFS Developer Mailing List, Justin Gibbs and George 
 Wilson.
 
 
 Bugs: 5313
 https://www.illumos.org/projects/illumos-gate//issues/5313
 
 
 Repository: illumos-gate
 
 
 Description
 ---
 
 sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/vdev_impl.h:
   Add two offset/lba based AVL trees to the vdev queue
   object.
 
 sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zio.h:
   Add a second AVL node within each ZIO so that vdev_queue.c
   can sort ZIOs by both type and priority.
 
 sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:
   Combine reads and writes, irrespecitve of their priorities
   into unified, offset sorted, trees.  Selection of the
   ZIO to issue is unchanged, but aggregation now uses the
   unified tree of the appropriate type so that aggregation
   across priority classes is possible.
 
 Original author:  Justin T. Gibbs just...@spectralogic.com
 
 
 Diffs
 -
 
   usr/src/uts/common/fs/zfs/sys/zio.h 
 f6cf259bf71a741cfee8e290be93c0600c5c9240 
   usr/src/uts/common/fs/zfs/sys/vdev_impl.h 
 6d9bcb17d00fd70d1585b9c2218086ca2239aab6 
   usr/src/uts/common/fs/zfs/vdev_queue.c 
 f47c4cd1e26dfa635dcf28e4ea3cd0cc8de4e3d1 
 
 Diff: https://reviews.csiden.org/r/128/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Andriy Gapon
 


___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: [OpenZFS Developer] Review Request 131: [5056] ZFS deadlock on db_mtx and dn_holds/Various improvements the dmu buf user API.

[George Wilson]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.csiden.org/r/131/#review406
---



usr/src/cmd/truss/Makefile.com
https://reviews.csiden.org/r/131/#comment333

Block shell comments must start with a blank # and end with one too.



usr/src/cmd/zhack/Makefile.com
https://reviews.csiden.org/r/131/#comment334

Block comment is missing starting and ending # line.



usr/src/cmd/zinject/Makefile.com
https://reviews.csiden.org/r/131/#comment335

Again.



usr/src/lib/brand/solaris10/s10_brand/Makefile.com
https://reviews.csiden.org/r/131/#comment336

Again.



usr/src/lib/libzfs/Makefile.com
https://reviews.csiden.org/r/131/#comment337

Again.



usr/src/uts/common/fs/zfs/dbuf.c
https://reviews.csiden.org/r/131/#comment338

Shouldn't you specify the priority? Are there any ramifications of not 
specifying a min and max alloc values?



usr/src/uts/common/fs/zfs/dmu_objset.c
https://reviews.csiden.org/r/131/#comment339

Was there a problem with calling dsl_dataset_is_snapshot() and being more 
explicity about the check? It seems like this could be error prone if sometime 
down the road this assumption is changed. Making it explicity seems much safer 
unless that logic is broken.



usr/src/uts/common/fs/zfs/dnode.c
https://reviews.csiden.org/r/131/#comment340

A comment here might be useful. Why don't we want special objects in the 
os_dnodes list?



usr/src/uts/common/fs/zfs/dsl_dir.c
https://reviews.csiden.org/r/131/#comment341

winner != NULL



usr/src/uts/common/fs/zfs/sa.c
https://reviews.csiden.org/r/131/#comment343

should be static.



usr/src/uts/common/fs/zfs/sa.c
https://reviews.csiden.org/r/131/#comment342

I'm not sure this is ever used, nuke it (and in sa.h too).



usr/src/uts/common/fs/zfs/spa_misc.c
https://reviews.csiden.org/r/131/#comment344

Update the comment to reflect the new behavior.



usr/src/uts/common/fs/zfs/sys/sa_impl.h
https://reviews.csiden.org/r/131/#comment345

Can you fix the whitespace while you're here?



usr/src/uts/intel/dev/Makefile
https://reviews.csiden.org/r/131/#comment346

Comment block requires start and end lines with just #.



usr/src/uts/intel/stmf_sbd/Makefile
https://reviews.csiden.org/r/131/#comment347

Comment block missing start and end lines.



usr/src/uts/sparc/stmf_sbd/Makefile
https://reviews.csiden.org/r/131/#comment348

Comment block missing start and end lines.


- George Wilson


On Dec. 15, 2014, 4:48 a.m., Justin Gibbs wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.csiden.org/r/131/
 ---
 
 (Updated Dec. 15, 2014, 4:48 a.m.)
 
 
 Review request for OpenZFS Developer Mailing List.
 
 
 Bugs: 5056
 https://www.illumos.org/projects/illumos-gate//issues/5056
 
 
 Repository: illumos-gate
 
 
 Description
 ---
 
 Various improvements the dmu buf user API.
 
 Submitted by: Justin Gibbs just...@spectralogic.com
 Submitted by: Will Andrews wi...@spectralogic.com
 Sponsored by: Spectra Logic Corporation
 
 Collect dmu buf user API support data into a new structure,
 dmu_buf_user_t.  Consumers of this interface must include a
 dmu_buf_user_t as a member of the user data structure that will be
 attached to a dmu buffer.  This reduces the size of dmu_buf_impl_t
 by two pointers.
 
 Queue dmu buf user eviction processing to a taskq.  This prevents
 FreeBSD witness(4) lock-order reversal warnings, potential deadlocks,
 and reduces stack depth.
 
 Convert objset eviction from a synchronous to an asynchronous process
 to accommodate the asynchronous invocation, via dmu buf user eviction,
 of dnode_buf_pageout().
 
 Modify existing users of the dmu buf user API to never access the dbuf to
 which their data was attached after user eviction has occurred.  Accessing
 the dbuf from the callback is no longer safe now that callbacks occur
 without the locks that used to protect them.  Enforce this in ZFS_DEBUG
 kernel builds by clearing the user's pointer to the dbuf, if any, at
 the time of eviction.  Callbacks have also been modified to clear their
 dbuf pointer so most errors are caught even on non ZFS_DEBUG kernels.
 However, this will not catch accesses from other contexts that occur
 between the time of eviction and the processing of the callback.
 
 Clarify programmer intent and improve readability by providing
 specialized functions for the common user data update actions remove
 and replace.
 
 Provide code-comment documentation for each API call.
 
 Perform runtime validation of proper API usage on ZFS_DEBUG kernels.
 
 uts/common/fs/zfs/sys/dbuf.h:
 uts/common/fs/zfs/dbuf.c:
   Add dbuf_verify_user() and call it from the dbuf user API and
   during dbuf eviction processing

Re: [discuss] assertion failed: space_map_open

[George Wilson via illumos-discuss]

On 10/25/14, 2:09 PM, Alexander Pyhalov wrote:

One other question, do you have an old 'zpool status' of the pool
you're trying to import? I'm trying to verify that the pool
configuration shown above under $import looks correct to you. Should
that pool have more than one device? Is that the correct device?


$ zpool status
  pool: data
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not 
support

the features. See zpool-features(5) for details.
  scan: none requested
config:

NAME STATE READ WRITE 
CKSUM

data ONLINE 0 0 0
  c4t6005076802808844B032d0  ONLINE 0 0 0
...
But it's a original disk, not the copy which I try to import.

The dump file is here:
ftp://fileserv.r61.net/Solaris/vmdump.2

---
System Administrator of Southern Federal University Computer Center

Is this disk, /dev/dsk/c10d1s0, still being used? If not, you could 
either unplug it or destroy the ZFS label on it by creating a new pool 
on that disk and then destroying that:


zpool create -f foo c10d1s0
zpool destroy c10d1s0

NOTE: only do the above commands if you are certain that there is 
nothing you care about on that disk.


The idea is to allow ZFS to find the correct disk to import from and 
avoid the panic. Once you have disconnected that disk or overwritten the 
label then you should be able to import your 'data' pool.


Let me know if that works.

Thanks,
George



---
illumos-discuss
Archives: https://www.listbox.com/member/archive/182180/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182180/21175430-2e6923be
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=21175430id_secret=21175430-6a77cda4
Powered by Listbox: http://www.listbox.com

Re: [discuss] assertion failed: space_map_open

[George Wilson via illumos-discuss]

This looks like it could be caused by illumos bug# 5213. We are trying 
to push a fix upstream but it's still out for review:


https://reviews.csiden.org/r/110/

Can you please provide the ::spa -v output from the core file?

Thanks,
George

On 10/24/14, 9:48 AM, Alexander Pyhalov via illumos-discuss wrote:

Hello.
I was moving OI Hipster installation from physical server to KVM VM.

I've created two lvm-based disks (the same size as original LUNs) and 
used dd to copy content from physical disks to LVM volumes.

One disk is root pool (32 GB) and another one is data pool (512 GB).

After turning VM on the root pool on VM looks good. But when I try to 
import the data pool, the system crashes.


# mdb -k unix.2 vmcore.2

Loading modules: [ unix genunix specfs mac cpu.generic uppc pcplusmp 
scsi_vhci zfs ip hook neti sockfs arp usba uhci stmf stmf_sbd s1394 
qlc fctl md lofs random idm sd mpt mpt_sas pmcs fcp emlxs ]

 ::status
debugging crash dump vmcore.2 (64-bit) from oi-build.mgmt.r61.net
operating system: 5.11 illumos-f8554bb (i86pc)
image uuid: 7a2132e7-2cf0-cb00-cfe1-b2f4c121412d
panic message:
assertion failed: space_map_open(msp-ms_sm, mos, object, 
msp-ms_start, msp-ms_size, vd-vdev_ashift, msp-ms_lock) == 0 
(0x32 == 0x0), file: ../../comm

on/fs/zfs/metaslab.c, line: 1231
dump content: kernel pages only
 ::stack
vpanic()
0xfba6542d()
metaslab_init+0x1cb(ff09126379c0, 1a, 1ed, 0)
vdev_metaslab_init+0x135(ff0918236540, 0)
vdev_load+0xd2(ff0918236540)
vdev_load+0x3b(ff0912986040)
spa_load_impl+0xa2d(ff0911aae000, fa2c3d708d545fa2, 
ff0915464a88, 3, 0, 1)

spa_load+0x14e(ff0911aae000, 3, 0, 1)
spa_tryimport+0xaa(ff0918305808)

zfs_ioc_pool_tryimport+0x51(ff09115cb000)

zfsdev_ioctl+0x4a7(b5, 5a06, 80425ec, 13, 
ff09183ae6a0, ff003e1fee68)
cdev_ioctl+0x39(b5, 5a06, 80425ec, 13, ff09183ae6a0, 
ff003e1fee68)
spec_ioctl+0x60(ff091194cc00, 5a06, 80425ec, 13, 
ff09183ae6a0, ff003e1fee68)
fop_ioctl+0x55(ff091194cc00, 5a06, 80425ec, 13, 
ff09183ae6a0, ff003e1fee68)

ioctl+0x9b(3, 5a06, 80425ec)
_sys_sysenter_post_swapgs+0x149()

Any ideas what could go wrong? I suppose that it could be just data 
corruption (but why?). Of course,  I can just recreate the data pool 
and send/recv data from physical server to VM, but the whole panic 
looks strange. I mean, why data corruption could happen?






---
illumos-discuss
Archives: https://www.listbox.com/member/archive/182180/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182180/21175430-2e6923be
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=21175430id_secret=21175430-6a77cda4
Powered by Listbox: http://www.listbox.com

Re: [OpenZFS Developer] [zfs] Recover from corrupted space map (illumos #4390)

[George Wilson]

On 7/7/14, 3:33 AM, Jan Schmidt via illumos-zfs wrote:
  
On Wed, June 25, 2014 at 16:15 (+0200), Keith Wesolowski Via Illumos-zfs wrote:

On Wed, Jun 25, 2014 at 01:47:54PM +0200, Jan Schmidt via illumos-zfs wrote:


That patch looks somewhat promising, though I have not tried it yet. How did you
decide which of the overlapping space map ranges to drop? From my understanding,
either range might be the one that's currently correct, isn't it?

It's actually worse than that, because there are a lot of different
cases, depending on whether the overlapping ranges are alloc or free,
whether there are overlapping sub-ranges within them, whether they're
partial or complete overlaps, etc.  And then there is the possibility of
subsequent ranges that partially overlap the previous bad ones.  You
didn't mention which form of corruption you're hitting or how severe it
is, so I don't know which cases might apply to you.  zdb is helpful in
getting a handle on that.

I have a different patch (George gets most of the credit, I take most of
the blame), that I used to recover spacemap corruption we had at Joyent
(albeit from a different cause, 4504).  It's intended for one-time use;
you boot it, it fixes the spacemaps by leaking ambiguous regions,
preferring to lose a little space rather than risk later overwriting of
data, and condenses them back out, then you reboot onto normal bits
again.  This covers a lot more cases; I tested many of them, but there
may yet be edge cases that aren't addressed.  I recommend building a
libzpool with this first and trying zdb with that before booting with
the zfs module.

Thanks for the explanation. We recovered our data. Using the most recent illumos
code already helped in importing the pool read-only.


This comes with absolutely no warranty of any kind and should be used
only where dumping the data somewhere else (harder than you might think,
since you can't create snapshots in read-only mode) and recreating the
pool is not an option.  It's on you to understand what it does and why
and to satisfy yourself that it will solve your problem safely before
using it.  The comments might help a little, but you're really on your
own.

See
https://github.com/wesolows/illumos-joyent/commit/dc4d7e06c8e0af213619f0aa517d819172911005

After backing up all data, we applied this patch and the non-readonly pool
import no longer crashed, printing ...

Jul  1 11:00:44 hostname genunix: [ID 882369 kern.warning] WARNING: zfs: freeing
overlapping segments: [fba5d0cee00,fba5d0cfa00) existing segment
[fba5d05e600,fba5d0cf400)
Jul  1 11:02:59 hostname genunix: [ID 882369 kern.warning] WARNING: zfs: freeing
overlapping segments: [12cf8b202400,12cf8b203000) existing segment
[12cf8b1d0c00,12cf8b202a00)

... several times (like 10 times each). After that, a full scrub of the pool
succeeded without any messages.

Do you think it is safe to continue using the repaired pool, or would you still
recommend to recreate it?
If all of the cases were  frees then you can continue using the pool and 
just realize that that space has been leaked and will never be 
allocatable. If the amount of space is significant then you may want to 
just recreate the pool.


Thanks,
George
___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: [OpenZFS Developer] zfs zio reordering in zio_vdev_io_start causing panics

[George Wilson]

On 5/7/14, 10:55 AM, Steven Hartland wrote:


- Original Message - From: George Wilson 
george.wil...@delphix.com

To: Steven Hartland kill...@multiplay.co.uk; developer@open-zfs.org
Sent: Wednesday, May 07, 2014 3:07 PM
Subject: Re: [OpenZFS Developer] zfs zio reordering in 
zio_vdev_io_start causing panics




On 5/7/14, 4:44 AM, Steven Hartland wrote:

- Original Message - From: George Wilson
I think there is probably another way we can solve this problem but 
I first want to get a better understanding of the corruption. We 
have not integrated the TRIM support upstream and I suspect that's 
the source of most of the problems. Can you confirm that with TRIM 
disabled that most of corruption you've seen does not occur? I'm 
trying to get context here since we've not seen this type of 
failure elsewhere.


In the case of TRIM if BIO delete's are disabled the free IO's don't 
get
sent to physical media and instead instantly return 
ZIO_PIPELINE_CONTINUE.

static int
vdev_geom_io_start(zio_t *zio)
{
   ...
   switch (zio-io_type) {
   case ZIO_TYPE_FREE:
   if (vdev_geom_bio_delete_disable)
   return (ZIO_PIPELINE_CONTINUE);
   }
}


For your simple corruption case could you change the code to do the 
following:


if (vdev_geom_bio_delete_disable) {
zio_interrupt(zio);
return (ZIO_PIPELINE_STOP);
}

I believe this is the way it should behave. Let me know how this work 
for you. I'm going to change the way the lower consumer work and do 
some testing also.


So your saying no path from vdev_*io_start(..) that queues an IO return
ZIO_PIPELINE_CONTINUE?

If so it looks like there's a number of locations where that needs fixing
at least in head of FreeBSD e.g.
static int
vdev_file_io_start(zio_t *zio)
{
...
   if (!vdev_readable(vd)) {
   zio-io_error = SET_ERROR(ENXIO);
   return (ZIO_PIPELINE_CONTINUE);
   }
-
static int
vdev_mirror_io_start(zio_t *zio)
{
...

   if (zio-io_type == ZIO_TYPE_READ) {
   if ((zio-io_flags  ZIO_FLAG_SCRUB)  !mm-mm_replacing) {
   ...
   return (ZIO_PIPELINE_CONTINUE);
   }
...

   return (ZIO_PIPELINE_CONTINUE);
}


Also should that be asserted to make it clear and testable e.g.
static int
zio_vdev_io_start(zio_t *zio)
{
   zio_t *sio = zio;
...
   ret = vd-vdev_ops-vdev_op_io_start(zio);
   ASSERT(ret != ZIO_PIPELINE_CONTINUE || sio == zio);

   return (ret);
}

This would also ensure the stack overflow issue shouldn't occur.

   Regards
   Steve
There are a couple of cases where it can work but I'm going to make it 
such that requires you to always return back ZIO_PIPELINE_STOP. 
Otherwise it's makes it too easy to introduce a programming error. For 
example, vdev_mirror_io_start() could return ZIO_PIPELINE_CONTINUE but 
vdev_disk_io_start() shouldn't.


I'll try to have a diff ready soon. I will be interested in your test 
results.


- George
___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: [OpenZFS Developer] zfs zio reordering in zio_vdev_io_start causing panics

[George Wilson]

On 5/7/14, 11:46 AM, Steven Hartland wrote:
- Original Message - From: George Wilson 
george.wil...@delphix.com


There are a couple of cases where it can work but I'm going to make 
it such that requires you to always return back ZIO_PIPELINE_STOP. 
Otherwise it's makes it too easy to introduce a programming error. 
For example, vdev_mirror_io_start() could return 
ZIO_PIPELINE_CONTINUE but vdev_disk_io_start() shouldn't.


I'll try to have a diff ready soon. I will be interested in your test 
results.


I'm going to give this a go but requeuing the request in the task handler
instead of processing it directly sounds like quite a waste.

Whats the objection to continuing directly with the updated zio?

   Regards
   Steve
It's a safety issue. If you need to change the zio then you effectively 
need to execute a new pipeline for that zio. If you don't want to hand 
this off to a different taskq then call zio_execute() instead of 
zio_interrupt(). For the lower layers it makes sense to call 
zio_interrupt() since we typically call the completion routines from the 
interrupt taskqs anyways.


- George


___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: [OpenZFS Developer] zfs zio reordering in zio_vdev_io_start causing panics

[George Wilson]

On 5/7/14, 12:50 PM, Steven Hartland wrote:
- Original Message - From: George Wilson 
george.wil...@delphix.com




On 5/7/14, 11:46 AM, Steven Hartland wrote:
- Original Message - From: George Wilson 
george.wil...@delphix.com


There are a couple of cases where it can work but I'm going to make 
it such that requires you to always return back ZIO_PIPELINE_STOP. 
Otherwise it's makes it too easy to introduce a programming error. 
For example, vdev_mirror_io_start() could return 
ZIO_PIPELINE_CONTINUE but vdev_disk_io_start() shouldn't.


I'll try to have a diff ready soon. I will be interested in your 
test results.


I'm going to give this a go but requeuing the request in the task 
handler

instead of processing it directly sounds like quite a waste.

Whats the objection to continuing directly with the updated zio?

It's a safety issue. If you need to change the zio then you 
effectively need to execute a new pipeline for that zio. If you don't 
want to hand this off to a different taskq then call zio_execute() 
instead of zio_interrupt(). For the lower layers it makes sense to 
call zio_interrupt() since we typically call the completion routines 
from the interrupt taskqs anyways.


Thanks for the explanation George, always good to get a better insight in
the the reasons behind things. This is the sort of thing that would be
great to capture in pipeline description comments for future readers.

Initial tests show the TRIM code still trips on the case where errors
return ZIO_PIPELINE_CONTINUE e.g.
   zio-io_error = SET_ERROR(ENOTSUP);
   return (ZIO_PIPELINE_CONTINUE);

I'm assuming in these cases we should be looking at the following 
instead?

   zio-io_error = SET_ERROR(ENOTSUP);
   zio_interrupt(zio);
   return (ZIO_PIPELINE_STOP);

And the same with:
   switch (zio-io_type) {
   case ZIO_TYPE_IOCTL:
   /* XXPOLICY */
   if (!vdev_readable(vd)) {
   zio-io_error = SET_ERROR(ENXIO);
   return (ZIO_PIPELINE_CONTINUE);
   }


   Regards
   Steve

Steve,

That's correct. I will add some more documentation about this with my 
proposed change.


Thanks,
George

___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: [OpenZFS Developer] zfs zio reordering in zio_vdev_io_start causing panics

[George Wilson]

Steven,

I think there is probably another way we can solve this problem but I 
first want to get a better understanding of the corruption. We have not 
integrated the TRIM support upstream and I suspect that's the source of 
most of the problems. Can you confirm that with TRIM disabled that most 
of corruption you've seen does not occur? I'm trying to get context here 
since we've not seen this type of failure elsewhere.


Since I'm not familiar with the TRIM implementation in FreeBSD I was 
wondering if you could explain the scenario that leads to the 
corruption. The fact that the pipeline doesn't allow the zio to change 
mid-pipeline is actually intentional so I don't think we want to make a 
change to allow this to occur. From code inspection it does look like 
the vdev_*_io_start() routines should never return 
ZIO_PIPELINE_CONTINUE. I will look at this closer but  it looks like 
there is a bug there.


Anyway, if you could give me more details to the corruption I would be 
happy to help you design a way that this can be implemented while still 
ensuring that a zio cannot change while the pipeline is still active. 
Thanks for diving into this and I will post more about the bugs that 
look to exist in the vdev_*_io_start() routines.


Thanks,
George

On 4/25/14, 8:53 PM, Steven Hartland wrote:

I've been working on adding IO priority support for TRIM back into
FreeBSD after the import of the new IO scheduling from illumos.

Based on avg's initial work and having got my head around the
requirements of the new scheduler I came up with the attached
zz-zfs-trim-priority.patch.

Most of the time this worked fine but as soon as bio_delete requests
where disabled using the follow I started getting panics:
sysctl vfs.zfs.vdev.bio_delete_disable=1

A simple dd is enough to trigger the panic e.g.
dd if=/dev/zero of=/data/random.dd bs=1m count=10240

The wide selection of panics all seemed to indicate queue corruption
with the main one erroring in vdev_queue_io_to_issue on the line:
zio = avl_first(vqc-vqc_queued_tree);

After a day of debugging and adding lots of additional validation
checks it became apparent that after removing a zio from vq_active_tree
both vq_active_tree and the associated vqc_queued_tree become corrupt.

By corrupt I mean that avl_numnodes is no longer in sync with a manual
count of the nodes using a tree walk.

In each case the vq_active_tree.avl_numnodes is one less than its actual
number of nodes and vqc_queued_tree.avl_numnodes is one greater than
its actual number of nodes.

After adding queue tracking to zio's it turned out that
vdev_queue_pending_remove was trying to remove a zio from vq_active_tree
which wasn't in that tree but was in write vqc_queued_tree tree.

As avl_remove doesn't do any validation on the node being present in
the tree it merrily tried to remove it resulting in nasty ness in both
trees.

The cause of this is in zio_vdev_io_start specifically
if ((zio = vdev_queue_io(zio)) == NULL)
return (ZIO_PIPELINE_STOP);

This can result in a different zio reaching:
return (vd-vdev_ops-vdev_op_io_start(zio));

When this happens and vdev_op_io_start returns ZIO_PIPELINE_CONTINUE
e.g. TRIM requests when bio_delete_disable=1 is set, the calling
zio_execute continues the pipeline for the zio it called
zio_vdev_io_start with, but that zio hasn't been processed and hence
isn't in the vq_active_tree but in one of vqc_queued_tree's.

Its not clear if any other paths can have their vdev_io_io_start
return ZIO_PIPELINE_CONTINUE but it definitely looks that way and
may well explain other panics I've seen in this area when for example
disks dropped.

I'm unsure if there's a more elegent fix but allowing pipeline stages
to change the processing zio by passing in a zio_t **ziop instead
of zio_t *zio as in the attached zfs-zio-queue-reorder.patch fixes the
issue.

Note: Patches are based on FreeBSD 10-RELEASE + some backports from
10-STABLE, mainly r260763: 4045 zfs write throttle  i/o scheduler,
so should apply to 10-STABLE and 11-CURRENT.

   Regards
   Steve


___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer


___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: [OpenZFS Developer] permanent error in a pool

[George Wilson]

Hmm, 'zpool scrub' should be able to clear that error. Can you tell me 
which options you used to 'zinject' and I'll try reproducing it here.


Thanks,
George

On 2/26/14 4:22 AM, Andriy Gapon wrote:

I did some experimenting and used zinject to introduce some non-correctable data
errors while reading a file. After that I destroyed a filesystem with the file
and all its snapshots, but the error was still reported in zpool status -v
output.  Maybe this is expected, although not very intuitive.  But what is
definitely unexpected to me is that neither zpool clear nor zpool scrub are able
to clear that error.  Looks like it is stuck with the pool until it is 
destroyed.

$ zpool status -v
   pool: pond
  state: ONLINE
status: One or more devices has experienced an error resulting in data
 corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
 entire pool from backup.
see: http://illumos.org/msg/ZFS-8000-8A
   scan: scrub repaired 0 in 47m with 0 errors on Wed Feb 26 11:13:30 2014
config:

 NAMESTATE READ WRITE 
CKSUM
 pondONLINE   0 0   
  0
   mirror-0  ONLINE   0 0   
  0
 gptid/fcf3558b-493b-11de-a8b9-001cc08221ff  ONLINE   0 0   
  0
 gptid/48782c6e-8fbd-11de-b3e1-00241d20d446  ONLINE   0 0   
  0

errors: Permanent errors have been detected in the following files:

 0x1b2d:0xb



___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer


___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: Maven just won't die! -- WTF is Takari?

[George Wilson]

- Shell: long-lived CLI process for Maven

A Maven shell? Interesting idea, how does that work? It seems like the idea
behind maven is set it and forget it.


On Wed, Jan 15, 2014 at 5:08 PM, Dan Tran dant...@gmail.com wrote:

 is it a new baby from Sonatype?

 -D


 On Wed, Jan 15, 2014 at 10:26 AM, Jason van Zyl ja...@tesla.io wrote:

  Hi,
 
  As some may know, a lot of work has been done at tesla.io on various
  advanced features in Maven but, unfortunately, not enough work for these
  features to see the light of day. It wouldn't be surprising if Maven
 users
  have no idea what these features are because I've not done a great job at
  communicating about the work. Some of the work is mine, some of other
 major
  contributors, and more recently from key customers.
 
  To make a long story short there's a lot of cool stuff to talk about, and
  the work as a new venue at takari.io! I'll be giving a webinar next week
  and here are some of the features I'd like to talk about:
 
  - Polyglot support: Ruby, Groovy, and Scala DSLs. These have all been
  actively worked on in the recent past, especially the Ruby and Scala
 DSLs.
  - Full incremental support: the complete Maven lifecycle including an
  incremental command line compiler based on JDT, all with m2e integration
  - Aggressive parallelization: a new parallelization mode that also
  optimizes scheduling based on critical path analysis
  - Generations: a new form of continuous delivery for Maven -- Smart delta
  protocol and no more SNAPHOTs!
  - Shell: long-lived CLI process for Maven
 
  Much of this work is functional, and the new parallelization mode and
  generations support are actively being used in production. We are still
  iterating on these specific features but they show a lot of promise.
 Where
  all of this code eventually lands is a question for the Maven development
  community. All of this work was developed outside of Apache, and how easy
  it is to integrate back into the Maven project remains to be seen. At the
  very least there is a lot of very interesting work and I wanted to start
  the dialog because Maven just isn't going to die :-)
 
  So please join us for a webinar, Tuesday, January 21 at 11:30AM EST (UTC
  -5 hours) to learn more about what we're working on and what we're trying
  to accomplish.
 
  All registrants will receive access to the recording, so if you can’t
 make
  it — you won’t have to miss out.
 
  Register here: http://goo.gl/vqSvL7
 
  Hope you can make it,
 
  Jason van Zyl
  --
  Jason van Zyl
  Founder,  Apache Maven
  http://twitter.com/jvanzyl
  -
 
 
 
 
 
 
 
 

Re: Maven won't die!

[George Wilson]

Sounds interesting. I signed up for the webinar and look forward to hearing
more.


On Thu, Jan 16, 2014 at 10:53 AM, Jason van Zyl ja...@takari.io wrote:

 Sorry I'm responding out of the original thread, but realized I had the
 wrong account subscribed to the user list. So I'll respond to the few
 questions here in one thread.

 @Dan: Takari is my new company and is not related to Sonatype. I want to
 focus all my time on the Maven ecosystem.

 @George: Has some of the characteristics of a workspace in the IDE but on
 the command line. I'll try to write something up.

 @Jeff: Yes, i think the generations support is path toward real CD for
 Maven. Snapshots and the release plugin are a PITA.

 Thanks,

 Jason

 --
 Jason van Zyl
 Founder,  Apache Maven
 http://twitter.com/jvanzyl
 -

Re: [OpenZFS Developer] l2arc_write_buffers scanning efficiency

[George Wilson]

LGTM.

- George

On 1/14/14 12:57 PM, Saso Kiselkov wrote:

On 1/14/14, 11:51 AM, Andriy Gapon wrote:

Recently I noticed that on one of our systems l2arc_feed_thread was consuming
non insignificant amount of CPU time.  I looked at some stats (FreeBSD has local
changes that add a few kstats for L2ARC) and it seems that
the thread was busy scanning and re-scanning buffers that are already in L2 ARC.
Buffers were being skipped because l2arc_write_eligible was returning false
because of ab-b_l2hdr != NULL.  As far as I can see the thread was doing a pass
per second and it typically scanned about 2 million buffers per pass.  Typically
walking over a buffer list was aborted due to passed_sz  headroom.

The system in question has a quite large ARC with maximum size of 60GB.  26GB
were actually in use and it seems that most of the buffers were rather small,
hash_elements == 3634055.

Perhaps, there could be some optimization to avoid pointless walking over
millions of buffers in situations like this?

P.S. Because of another local FreeBSD change the feed thread was scanning about
16 times more buffers than it would on illumos, so the issue was more prominent
with the thread consuming about 40% of a core.

That leak should have been fixed... yep found the webrev:
http://cr.illumos.org/~webrev/skiselkov/3995/
Unfortunately, it appears I dropped the ball on this one and forgot to
submit an RTI for it.

Could people please give me a quick ok on the above webrev again? I've
updated it in place. All it really does is move the
l2arc_release_cdata_buf step in front of the mutex_trylock - since the
b_tmp_cdata pointer is thread-local, we don't need to grab locks it to
manipulate it. The rest of the changes are just renaming 'abl2' to
'l2hdr' to be consistent across all of arc.c.

Cheers,


___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: [OpenZFS Developer] Metaslab allocation question

[George Wilson]

Ilya,

I think there is a bigger problem here that your vdev_mirror() change 
would cover up. The problems is that we should never have a bp with a 
dva that is out of range. So we need to understand how that is occurring.


As for you question about metaslab_allocate_dva(), I'm assuming you're 
referring to this code path:


} else if (d != 0) {
vd = vdev_lookup_top(spa, DVA_GET_VDEV(dva[d - 1]));
mg = vd-vdev_mg-mg_next;

This vdev can't be NULL since the configuration should not have changed 
while we're in the middle of doing allocations.


Even for gang blocks we should never have a NULL vdev. I'm assuming 
you're referring to this logic:


if (vd != NULL) {
mg = vd-vdev_mg;

if (flags  METASLAB_HINTBP_AVOID 
mg-mg_next != NULL)
mg = mg-mg_next;
} else {
mg = mc-mc_rotor;
}

This is meant to handle device logs which can be removed since the ZIL 
may have a reference to an old log block on a device that no longer exists.


I think we need to find the root cause of where this out of range dva is 
coming from. If you have old core files that you could share please 
point me at them.


Thanks,
George

On 12/18/13 1:33 PM, Ilya Usvyatsky wrote:
I am looking at a rare but nasty case of corruption in which a block 
pointer has one of ditto dva's out of range.

This causes vdev_lookup_top() to return NULL for a vdev pointer.
Going through the callers of vdev_lookup_top(), I noticed that in some 
cases NULL vdev are handled properly, while in others it is not.
In particular, in case of an import, ditto blocks are handled by a 
mirror vdev code path that does not have proper handling for NULL and 
would panic the system.


My question, though, is not about the mirror (for which I have a fix 
that I will upstream eventually),but about metaslab allocator.
In particular, I am looking at metaslab_allocate() and 
metaslab_allocate_dva().
In the latter function NULL vdev is properly handled in the case of a 
gang block (since a hint may be stale and the device may be gone).
However, that very same function does not handle NULL vdev in case of 
a regular block pointer with ditto blocks.
The dilemma I am facing here is that I can either just use rotor 
(i.e.. mimic gang block behavior), or return an error immediately.
In the latter case, the caller, metaslab_allocate() would handle it 
properly.
I am inclined to go with the second option, but I would greatly 
appreciate an insight on this from someone who is familiar with the 
internal logic and the theory behind metaslab allocator.


Thank you very much,
Ilya.



___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer


___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: [OpenZFS Developer] zio parent and zio children relation.

[George Wilson]

On 12/17/13 5:21 AM, Gaurav Mahajan wrote:

Hi all,

I am trying to understand relations of root ZIO and children ZIO.

This is what I have understood.. Please correct me if I'm wrong.

Usually whenever we want to do a series of IO operations like in sync 
thread.
We create a root ZIO with zio_root(). Now this root ZIO becomes parent 
for every ZIO that we create while syncing the async data to disk (in 
dbuf_sync_leaf).


All the child ZIO are issued using zio_nowait()
After issuing all the children ZIO at the end we call zio_wait() on 
root ZIO.


So the question that comes in my mind is that after zio_wait for root 
ZIO is over, are we guaranteed that all the children ZIO are complete.?


Yes, the root zio cannot complete until all its children have completed.

- George


complete in sense like block allocation and data write are done and 
io_done callback are complete.


I may be wrong with my understanding. Please correct me.

Thanks !!!
Gaurav.



___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer


___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: [OpenZFS Developer] priority for TRIM zio

[George Wilson]

Andriy,

The fix looks good but I have a couple of questions:

1. Are you sure you want the TRIM priority to be lower than SCRUB?
2. You have zfs_vdev_trim_max_active set to 1 with an /* XXX */ next to 
it. Does this mean you haven't settled on the final value for active max?


Thanks,
George

On 11/30/13 10:13 AM, Andriy Gapon wrote:

[resending with CC fixed]

Matt,

as you most likely know, ZFS/FreeBSD already has ability to use TRIM command
with disks that support it.
As the command can be quite slow and it is a maintenance kind of command, it
makes sense to assign a very low priority to it.

I've come up with the following change that introduces a new priority for the
TRIM zios:
http://people.freebsd.org/~avg/zfs-trim-priority.diff
To be honest, this change actually attempts to restore what we already had in
FreeBSD before I merged your write throttle  i/o scheduler performance work.

Could you please review the change?
I am not sure if I correctly translated my intent to the min_active and
max_active values.  I will greatly appreciate your help with these.

Thank you!


___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer


___
developer mailing list
developer@open-zfs.org
http://lists.open-zfs.org/mailman/listinfo/developer

Re: maven-failsafe-plugin: what is it actually intended for?

[George Wilson]

FWIW, I tend to think less about the particular definitions and
semantics of testing types and think more about the needs associated
with tests. For tests which depend on deployed resources, etc... I use
failsafe. For tests which can be handled locally whether resources are
mocked or called externally (but for which said execution does not
require deployment- IE sending a request and handling a response)- I
use surefire.

Back to the testing definitions. In my mind, as others have stated
unit tests are meant to test individual units and features within a
piece of code. Personally, I am willing to include in that features
which depend on multiple modules within a project. To me it is more
philosophical than anything, by the strictest definition I would
imaging Unit tests are meant to test functionality within one class,
such as a method. For instance, say you have a method in a shopping
cart class which creates a new account with data XYZ. The customer
object is created and in that creation generates a predictable user ID
(based on name and birthday for instance).

Integration tests, to me, accomplish two things, they test that these
individual units play nice with each other (particularly across
projects), and they test that a particular compiled project
integrates well into its deployed environment. This includes playing
nice with deployed resources, the server environment, etc

I agree with the statement that time is not a defining parameter in
the discussion of unit vs. integration test. If you are testing
something that just plain takes a lot of time to do (such as a complex
calculation) then it seems that could very plausibly be considered a
unit test- even if it takes an hour (I have written single methods
with that behavior before- computing Fourier series coefficients for
instance). I think the defining quality is what exactly is being
tested and what phase in the project lifecycle it is mean to occur in.

Again though, for purposes of setting up maven, I think it terms of
which plugin do I actually need. Failsafe is much easier to use for
testing deployed functionality, surefire for unit or light
integration functionality (IE between locally testable compilation
units).

I hope that long winded answer helps :s

On Wed, Nov 13, 2013 at 10:05 AM, Ron Wheeler
rwhee...@artifact-software.com wrote:
 You will probably get better answers.


 On 13/11/2013 12:09 PM, James Green wrote:

 So where should one place a test that intends on exercising code against
 something real? We have bits here that involve http calls that pre-date
 soap and we therefore have no mock.


 It depends on what you are testing.
 If it is a webapp, you may find that you have to put the script in a
 spreadsheet that you give to a human.
 If you have a keystoke emulator, then the script will go in the emulator.

 Integration tests in general would be separate projects.

 Mocks have nothing to do with soap so I am not sure what age has to do with
 testing.



 A repeat of the second question from my original post: does the integrate
 test execute against the artefact produced or against the original source
 code?

 You always want to test against the artifact produced in an integration
 test.
 Unit test is usually in the project that produces the artifact and has to
 pass before the artifact is produced.


 Ron



 On 13 November 2013 15:59, Stephen Connolly
 stephen.alan.conno...@gmail.com

 wrote:
 On 13 November 2013 15:20, James Green james.mk.gr...@gmail.com wrote:

 I love the FAQ entry that states that it is intended for running
 integration tests.

 The next entry should read: What do you call an integration test?

 Any test that takes more than 1 second to run is *not* a unit test.

 Most tests that take more than 50ms to run are *not* unit tests... but
 there can be some exceptions

 If a unit test needs to call out to other systems, it will typically use
 a
 mock.

 If your test is actually calling out to other systems (which could be
 code
 from a dependency, etc - i.e. not just a TCP socket, could be a call
 within
 JVM) then it is testing the integration of those two parts... therefore
 it
 is not a unit test.

 There is no hard and fast rule as to where the transition occurs... but
 we
 know that tests who's execution time is greater than 1 second are not
 unit
 tests... and hence are integration tests...

 HTH

 I've asked around and no-one comes up with a consistent answer. I guess

 it

 depends on what is executing the integration test. In this case maven is
 invoking someone after the packaging phase so should I expect to run

 tests

 against the packaged binary artefact? Is that the purpose here?

 Thanks,

 James



 --
 Ron Wheeler
 President
 Artifact Software Inc
 email: rwhee...@artifact-software.com
 skype: ronaldmwheeler
 phone: 866-970-2435, ext 102


 -
 To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
 For additional 

Re: maven-failsafe-plugin: what is it actually intended for?

[George Wilson]

James, see my response. It sounds to me like you may need to use
Failsafe- particularly if you need to deploy your artifact to your
application server and access system resources. If you are doing
simple HTTP requests and want to set it up as a unit test- my advise
if capture a response and save it- then create a mock out of that. If
you need to run a test which depends on the current state of the
external resource- it may be a better idea to think of it in terms of
integration. Again though, i completely depends on what you are
testing for and what effects you expect from the change in state. (A
test which grabs a webpage and scrapes off the title and manipulates
it somehow- can easily be treated in terms of a unit test if you set
it up properly though some might feel that is an abuse of the
idea.)

On Wed, Nov 13, 2013 at 9:09 AM, James Green james.mk.gr...@gmail.com wrote:
 So where should one place a test that intends on exercising code against
 something real? We have bits here that involve http calls that pre-date
 soap and we therefore have no mock.

 A repeat of the second question from my original post: does the integrate
 test execute against the artefact produced or against the original source
 code?


 On 13 November 2013 15:59, Stephen Connolly stephen.alan.conno...@gmail.com
 wrote:

 On 13 November 2013 15:20, James Green james.mk.gr...@gmail.com wrote:

  I love the FAQ entry that states that it is intended for running
  integration tests.
 
  The next entry should read: What do you call an integration test?
 

 Any test that takes more than 1 second to run is *not* a unit test.

 Most tests that take more than 50ms to run are *not* unit tests... but
 there can be some exceptions

 If a unit test needs to call out to other systems, it will typically use a
 mock.

 If your test is actually calling out to other systems (which could be code
 from a dependency, etc - i.e. not just a TCP socket, could be a call within
 JVM) then it is testing the integration of those two parts... therefore it
 is not a unit test.

 There is no hard and fast rule as to where the transition occurs... but we
 know that tests who's execution time is greater than 1 second are not unit
 tests... and hence are integration tests...

 HTH

 
  I've asked around and no-one comes up with a consistent answer. I guess
 it
  depends on what is executing the integration test. In this case maven is
  invoking someone after the packaging phase so should I expect to run
 tests
  against the packaged binary artefact? Is that the purpose here?
 
  Thanks,
 
  James
 


-
To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
For additional commands, e-mail: users-h...@maven.apache.org

Re: Strange issue with generated jars

[George Wilson]

Unfortunately, my company's security policies do not allow for the
downloading and building of external projects without approval from IT
and security so I cannot really test your code (not without going to a
committee, etc...). Any chance you can post the errors you are
getting? Is this a JVM heap issue perhaps? Have you tried increasing
your Java memory? Just a thought since you mentioned that javac is
what seems to die here.

On Wed, Nov 13, 2013 at 5:57 AM,  org.apache.maven.u...@io7m.com wrote:
 On Tue, 12 Nov 2013 13:02:46 +
 org.apache.maven.u...@io7m.com wrote:

 Hello.

 I've run into a strange but easily reproduced problem with the jar files
 generated by Maven. Essentially, if I generate a jar file containing a
 large number of files (= 65536, in practice), then javac becomes unable
 to resolve classes from that jar file. This only occurs with jars produced
 by the Maven jar plugin, and only when the number of files is large (as
 demonstrated below).

 An example build, using the maven exec plugin to generate a large
 number (65525) of files:

 http://waste.io7m.com/2013/11/12/jarbug.zip

 Can anyone else reproduce this problem? This would seem to indicate a serious
 bug somewhere. I'm trying to eliminate Maven as a cause.

 M

 -
 To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
 For additional commands, e-mail: users-h...@maven.apache.org


-
To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
For additional commands, e-mail: users-h...@maven.apache.org

Re: Strange issue with generated jars

[George Wilson]

A crippling restriction or not, it is company policy which I do not
have any authority over. Its one thing to try a snippet from stack
overflow but the restriction is in regards to building a larger
project like a maven build. The concern is not being able to evaluate
what the code or individual scripts do.
I do not have time right this minute to set up a VM instance but
perhaps could later on workload permitting. It seems that since you
have run this and generated errors, it would be easier as a first step
diagnostic to post your error messages- output cleaned if need be of
course. FWIW, I am not sure that it is really necessary to ask people
to go out of their way to build VM systems, when the error output
might solve the problem immediately.

On Wed, Nov 13, 2013 at 11:37 AM, Curtis Rueden ctrue...@wisc.edu wrote:
 Hi George,

 That's a new one on me. Can you build in an isolated VM? On a personal
 machine while at work? Not being able to try out code from the Internet
 seems like a crippling restriction to me.

 -Curtis
  On Nov 13, 2013 1:34 PM, George Wilson rmws...@gmail.com wrote:

 Unfortunately, my company's security policies do not allow for the
 downloading and building of external projects without approval from IT
 and security so I cannot really test your code (not without going to a
 committee, etc...). Any chance you can post the errors you are
 getting? Is this a JVM heap issue perhaps? Have you tried increasing
 your Java memory? Just a thought since you mentioned that javac is
 what seems to die here.

 On Wed, Nov 13, 2013 at 5:57 AM,  org.apache.maven.u...@io7m.com wrote:
  On Tue, 12 Nov 2013 13:02:46 +
  org.apache.maven.u...@io7m.com wrote:
 
  Hello.
 
  I've run into a strange but easily reproduced problem with the jar files
  generated by Maven. Essentially, if I generate a jar file containing a
  large number of files (= 65536, in practice), then javac becomes unable
  to resolve classes from that jar file. This only occurs with jars
 produced
  by the Maven jar plugin, and only when the number of files is large (as
  demonstrated below).
 
  An example build, using the maven exec plugin to generate a large
  number (65525) of files:
 
  http://waste.io7m.com/2013/11/12/jarbug.zip
 
  Can anyone else reproduce this problem? This would seem to indicate a
 serious
  bug somewhere. I'm trying to eliminate Maven as a cause.
 
  M
 
  -
  To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
  For additional commands, e-mail: users-h...@maven.apache.org
 

 -
 To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
 For additional commands, e-mail: users-h...@maven.apache.org



-
To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
For additional commands, e-mail: users-h...@maven.apache.org

Re: Strange issue with generated jars

[George Wilson]

Hi Curtis,
I apologize, I had not recognized that you were not the OP.

If you just look at it, you can see that it doesn't do anything bad.
- You are correct, I did not bother to download it as it was presented
as being a project with several thousand classes. Perhaps in the
future I should download such things.



On Wed, Nov 13, 2013 at 2:32 PM, Curtis Rueden ctrue...@wisc.edu wrote:
 Hi,

 org.apache.maven.u...@io7m.com wrote:
 Can anyone else reproduce this problem?

 OK, I ran the example (mvn clean package) and the project builds
 successfully on my system:

 $ mvn -v
 Apache Maven 3.1.1 (0728685237757ffbf44136acec0402957f723d9a; 2013-09-17
 10:22:22-0500)
 Maven home: /usr/local/Cellar/maven/3.1.1/libexec
 Java version: 1.7.0_45, vendor: Oracle Corporation
 Java home:
 /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/jre
 Default locale: en_US, platform encoding: UTF-8
 OS name: mac os x, version: 10.9, arch: x86_64, family: mac

 $ jar tf code/target/code-0.1.0.jar |wc
   11  11 270

 $ jar tf resources/target/resources-0.1.0.jar |wc
65547   65547 2151861

 George Wilson wrote:
 the restriction is in regards to building a larger project like a
 maven build.

 Well, it's 38 lines of code:

 $ find . -name '*.java' | xargs wc
   10  18 157 ./code/src/main/java/com/io7m/jarbug/Main.java
   28  74 694
 ./resources/src/main/java/com/io7m/jarbug/MakeSources.java
   38  92 851 total

 If you just look at it, you can see that it doesn't do anything bad.

 I am not sure that it is really necessary to ask people to go out of
 their way to build VM systems

 I am not the OP, and was not asking you to do anything. I was merely
 inquiring whether you could perhaps avoid your company's bureaucracy via
 some technical means.

 It seems that since you have run this and generated errors, it would
 be easier as a first step diagnostic to post your error messages-
 output cleaned if need be of course.

 I agree that it wouldn't hurt for the OP to post his error messages, too.
 But that said, IMHO, posting a complete example project demonstrating the
 issue is about the *best* thing you can do.

 Regards,
 Curtis


 On Wed, Nov 13, 2013 at 4:16 PM, George Wilson rmws...@gmail.com wrote:

 A crippling restriction or not, it is company policy which I do not
 have any authority over. Its one thing to try a snippet from stack
 overflow but the restriction is in regards to building a larger
 project like a maven build. The concern is not being able to evaluate
 what the code or individual scripts do.
 I do not have time right this minute to set up a VM instance but
 perhaps could later on workload permitting. It seems that since you
 have run this and generated errors, it would be easier as a first step
 diagnostic to post your error messages- output cleaned if need be of
 course. FWIW, I am not sure that it is really necessary to ask people
 to go out of their way to build VM systems, when the error output
 might solve the problem immediately.

 On Wed, Nov 13, 2013 at 11:37 AM, Curtis Rueden ctrue...@wisc.edu wrote:
  Hi George,
 
  That's a new one on me. Can you build in an isolated VM? On a personal
  machine while at work? Not being able to try out code from the Internet
  seems like a crippling restriction to me.
 
  -Curtis
   On Nov 13, 2013 1:34 PM, George Wilson rmws...@gmail.com wrote:
 
  Unfortunately, my company's security policies do not allow for the
  downloading and building of external projects without approval from IT
  and security so I cannot really test your code (not without going to a
  committee, etc...). Any chance you can post the errors you are
  getting? Is this a JVM heap issue perhaps? Have you tried increasing
  your Java memory? Just a thought since you mentioned that javac is
  what seems to die here.
 
  On Wed, Nov 13, 2013 at 5:57 AM,  org.apache.maven.u...@io7m.com
 wrote:
   On Tue, 12 Nov 2013 13:02:46 +
   org.apache.maven.u...@io7m.com wrote:
  
   Hello.
  
   I've run into a strange but easily reproduced problem with the jar
 files
   generated by Maven. Essentially, if I generate a jar file containing
 a
   large number of files (= 65536, in practice), then javac becomes
 unable
   to resolve classes from that jar file. This only occurs with jars
  produced
   by the Maven jar plugin, and only when the number of files is large
 (as
   demonstrated below).
  
   An example build, using the maven exec plugin to generate a large
   number (65525) of files:
  
   http://waste.io7m.com/2013/11/12/jarbug.zip
  
   Can anyone else reproduce this problem? This would seem to indicate a
  serious
   bug somewhere. I'm trying to eliminate Maven as a cause.
  
   M
  
   -
   To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
   For additional commands, e-mail: users-h...@maven.apache.org