Re: [Discuss] NOTICE file for transitive "NOTICE"s

[Richard Hillegas]

Thanks, Sean!

Sean Owen <so...@cloudera.com> wrote on 09/25/2015 06:35:46 AM:

> From: Sean Owen <so...@cloudera.com>
> To: Reynold Xin <r...@databricks.com>, Richard Hillegas/San
> Francisco/IBM@IBMUS
> Cc: "dev@spark.apache.org" <dev@spark.apache.org>
> Date: 09/25/2015 07:21 PM
> Subject: Re: [Discuss] NOTICE file for transitive "NOTICE"s
>
> Work underway at ...
>
> https://issues.apache.org/jira/browse/SPARK-10833
> https://github.com/apache/spark/pull/8919
>
>
>
> On Fri, Sep 25, 2015 at 8:54 AM, Sean Owen <so...@cloudera.com> wrote:
> > Update: I *think* the conclusion was indeed that nothing needs to
> > happen with NOTICE.
> > However, along the way in
> > https://issues.apache.org/jira/browse/LEGAL-226 it emerged that the
> > BSD/MIT licenses should be inlined into LICENSE (or copied in the
> > distro somewhere). I can get on that -- just some grunt work to copy
> > and paste it all.
> >
> > On Thu, Sep 24, 2015 at 6:55 PM, Reynold Xin <r...@databricks.com>
wrote:
> >> Richard,
> >>
> >> Thanks for bringing this up and this is a great point. Let's start
another
> >> thread for it so we don't hijack the release thread.
> >>
> >>
> >>
> >> On Thu, Sep 24, 2015 at 10:51 AM, Sean Owen <so...@cloudera.com>
wrote:
> >>>
> >>> On Thu, Sep 24, 2015 at 6:45 PM, Richard Hillegas
<rhil...@us.ibm.com>
> >>> wrote:
> >>> > Under your guidance, I would be happy to help compile a NOTICE file
> >>> > which
> >>> > follows the pattern used by Derby and the JDK. This effort might
proceed
> >>> > in
> >>> > parallel with vetting 1.5.1 and could be targeted at a later
release
> >>> > vehicle. I don't think that the ASF's exposure is greatly increased
by
> >>> > one
> >>> > more release which follows the old pattern.
> >>>
> >>> I'd prefer to use the ASF's preferred pattern, no? That's what we've
> >>> been trying to do and seems like we're even required to do so, not
> >>> follow a different convention. There is some specific guidance there
> >>> about what to add, and not add, to these files. Specifically, because
> >>> the AL2 requires downstream projects to embed the contents of NOTICE,
> >>> the guidance is to only include elements in NOTICE that must appear
> >>> there.
> >>>
> >>> Put it this way -- what would you like to change specifically? (you
> >>> can start another thread for that)
> >>>
> >>> >> My assessment (just looked before I saw Sean's email) is the same
as
> >>> >> his. The NOTICE file embeds other projects' licenses.
> >>> >
> >>> > This may be where our perspectives diverge. I did not find those
> >>> > licenses
> >>> > embedded in the NOTICE file. As I see it, the licenses are cited
but not
> >>> > included.
> >>>
> >>> Pretty sure that was meant to say that NOTICE embeds other projects'
> >>> "notices", not licenses. And those notices can have all kinds of
> >>> stuff, including licenses.
> >>>
> >>> -
> >>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> >>> For additional commands, e-mail: dev-h...@spark.apache.org
> >>>
> >>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> For additional commands, e-mail: dev-h...@spark.apache.org
>

Re: [Discuss] NOTICE file for transitive "NOTICE"s

[Steve Loughran]

> On 24 Sep 2015, at 21:11, Sean Owen  wrote:
> 
> Yes, but the ASF's reading seems to be clear:
> http://www.apache.org/dev/licensing-howto.html#permissive-deps
> "In LICENSE, add a pointer to the dependency's license within the
> source tree and a short note summarizing its licensing:"
> 
> I'd be concerned if you get a different interpretation from the ASF. I
> suppose it's OK to ask the question again, but for the moment I don't
> see a reason to believe there's a problem.

Having looked at the notice, I actually see a lot more thorough that most ASF 
projects.

in contrast, here is the hadoop one: 

---
This product includes software developed by The Apache Software
Foundation (http://www.apache.org/).
---

regarding the spark one, I don't see that you need to refer to transitive 
dependencies for the non-binary distros, and, for any binaries, to bother 
listing the licensing of all the ASF dependencies. Things pulled in from 
elsewhere & pasted in, that's slightly more complex. I've just been dealing 
with the issue of taking an openstack-applied patch to the hadoop swift object 
store code -and, because the licenses are compatible, we're just going to stick 
it in as-is.

Uber-JARs, such as spark.jar, do contain lots of classes from everywhere. I 
don't know the status of them. You could probably get maven to work out the 
licensing if all the dependencies declare their license.

On that topic, note that marcelo's proposal to break up that jar and add 
lib/*.jar to the CP would allow codahale's ganglia support to come in just by 
dropping in the relevant LGPL JAR, avoiding the need to build a custom spark 
JAR tainted by the transitive dependency.

-Steve

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: [Discuss] NOTICE file for transitive "NOTICE"s

[Sean Owen]

Update: I *think* the conclusion was indeed that nothing needs to
happen with NOTICE.
However, along the way in
https://issues.apache.org/jira/browse/LEGAL-226 it emerged that the
BSD/MIT licenses should be inlined into LICENSE (or copied in the
distro somewhere). I can get on that -- just some grunt work to copy
and paste it all.

On Thu, Sep 24, 2015 at 6:55 PM, Reynold Xin  wrote:
> Richard,
>
> Thanks for bringing this up and this is a great point. Let's start another
> thread for it so we don't hijack the release thread.
>
>
>
> On Thu, Sep 24, 2015 at 10:51 AM, Sean Owen  wrote:
>>
>> On Thu, Sep 24, 2015 at 6:45 PM, Richard Hillegas 
>> wrote:
>> > Under your guidance, I would be happy to help compile a NOTICE file
>> > which
>> > follows the pattern used by Derby and the JDK. This effort might proceed
>> > in
>> > parallel with vetting 1.5.1 and could be targeted at a later release
>> > vehicle. I don't think that the ASF's exposure is greatly increased by
>> > one
>> > more release which follows the old pattern.
>>
>> I'd prefer to use the ASF's preferred pattern, no? That's what we've
>> been trying to do and seems like we're even required to do so, not
>> follow a different convention. There is some specific guidance there
>> about what to add, and not add, to these files. Specifically, because
>> the AL2 requires downstream projects to embed the contents of NOTICE,
>> the guidance is to only include elements in NOTICE that must appear
>> there.
>>
>> Put it this way -- what would you like to change specifically? (you
>> can start another thread for that)
>>
>> >> My assessment (just looked before I saw Sean's email) is the same as
>> >> his. The NOTICE file embeds other projects' licenses.
>> >
>> > This may be where our perspectives diverge. I did not find those
>> > licenses
>> > embedded in the NOTICE file. As I see it, the licenses are cited but not
>> > included.
>>
>> Pretty sure that was meant to say that NOTICE embeds other projects'
>> "notices", not licenses. And those notices can have all kinds of
>> stuff, including licenses.
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
>> For additional commands, e-mail: dev-h...@spark.apache.org
>>
>

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: [Discuss] NOTICE file for transitive "NOTICE"s

[Sean Owen]

On Fri, Sep 25, 2015 at 10:06 AM, Steve Loughran  wrote:
> regarding the spark one, I don't see that you need to refer to transitive 
> dependencies for the non-binary distros, and, for any binaries, to bother 
> listing the licensing of all the ASF dependencies. Things pulled in from 
> elsewhere & pasted in, that's slightly more complex.

The requirements for including source can be different. There's not
much of it. There's not really a "transitive dependency" for source,
as it is self-contained if copied into the project. I think the source
stuff is dealt with correctly in LICENSE.

Yes you also don't end up needing to repeat the licensing for ASF
dependencies. The issue is BSD/MIT here as far as I can tell
(so-called permissive licenses).


> Uber-JARs, such as spark.jar, do contain lots of classes from everywhere. I 
> don't know the status of them. You could probably get maven to work out the 
> licensing if all the dependencies declare their license.

Indeed, that's exactly why we have to deal with license stuff since
Spark does redistribute other code (not just depend on it). And yes,
using Maven to dig out this info is just what I have done :)

It's not that we missed dependencies, and it's not an issue of NOTICE,
but rather BSD/MIT licenses in LICENSE. The net-net is: inline them.


> On that topic, note that marcelo's proposal to break up that jar and add 
> lib/*.jar to the CP would allow codahale's ganglia support to come in just by 
> dropping in the relevant LGPL JAR, avoiding the need to build a custom spark 
> JAR tainted by the transitive dependency.

(We still couldn't distribute the LGPL bits in Spark, but I don't
think you're suggesting that)

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: [Discuss] NOTICE file for transitive "NOTICE"s

[Richard Hillegas]

Thanks for forking the new email thread, Reynold. It is entirely possible
that I am being overly skittish. I have posed a question for our legal
experts: https://issues.apache.org/jira/browse/LEGAL-226

To answer Sean's question on the previous email thread, I would propose
making changes like the following to the NOTICE file:

Replace a stanza like this...

"This product contains a modified version of 'JZlib', a re-implementation
of
zlib in pure Java, which can be obtained at:

  * LICENSE:
* license/LICENSE.jzlib.txt (BSD Style License)
  * HOMEPAGE:
* http://www.jcraft.com/jzlib/;

...with full license text like this

"This product contains a modified version of 'JZlib', a re-implementation
of
zlib in pure Java, which can be obtained at:

  * HOMEPAGE:
* http://www.jcraft.com/jzlib/

The ZLIB license text follows:

JZlib 0.0.* were released under the GNU LGPL license.  Later, we have
switched
over to a BSD-style license.

--
Copyright (c) 2000-2011 ymnk, JCraft,Inc. All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

  1. Redistributions of source code must retain the above copyright notice,
 this list of conditions and the following disclaimer.

  2. Redistributions in binary form must reproduce the above copyright
 notice, this list of conditions and the following disclaimer in
 the documentation and/or other materials provided with the
distribution.

  3. The names of the authors may not be used to endorse or promote
products
 derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESSED OR IMPLIED
WARRANTIES,
INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
AND
FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL JCRAFT,
INC. OR ANY CONTRIBUTORS TO THIS SOFTWARE BE LIABLE FOR ANY DIRECT,
INDIRECT,
INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE,
EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE."

Thanks,
-Rick



Reynold Xin  wrote on 09/24/2015 10:55:53 AM:

> From: Reynold Xin 
> To: Sean Owen 
> Cc: Richard Hillegas/San Francisco/IBM@IBMUS, "dev@spark.apache.org"
> 
> Date: 09/24/2015 10:56 AM
> Subject: [Discuss] NOTICE file for transitive "NOTICE"s
>
> Richard,
>
> Thanks for bringing this up and this is a great point. Let's start
> another thread for it so we don't hijack the release thread.
>
> On Thu, Sep 24, 2015 at 10:51 AM, Sean Owen  wrote:
> On Thu, Sep 24, 2015 at 6:45 PM, Richard Hillegas 
wrote:
> > Under your guidance, I would be happy to help compile a NOTICE file
which
> > follows the pattern used by Derby and the JDK. This effort might
proceed in
> > parallel with vetting 1.5.1 and could be targeted at a later release
> > vehicle. I don't think that the ASF's exposure is greatly increased by
one
> > more release which follows the old pattern.
>
> I'd prefer to use the ASF's preferred pattern, no? That's what we've
> been trying to do and seems like we're even required to do so, not
> follow a different convention. There is some specific guidance there
> about what to add, and not add, to these files. Specifically, because
> the AL2 requires downstream projects to embed the contents of NOTICE,
> the guidance is to only include elements in NOTICE that must appear
> there.
>
> Put it this way -- what would you like to change specifically? (you
> can start another thread for that)
>
> >> My assessment (just looked before I saw Sean's email) is the same as
> >> his. The NOTICE file embeds other projects' licenses.
> >
> > This may be where our perspectives diverge. I did not find those
licenses
> > embedded in the NOTICE file. As I see it, the licenses are cited but
not
> > included.
>
> Pretty sure that was meant to say that NOTICE embeds other projects'
> "notices", not licenses. And those notices can have all kinds of
> stuff, including licenses.
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> For additional commands, e-mail: dev-h...@spark.apache.org

Re: [Discuss] NOTICE file for transitive "NOTICE"s

[Sean Owen]

Have a look at http://www.apache.org/dev/licensing-howto.html#mod-notice
though, which makes a good point about limiting what goes into NOTICE
to what is required. That's what makes me think we shouldn't do this.

On Thu, Sep 24, 2015 at 7:24 PM, Richard Hillegas  wrote:
> To answer Sean's question on the previous email thread, I would propose
> making changes like the following to the NOTICE file:

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: [Discuss] NOTICE file for transitive "NOTICE"s

[Sean Owen]

Yes, the issue of where 3rd-party license information goes is
different, and varies by license. I think the BSD/MIT licenses are all
already listed in LICENSE accordingly. Let me know if you spy an
omission.

On Thu, Sep 24, 2015 at 8:36 PM, Richard Hillegas <rhil...@us.ibm.com> wrote:
> Thanks for that pointer, Sean. It may be that Derby is putting the license
> information in the wrong place, viz. in the NOTICE file. But the 3rd party
> license text may need to go somewhere else. See for instance the advice a
> little further up the page at
> http://www.apache.org/dev/licensing-howto.html#permissive-deps
>
> Thanks,
> -Rick
>
> Sean Owen <so...@cloudera.com> wrote on 09/24/2015 12:07:01 PM:
>
>> From: Sean Owen <so...@cloudera.com>
>> To: Richard Hillegas/San Francisco/IBM@IBMUS
>> Cc: "dev@spark.apache.org" <dev@spark.apache.org>
>> Date: 09/24/2015 12:08 PM
>> Subject: Re: [Discuss] NOTICE file for transitive "NOTICE"s
>
>
>>
>> Have a look at http://www.apache.org/dev/licensing-howto.html#mod-notice
>> though, which makes a good point about limiting what goes into NOTICE
>> to what is required. That's what makes me think we shouldn't do this.
>>
>> On Thu, Sep 24, 2015 at 7:24 PM, Richard Hillegas <rhil...@us.ibm.com>
>> wrote:
>> > To answer Sean's question on the previous email thread, I would propose
>> > making changes like the following to the NOTICE file:
>>

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: [Discuss] NOTICE file for transitive "NOTICE"s

[Richard Hillegas]

Thanks for that pointer, Sean. It may be that Derby is putting the license
information in the wrong place, viz. in the NOTICE file. But the 3rd party
license text may need to go somewhere else. See for instance the advice a
little further up the page at
http://www.apache.org/dev/licensing-howto.html#permissive-deps

Thanks,
-Rick

Sean Owen <so...@cloudera.com> wrote on 09/24/2015 12:07:01 PM:

> From: Sean Owen <so...@cloudera.com>
> To: Richard Hillegas/San Francisco/IBM@IBMUS
> Cc: "dev@spark.apache.org" <dev@spark.apache.org>
> Date: 09/24/2015 12:08 PM
> Subject: Re: [Discuss] NOTICE file for transitive "NOTICE"s
>
> Have a look at http://www.apache.org/dev/licensing-howto.html#mod-notice
> though, which makes a good point about limiting what goes into NOTICE
> to what is required. That's what makes me think we shouldn't do this.
>
> On Thu, Sep 24, 2015 at 7:24 PM, Richard Hillegas <rhil...@us.ibm.com>
wrote:
> > To answer Sean's question on the previous email thread, I would propose
> > making changes like the following to the NOTICE file:
>