Re: Should we do binary releases?

[Nicolas Malin]

Long thread at all :)

I agree with Jacopo to don't create binary package at this time. We 
missing time to improve the project :( so don't waste time on non 
valuable task.

I understand the mostly reason for QA and no internet access (and I'm 
concerned :)) but if an administrator team have the skill to block 
internet, they have also the skill to create a proxy cache for gradle 
(or something like that).

But sure thanks jacques to raise this potential problem !

Nicolas

Le 25/08/2016 à 13:03, Jacques Le Roux a écrit :
Le 25/08/2016 à 12:32, Taher Alkhateeb a écrit :
Hi Jacques,

Ok great thank you for clarifying.

It is hard to find modern systems that do not utilize the internet.
Anything from node.js, ruby on rails, grails ... the list goes on and 
on.
Even on the user interface, most of the javascript you see when you 
visit a
page requires an internet connection to pull the resources for the UI to
work. Many projects rely less and less on downloading and storing 
copies of
anything. We are in the age where everything connects to everything and
cloud computing is the norm. For example, most websites do not keep a 
local
copy of jQuery, but the client (browser) fetches it on demand. This both
reduces the load on the website server and improves the experience 
for the
user.

I can't get into details but I speak about one of the most important 
Internet services provider (in 2012: 2,4 G€ revenue, 6000+ employees 
almost same number of contractors, Market Cap in 2015: 7.40 G€)
The idea is if your servers cannot connect to the Internet (but 
Internet can connect to them) you are already safer. They have of 
course also several firewalls layers, etc. (not really fun to work with)

Now for the less common cases where people do not have internet (wow) 
there
are workarounds:
- ./gradlew --offline yourCommandsHere. The --offline flag 
description is:
"The build should operate without accessing network resources" 
However you
should have the cache downloaded before using this flag

Thanks Taher, seems we have almost our workaround already documented :)

- You can also copy the .gradle cache from another computer and start 
using
it with the --offline flag

Yep, I thought about that. I needed to extract only the OFBiz related 
libs for OWASP-DC but with OFBIZ-7930 it's no longer needed.
This nicely completes the point above!

- You can always customize for special deployment requirements on 
your own.
Gradle makes it very easy as is proven by your patch in OFBIZ-7783 in 
which
you copied the libs in 3 lines of code!

I agree, from a developer perspective Gradle is the best build system 
I know.
Also a good tool for a sysadmin/devops as long as your GRC allows them 
https://en.wikipedia.org/wiki/Governance,_risk_management,_and_compliance

Jacques


Regards,

Taher Alkhateeb

On Thu, Aug 25, 2016 at 1:02 PM, Jacques Le Roux <
jacques.le.r...@les7arts.com> wrote:

Le 25/08/2016 à 11:33, Taher Alkhateeb a écrit :

Hi Jacques,

Sorry but I'm a little confused. I note the following:

- OFBiz did not create binary releases in the past

Mmm, this is a delicate thing, I'll not say more, you might check by
yourself.

- You started a thread to discuss whether we should create binary 
releases
Yep, nothing prevents us to deliver binary packages (package is the 
right
name as Jacopo outlined)

- When I ask you for the purpose of these releases you reply by saying,
that's why I started this thread.

The purpose is possibly ease for our users.
I initially thought about the case an user is unable to use Gradle 
on her
test/QA/Prod servers (no internet connection on these servers, I was 
there,
did not get the t-shirt, survived). Then the OOTB setting does not 
work and
the user has to find a workaround.
So I though that by providing a binary package we would help users 
in this
and other similar cases. Another possibility is to document a 
workaround.
Nothing is mandatory, only well done source releases are mandatory.

What is it that you are seeking? Are you interested in binary 
releases and
want to know if it is a good idea to pursue?

Yep, exactly

If you are interested, then I
would qualify that as the "purpose" that I asked you about. If you 
are not
interested, then why did you start the thread?

To know if the community is interested. Jacopo at least is not, and 
you as
well I believe.

I'm now in the same mindset  because, as Jacopo said, it's much work 
and I
now think that simply document a workaround for the case above (and
similar) is enough (like using a local Gradle repository)
We can of course neglect it but it could be a difficult turn for some
users w/o this documentation

Jacques



Regards,

Taher Alkhateeb

On Thu, Aug 25, 2016 at 7:32 AM, Jacques Le Roux <
jacques.le.r...@les7arts.com> wrote:

Le 24/08/2016 à 23:15, Taher Alkhateeb a écrit :
Hi Jacques,
I'm not sure how am I supposed to understand it? To me it seems 
clear ..
You cannot add binaries unless they are the result of compiling the
source
code of the release you are preparing, it's written so very 
clearly. It
also makes sense as it is saying that you can provide binary 
releases
that
represent the binary form of YOUR code.

Eventually it boils down to this
http://mail-archives.apache.org/mod_mbox/www-legal-discuss/
201606.mbox/%3cCAAS6=7gVXGHqeKVeFV_r1849Qpi0+Ca0jc2QWQBQfRdZ
ncw...@mail.gmail.com%3e

<<Untrusted jar files (from wherever) are allowed. They must 
represent
compilation of open source dependencies>>

BTW from this complete answer it seems not recommended to release
binaries
though they can also be done by a 3rd party (ie not endorsed by 
the ASF)

On a different but relevant note, why do we want binary releases 
in the

first place? What is the purpose?

The question of this thread is "Should we do binary releases?"
It seems more and more to me that we should neglect them, notably for
security reasons.
Note though that from my OWASP dependency checks (OWAPS-DC), so far
Gradle does not guarantee you from vulnerabilities as I was hoping 
for.
This still needs to be clarified because OWAPS-DC generates a lot of
false
positive...
In this area there is nothing worse than a false sense of 
security. And
it's our responsibility to do our best for our users.

But in last resort, it's the community to decide if we do binary 
releases
or not and the reasons for that. Should we do a vote for that?

Jacques


This is not a desktop application or a

web server that you just want to fire up and start using. There is
preparation work (loading data, configuring, etc ...). It would make
sense
to have a binary version of Tomcat, because I just want to start 
it up
with
defaults and run web applications against it. It would also make 
sense
to
want a binary version of a desktop application because I just 
want to
use
it. The story is completely different with OFBiz, this is not some
software
that you just compile and ship, it's a very customizable, tweakable
system
with many moving parts, especially the database! Having the build 
system
is
essential to its operation, so the whole idea of a binary 
stripped out
release does not make much sense to me.

Taher Alkhateeb

On Wed, Aug 24, 2016 at 11:54 PM, Jacques Le Roux <
jacques.le.r...@les7arts.com> wrote:

Taher,

Wait, either Tomcat, Ant and JMeter are doing it wrong or we don't
understand this sentence (I agree with you) or it's incomplete.

Because if you download each of their binary releases you will 
find in
them "binary/bytecode files" which are not the "result of compiling
that
version of the source code release"

Tomcat: ecj

Ant: ivy (+ 3 optionals)

JMeter: ~50 externals libs

I just checked Wicket: only own binaries, not even optionals 
like Ant.

For Tomcat and Ivy it's maybe optional, but for JMeter it's not it
seems.
I mean JMeter seems to depends on these external libs and they are
delivered in the binary. To be confirmed because I did not dig 
deeper.

It's even more obvious on Geronimo download page:
http://geronimo.apache.org/apache-geronimo-v301-release.html

<<Following distributions use Tomcat as the Web container and 
Axis2 as
the
Web Services engine.>>

I did download the 91 MB, and can confirm it has a total of 346 
jars,
most
not being "result of compiling that version of the source code 
release"

I guess the external libraries are runtime dependencies, in certain
cases
only optional.

I also read at http://www.apache.org/legal/resolved.html#category-b

<<software under the following licenses may be included in 
binary form
within an Apache product if the inclusion is appropriately 
labeled (see
below):>>

So I don't think we can say "In other words we *cannot* include the
dependencies in the binary releases anyway. So people *must* use 
Gradle
to
download the dependencies"

Jacques


Le 24/08/2016 à 17:12, Taher Alkhateeb a écrit :

Hi Jacques,

The discussion we had in OFBIZ-7783 was basically around 
whether or
not
we
should have a task to copy the gradle dependencies into a certain
directory. We went through many discussions, the last one being 
that
this
task might be needed for binary releases.

However, if you look at the reference that _you_ provided you will
notice
that is says that you "may only add binary/bytecode files that 
are the
result of compiling that version of the source code release"

We are _NOT_ compiling any of the dependencies, instead, the build
system
downloads them from jcenter in a precompiled form. In other 
words we
cannot
include the dependencies in the binary releases anyway. So 
people must
use
Gradle to download the dependencies, and so the whole purpose 
of the
binary
release becomes unnecessary as you must have gradle and java 
installed
on
your computer.

Taher Alkhateeb

On Wed, Aug 24, 2016 at 5:36 PM, Jacques Le Roux <
jacques.le.r...@les7arts.com> wrote:

Hi,

At https://issues.apache.org/jira/browse/OFBIZ-7783 we recently 
had a
discussion with Taher about doing or not binary releases.

This is how the ASF defines a binary release (
http://www.apache.org/dev/release.html#what)

<<All releases are in the form of the source materials needed 
to make
changes to the software being released. In some cases,
binary/bytecode
packages are also produced as a convenience to users that 
might not
have
the appropriate tools to build a compiled version of the 
source. In
all
such cases, the binary/bytecode package must have the same 
version
number
as the source release and may only add binary/bytecode files 
that are
the
result of compiling that version of the source code release.>>

So the question is simple (not the answer, you need to think 
ahead):
do
we
want to do binary releases? It comes with some burden, does it 
worth
it?
No
needs to rush an answer :)

If you want more information you can already look at the 
conversation
we
had Pierre, Taher and I at OFBIZ-7783

Thanks

Jacques