Re: zlib configuration : system vs. bundled
On 17/05/2019 09:18, Baesken, Matthias wrote: Hi Alan, thanks for the info . Are you aware of a way to reliably see the info (e.g. in hs_err file) what version of libz was used at runtime ? On linux (with some luck ) we see it in hs_err , at least a lot of distros put the version into the libname , and then it is displayed in the hs_err section Dynamic libraries: ... /lib64/libz.so.1.2.8 .../lib64/libz.so.1.2.8 ... /lib64/libz.so.1.2.8 On macOSX you see nothing just because the version is not in the libname : Dynamic libraries: ... /usr/lib/libz.1.dylib I don't have any insight into how libz is built for macOS. One comment on this is that the frequency of crashes in libz should be significantly reduced in recent releases as the central directory is no longer mmapped. At least in my experience, almost every crash I've seen involving libz or the zip code is because the zip file is replaced while it is mapped. -Alan
RE: zlib configuration : system vs. bundled
Hi Alan, thanks for the info . Are you aware of a way to reliably see the info (e.g. in hs_err file) what version of libz was used at runtime ? On linux (with some luck ) we see it in hs_err , at least a lot of distros put the version into the libname , and then it is displayed in the hs_err section Dynamic libraries: ... /lib64/libz.so.1.2.8 .../lib64/libz.so.1.2.8 ... /lib64/libz.so.1.2.8 On macOSX you see nothing just because the version is not in the libname : Dynamic libraries: ... /usr/lib/libz.1.dylib Best regards, Matthias > -Original Message- > From: Alan Bateman > Sent: Freitag, 17. Mai 2019 09:47 > To: Baesken, Matthias ; 'build- > d...@openjdk.java.net' > Subject: Re: zlib configuration : system vs. bundled > > On 16/05/2019 15:18, Baesken, Matthias wrote: > > Hello Alan, > > > > I found > > > > http://mail.openjdk.java.net/pipermail/core-libs-dev/2015- > March/032106.html > > > > and > > > > http://mail.openjdk.java.net/pipermail/build-dev/2016- > February/thread.html#16602 > > > > but without much details on real or potential performance > improvements . > > > > Both discussion threads are pretty old. I do not think they cover the > changes done in the meantime in jdk9 and higherin the java.util.zip > package . > > I think a lot of coding there moved from C to Java , so the libz is > > used in > JDK less these days than in the "old times " . > Yes, the ZipFile implementation has changed significantly (many of the > motivations are listed in JDK-8142508) but the compression and checksum > (except CRC32C) will use libz. You'll see hotspot using it too, say when > running with -Xbootclasspath/a to add a JAR file to the boot class path. > There has many threads here and on core-libs-dev about using the bundled > vs. system zlib. It was an issue for the Linux distros in the early days > of OpenJDK as they have policies to not include copies of libraries that > the OS provides. At one point there was a patch proposing a XX option to > select the system vs. bundled zlib at run-time, the motivation being to > be able to switch to the Intel IPP implementation. It's impossible to > please everyone but I think the default that we settled on is probably > the best. > > -Alan
Re: zlib configuration : system vs. bundled
On 16/05/2019 15:18, Baesken, Matthias wrote: Hello Alan, I found http://mail.openjdk.java.net/pipermail/core-libs-dev/2015-March/032106.html and http://mail.openjdk.java.net/pipermail/build-dev/2016-February/thread.html#16602 but without much details on real or potential performance improvements . Both discussion threads are pretty old. I do not think they cover the changes done in the meantime in jdk9 and higherin the java.util.zip package . I think a lot of coding there moved from C to Java , so the libz is used in JDK less these days than in the "old times " . Yes, the ZipFile implementation has changed significantly (many of the motivations are listed in JDK-8142508) but the compression and checksum (except CRC32C) will use libz. You'll see hotspot using it too, say when running with -Xbootclasspath/a to add a JAR file to the boot class path. There has many threads here and on core-libs-dev about using the bundled vs. system zlib. It was an issue for the Linux distros in the early days of OpenJDK as they have policies to not include copies of libraries that the OS provides. At one point there was a patch proposing a XX option to select the system vs. bundled zlib at run-time, the motivation being to be able to switch to the Intel IPP implementation. It's impossible to please everyone but I think the default that we settled on is probably the best. -Alan
RE: zlib configuration : system vs. bundled
Hello Alan, I found http://mail.openjdk.java.net/pipermail/core-libs-dev/2015-March/032106.html and http://mail.openjdk.java.net/pipermail/build-dev/2016-February/thread.html#16602 but without much details on real or potential performance improvements . Both discussion threads are pretty old. I do not think they cover the changes done in the meantime in jdk9 and higherin the java.util.zip package . I think a lot of coding there moved from C to Java , so the libz is used in JDK less these days than in the "old times " . > > Sure but the opposite can arise too, say where someone is using a JDK > release that bundles an older version of zlib. > True , unfortunately the old JDK with bundled libz would not use the "great new current libz" from the distro (in case there is such a recent version available ), People would need to patch the JDK . At least we have the freedom to choose with the configure-flags . Best regards, Matthias > On 15/05/2019 09:16, Baesken, Matthias wrote: > > Hi Alan, thanks for pointing me at the old discussion . > > > > http://mail.openjdk.java.net/pipermail/build-dev/2016- > February/016602.html > > > > talks about performance benefits . Are you aware of some benchmarks > that showed the improvements ? > If you can find the mails that references the Intel IPP library then you > might have links to performance data comparing the different > implementations. There is also mails somewhere in the archive with a > proposal or patch to have the JDK select an alternative implementation > at run-time, equivalent to LD_PRELOAD. I think we ended up with the > right default. > > > > > In reality, if you have the latest distro versions you might be lucky and > > you > have a nice recent zlib 1.2.11 . > > However on older distros , you run in reality into older zlibs (often I > > see > 1.2.8). I don't think that this is a very good status . > Sure but the opposite can arise too, say where someone is using a JDK > release that bundles an older version of zlib. > > I agree with the comment that the building instructions need to be > sync'ed up. > > -Alan.
Re: zlib configuration : system vs. bundled
On 16/05/2019 12:34, Baesken, Matthias wrote: Hello, the updated webrev is here : http://cr.openjdk.java.net/~mbaesken/webrevs/8223944.1/ This looks good to me. -Alan
Re: zlib configuration : system vs. bundled
Looks ok. /Erik On 2019-05-16 04:34, Baesken, Matthias wrote: Hello, the updated webrev is here : http://cr.openjdk.java.net/~mbaesken/webrevs/8223944.1/ Best regards, Matthias On 2019-05-15 01:16, Baesken, Matthias wrote: Hi Alan, thanks for pointing me at the old discussion . http://mail.openjdk.java.net/pipermail/build-dev/2016- February/016602.html talks about performance benefits . Are you aware of some benchmarks that showed the improvements ? In reality, if you have the latest distro versions you might be lucky and you have a nice recent zlib 1.2.11 . However on older distros , you run in reality into older zlibs (often I see 1.2.8). I don't think that this is a very good status . At least I think building.md should be fixed to state the real status , the current info is wrong : "Certain third-party libraries used by the JDK (libjpeg, giflib, libpng, lcms and zlib) are included in the JDK repository. The default behavior of the JDK build is to use this version of these libraries, but they might be replaced by an external version. To do so, specify `system` as the `` option in these arguments. (The default is `bundled`)." Btw how is building.html generated , is this coming from building.md ? Yes, you need to configure with pandoc. Then you run "make update-build-docs". /Erik
RE: zlib configuration : system vs. bundled
Hello, the updated webrev is here : http://cr.openjdk.java.net/~mbaesken/webrevs/8223944.1/ Best regards, Matthias > > > On 2019-05-15 01:16, Baesken, Matthias wrote: > > Hi Alan, thanks for pointing me at the old discussion . > > > > http://mail.openjdk.java.net/pipermail/build-dev/2016- > February/016602.html > > > > talks about performance benefits . Are you aware of some benchmarks > that showed the improvements ? > > > > In reality, if you have the latest distro versions you might be lucky and > > you > have a nice recent zlib 1.2.11 . > > However on older distros , you run in reality into older zlibs (often I > > see > 1.2.8). I don't think that this is a very good status . > > > > > > At least I think building.md should be fixed to state the real status , > > the > current info is wrong : > > > > > > "Certain third-party libraries used by the JDK (libjpeg, giflib, libpng, > > lcms > > and zlib) are included in the JDK repository. The default behavior of the > > JDK build is to use this version of these libraries, but they might be > > replaced by an external version. To do so, specify `system` as the > `` > > option in these arguments. (The default is `bundled`)." > > > > > > > > Btw how is building.html generated , is this coming from building.md ? > > Yes, you need to configure with pandoc. Then you run "make > update-build-docs". > > /Erik >
Re: zlib configuration : system vs. bundled
On 2019-05-15 01:16, Baesken, Matthias wrote:
Hi Alan, thanks for pointing me at the old discussion .
http://mail.openjdk.java.net/pipermail/build-dev/2016-February/016602.html
talks about performance benefits . Are you aware of some benchmarks that
showed the improvements ?
In reality, if you have the latest distro versions you might be lucky and you
have a nice recent zlib 1.2.11 .
However on older distros , you run in reality into older zlibs (often I see
1.2.8). I don't think that this is a very good status .
At least I think building.md should be fixed to state the real status , the
current info is wrong :
"Certain third-party libraries used by the JDK (libjpeg, giflib, libpng, lcms
and zlib) are included in the JDK repository. The default behavior of the
JDK build is to use this version of these libraries, but they might be
replaced by an external version. To do so, specify `system` as the ``
option in these arguments. (The default is `bundled`)."
Btw how is building.html generated , is this coming from building.md ?
Yes, you need to configure with pandoc. Then you run "make
update-build-docs".
/Erik
Best regards, Matthias
-Original Message-
From: Alan Bateman
Sent: Dienstag, 14. Mai 2019 17:47
To: Baesken, Matthias ; 'build-
d...@openjdk.java.net'
Subject: Re: zlib configuration : system vs. bundled
On 14/05/2019 15:58, Baesken, Matthias wrote:
:
On the other OS platforms, in case a zlib is found on the system :
if test "x${ZLIB_FOUND}" != "xyes"; then
# If we don't find any system...set default to bundled
DEFAULT_ZLIB=bundled
fi
we use it from the system .
Wouldn't it be more consistent to have zlib=bundled as well as default for
the other UNIX platforms + MacOSX ?
( people who wish to use the system zlib still can configure it )
Otherwise we often run into using old zlib installations at build time which
might not be desired.
It was a deliberate change in JDK 9 to use the system zlib if possible.
Windows is the outlier. If you through the archives of core-libs-dev
then you should find several discussions about this, I think the most
recent was in 2016, subject line "JDK-8031767 Support system or
alternative implementations of zlib".
-Alan
Re: 8223944: fix zlib related building docu and comments - was : RE: zlib configuration : system vs. bundled
Thanks for updating the docs! I have some details to comment on.
In general, could you try to keep the doc/comments within the 80 char
limit and remove the arbitrary line breaks so that it looks uniform?
More specifically:
doc/building.md
725: Break line.
727: Grammatical nit: might -> may
728: Break line.
make/autoconf/lib-bundled.m4
30: Break line. Spelling: refered -> referred
31: Space before dot. Remove line break.
33: Remove line break.
34: Grammatical nit: might -> may
/Erik
On 2019-05-15 02:32, Baesken, Matthias wrote:
Btw I adjusted the build docu and some m4 file comments regarding the zlib
usage :
https://bugs.openjdk.java.net/browse/JDK-8223944
http://cr.openjdk.java.net/~mbaesken/webrevs/8223944.0/
Best regards, Matthias
-Original Message-
From: Baesken, Matthias
Sent: Mittwoch, 15. Mai 2019 10:16
To: 'Alan Bateman' ; 'build-
d...@openjdk.java.net'
Subject: RE: zlib configuration : system vs. bundled
Hi Alan, thanks for pointing me at the old discussion .
http://mail.openjdk.java.net/pipermail/build-dev/2016-
February/016602.html
talks about performance benefits . Are you aware of some benchmarks that
showed the improvements ?
In reality, if you have the latest distro versions you might be lucky and you
have a nice recent zlib 1.2.11 .
However on older distros , you run in reality into older zlibs (often I see
1.2.8). I don't think that this is a very good status .
At least I think building.md should be fixed to state the real status , the
current info is wrong :
"Certain third-party libraries used by the JDK (libjpeg, giflib, libpng, lcms
and zlib) are included in the JDK repository. The default behavior of the
JDK build is to use this version of these libraries, but they might be
replaced by an external version. To do so, specify `system` as the ``
option in these arguments. (The default is `bundled`)."
Btw how is building.html generated , is this coming from building.md ?
Best regards, Matthias
-Original Message-
From: Alan Bateman
Sent: Dienstag, 14. Mai 2019 17:47
To: Baesken, Matthias ; 'build-
d...@openjdk.java.net'
Subject: Re: zlib configuration : system vs. bundled
On 14/05/2019 15:58, Baesken, Matthias wrote:
:
On the other OS platforms, in case a zlib is found on the system :
if test "x${ZLIB_FOUND}" != "xyes"; then
# If we don't find any system...set default to bundled
DEFAULT_ZLIB=bundled
fi
we use it from the system .
Wouldn't it be more consistent to have zlib=bundled as well as default
for
the other UNIX platforms + MacOSX ?
( people who wish to use the system zlib still can configure it )
Otherwise we often run into using old zlib installations at build time which
might not be desired.
It was a deliberate change in JDK 9 to use the system zlib if possible.
Windows is the outlier. If you through the archives of core-libs-dev
then you should find several discussions about this, I think the most
recent was in 2016, subject line "JDK-8031767 Support system or
alternative implementations of zlib".
-Alan
Re: zlib configuration : system vs. bundled
On 15/05/2019 09:16, Baesken, Matthias wrote: Hi Alan, thanks for pointing me at the old discussion . http://mail.openjdk.java.net/pipermail/build-dev/2016-February/016602.html talks about performance benefits . Are you aware of some benchmarks that showed the improvements ? If you can find the mails that references the Intel IPP library then you might have links to performance data comparing the different implementations. There is also mails somewhere in the archive with a proposal or patch to have the JDK select an alternative implementation at run-time, equivalent to LD_PRELOAD. I think we ended up with the right default. In reality, if you have the latest distro versions you might be lucky and you have a nice recent zlib 1.2.11 . However on older distros , you run in reality into older zlibs (often I see 1.2.8). I don't think that this is a very good status . Sure but the opposite can arise too, say where someone is using a JDK release that bundles an older version of zlib. I agree with the comment that the building instructions need to be sync'ed up. -Alan.
8223944: fix zlib related building docu and comments - was : RE: zlib configuration : system vs. bundled
Btw I adjusted the build docu and some m4 file comments regarding the zlib
usage :
https://bugs.openjdk.java.net/browse/JDK-8223944
http://cr.openjdk.java.net/~mbaesken/webrevs/8223944.0/
Best regards, Matthias
> -Original Message-
> From: Baesken, Matthias
> Sent: Mittwoch, 15. Mai 2019 10:16
> To: 'Alan Bateman' ; 'build-
> d...@openjdk.java.net'
> Subject: RE: zlib configuration : system vs. bundled
>
> Hi Alan, thanks for pointing me at the old discussion .
>
> http://mail.openjdk.java.net/pipermail/build-dev/2016-
> February/016602.html
>
> talks about performance benefits . Are you aware of some benchmarks that
> showed the improvements ?
>
> In reality, if you have the latest distro versions you might be lucky and
> you
> have a nice recent zlib 1.2.11 .
> However on older distros , you run in reality into older zlibs (often I see
> 1.2.8). I don't think that this is a very good status .
>
>
> At least I think building.md should be fixed to state the real status , the
> current info is wrong :
>
>
> "Certain third-party libraries used by the JDK (libjpeg, giflib, libpng, lcms
> and zlib) are included in the JDK repository. The default behavior of the
> JDK build is to use this version of these libraries, but they might be
> replaced by an external version. To do so, specify `system` as the ``
> option in these arguments. (The default is `bundled`)."
>
>
>
> Btw how is building.html generated , is this coming from building.md ?
>
> Best regards, Matthias
>
>
>
>
> > -----Original Message-
> > From: Alan Bateman
> > Sent: Dienstag, 14. Mai 2019 17:47
> > To: Baesken, Matthias ; 'build-
> > d...@openjdk.java.net'
> > Subject: Re: zlib configuration : system vs. bundled
> >
> > On 14/05/2019 15:58, Baesken, Matthias wrote:
> > > :
> > >
> > > On the other OS platforms, in case a zlib is found on the system :
> > >
> > >if test "x${ZLIB_FOUND}" != "xyes"; then
> > > # If we don't find any system...set default to bundled
> > > DEFAULT_ZLIB=bundled
> > >fi
> > >
> > > we use it from the system .
> > > Wouldn't it be more consistent to have zlib=bundled as well as default
> for
> > the other UNIX platforms + MacOSX ?
> > > ( people who wish to use the system zlib still can configure it )
> > >
> > > Otherwise we often run into using old zlib installations at build time
> > > which
> > might not be desired.
> > >
> > It was a deliberate change in JDK 9 to use the system zlib if possible.
> > Windows is the outlier. If you through the archives of core-libs-dev
> > then you should find several discussions about this, I think the most
> > recent was in 2016, subject line "JDK-8031767 Support system or
> > alternative implementations of zlib".
> >
> > -Alan
RE: zlib configuration : system vs. bundled
Hi Alan, thanks for pointing me at the old discussion .
http://mail.openjdk.java.net/pipermail/build-dev/2016-February/016602.html
talks about performance benefits . Are you aware of some benchmarks that
showed the improvements ?
In reality, if you have the latest distro versions you might be lucky and you
have a nice recent zlib 1.2.11 .
However on older distros , you run in reality into older zlibs (often I see
1.2.8). I don't think that this is a very good status .
At least I think building.md should be fixed to state the real status , the
current info is wrong :
"Certain third-party libraries used by the JDK (libjpeg, giflib, libpng, lcms
and zlib) are included in the JDK repository. The default behavior of the
JDK build is to use this version of these libraries, but they might be
replaced by an external version. To do so, specify `system` as the ``
option in these arguments. (The default is `bundled`)."
Btw how is building.html generated , is this coming from building.md ?
Best regards, Matthias
> -Original Message-
> From: Alan Bateman
> Sent: Dienstag, 14. Mai 2019 17:47
> To: Baesken, Matthias ; 'build-
> d...@openjdk.java.net'
> Subject: Re: zlib configuration : system vs. bundled
>
> On 14/05/2019 15:58, Baesken, Matthias wrote:
> > :
> >
> > On the other OS platforms, in case a zlib is found on the system :
> >
> >if test "x${ZLIB_FOUND}" != "xyes"; then
> > # If we don't find any system...set default to bundled
> > DEFAULT_ZLIB=bundled
> >fi
> >
> > we use it from the system .
> > Wouldn't it be more consistent to have zlib=bundled as well as default
> > for
> the other UNIX platforms + MacOSX ?
> > ( people who wish to use the system zlib still can configure it )
> >
> > Otherwise we often run into using old zlib installations at build time which
> might not be desired.
> >
> It was a deliberate change in JDK 9 to use the system zlib if possible.
> Windows is the outlier. If you through the archives of core-libs-dev
> then you should find several discussions about this, I think the most
> recent was in 2016, subject line "JDK-8031767 Support system or
> alternative implementations of zlib".
>
> -Alan
Re: zlib configuration : system vs. bundled
On 14/05/2019 15:58, Baesken, Matthias wrote:
:
On the other OS platforms, in case a zlib is found on the system :
if test "x${ZLIB_FOUND}" != "xyes"; then
# If we don't find any system...set default to bundled
DEFAULT_ZLIB=bundled
fi
we use it from the system .
Wouldn't it be more consistent to have zlib=bundled as well as default for
the other UNIX platforms + MacOSX ?
( people who wish to use the system zlib still can configure it )
Otherwise we often run into using old zlib installations at build time which
might not be desired.
It was a deliberate change in JDK 9 to use the system zlib if possible.
Windows is the outlier. If you through the archives of core-libs-dev
then you should find several discussions about this, I think the most
recent was in 2016, subject line "JDK-8031767 Support system or
alternative implementations of zlib".
-Alan
