Re: Adding Microbenchmarks to the JDK forest/trees (JEP-230)

[joe darcy]

Hello,

On 12/4/2014 4:34 PM, David Holmes wrote:

Hi Staffan,

On 2/12/2014 10:08 AM, Staffan Friberg wrote:

Hi,

Hopefully this is the right list for this discussion.

As part of adding Microbenchmarks to the OpenJDK source tree, I'm trying
to understand how we best would add the benchmark sources to the
existing OpenJDK tree structure.


Is there a reason this needs to be inside the OpenJDK instead of a 
stand-alone project? If it ends up in its own repo and the repo is not 
needed by anything else, then it is already like a stand-alone project.


I think David is raising a good question here. A related question is if 
we want to update/fix the microbenchmarks, in how many release trains 
will we want to make that fix? If the expected answer is much greater 
than one, to me that would seem to me to argue for a separate forest in 
the overall OpenJDK effort, but not bundled into a particular release.


For example, in the past, the webrev.ksh script was included in the JDK 
forest. That was an improvement over every engineer having his or her 
personal fork, but still made keeping webrev updated unnecessarily 
difficult since any changes would need to be done multiple time and 
there is nothing fundamentally binding a version of webrev to a 
particular JDK release. Likewise, even though (to my knowledge) jtreg is 
only used for testing the JDK, jtreg has its own repository and release 
schedule separate from any JDK release.


So if the microbenchmarks are to a first-approximation version agnostic, 
then they should probably have a forest not associated with a JDK 
release. If they are tightly bound to a release, that argues for putting 
them into the JDK forest itself.


-Joe




David


Since the microbenchmark suite will cover all parts of the JDK, covering
HotSpot, JDK libraries and Nashorn, it would be preferred to add the
microbenchmark directory as a new top level directory. Something similar
to the following structure. Having "benchmark" as the top-level
directory would allow us to later add different types of benchmarks
without colliding with the microbenchmark suite.

/
benchmark/microbenchmark/...
hotspot/...
jdk/...
nashorn/...

With this as the premise I can see the following 3 options for how this
could be added to the source code layout

1. Part of jdk-root repository
  * Only makes sense if we want to move in a direction with fewer
trees (and eventually a single tree)
2. Part of another already existing tree
  * Not sure if this is possible without converting and moving the
directory to a subdirectory of that tree
3. New tree in the forest/tree structure
  * Most logical option as it follows the current setup and 
structure



Anyone have any comments and/or concerns on the suggested directory
location and the tree structure in option 3.

Would the build-dev team be the right group to later help setup a new
tree if decided to be the right way to go?

Regards,
Staffan

Re: Confirming JDK 8 GA version

[Medi Montaseri]

Thank you very much David and Erik for clarifying these issues for me. Looks 
like we all believe it doing it right, doing it once.
I am impressed. Good job

Medi


On 12/04/2014 04:41 PM, David Holmes wrote:
Just to further clarify things, the hotspot version is set in the build files:

jdk8/hotspot/make/hotspot_version

HS_MAJOR_VER=25
HS_MINOR_VER=0
HS_BUILD_NUMBER=70

so that is why you see it in your local build. The JDK version string, as Erik 
says, takes on values based on configure arguments (if passed) else defaults 
(that can include timestamps). The official Oracle JDK builds use a specific 
format for the JDK version string.

Also if you look in hotspot/.hgtags you will see the last two lines:

9f9179e8f0cfe74c08f3716cf3c38e21e1de4c4a hs25-b70
0c94c41dcd70e9a9b4d96e31275afd5a73daa72d jdk8-b132

confirming that JDK8-b132 corresponds to hotspot hs25-b70.

David

On 4/12/2014 6:50 PM, Erik Joelsson wrote:

On 2014-12-03 21:54, Medi Montaseri wrote:
Thanks David and Erik,

A bit more confused...I don't know what hotspot is.
Hotspot is the jvm, basically libjvm.so (and a few of the support libs)
and has historically been treated as a separate project from the rest of
the jdk. Because of that it has its own version string. When you run
java -version in jdk8, you see three lines. The last one that says "VM"
belongs to Hotspot. It ignores the input from configure.
I am aware of configure --with-build-number and --with-milestone and
indeed use themand that is why I get

bin/java -version
openjdk version "1.8.0-20141202"// note how this version
is stamped with a date
OpenJDK Runtime Environment (build 1.8.0-20141202-HDS-174701)
// note how this portion is also decorated with date, label and more
numbers
OpenJDK 64-Bit Server VM (build 25.0-b70, mixed mode)
   ///
what is this b70

The b70 makes me think that I am not working with the tip
Look at that line with "VM" and "b70" and compare it to the same line
from jdk8 GA. They are the exact same.
Let me ask the question another way...

If I pull the latest and "hg tags" says b132 as in

 unix> cd srcDir/jdk ; hg tags | head -2
tip 9107:687fd7c7986d
jdk8-b132   9106:43cb25339b55

Can I then be sure that I am indeed working with the tip...?
AFAIK yes.

David...you see how yours is showing the b132

java version "1.8.0"
Java(TM) SE Runtime Environment (build 1.8.0-b132)   // look
there is the b132 thing  -b132
Java HotSpot(TM) Server VM (build 25.0-b70, mixed mode)
Yes, this shows b132, in the line that is only defined by configure
arguments, and that you are setting to something else.

/Erik
Again, I need to make sure I have the latest and be able to query some
binary (java or javac) to show that it is the latest.

Thanks
Medi


From: Erik Joelsson [erik.joels...@oracle.com]
Sent: Wednesday, December 03, 2014 12:33 AM
To: David Holmes; Medi Montaseri; 
build-dev@openjdk.java.net
Subject: Re: Confirming JDK 8 GA version

The source/makefiles are not aware of the build number. It's assigned
from an external source at the time of building a promoted build. If you
want your version number to better match 8 GA, you will have to play
around with the configure arguments --with-build-number and possibly
--with-milestone.

Also, as David pointed out, in JDK 8, hotspot has it's own numbering.

/Erik

On 2014-12-03 06:13, David Holmes wrote:
On 3/12/2014 12:25 PM, Medi Montaseri wrote:
Hi,

Just wanted to confirm some issues:
I am interested in building the latest (tip) of jdk8 GA release.

1- jdk8 GA is different from jdk8u (u is for updates), I should get
the src from http://hg.openjdk.java.net/jdk8/jdk8
2- The latest tagged version of the src is
"jdk8-b132   9106:43cb25339b55"
3- I confirm that I have the latest src (for GA) by running "hg tags"
like
 unix> cd srcDir/jdk ; hg tags | head -2
tip 9107:687fd7c7986d
jdk8-b132   9106:43cb25339b55

I go ahead and build everything and then run "java -version" and I see

openjdk version "1.8.0-20141202"
OpenJDK Runtime Environment (build 1.8.0-20141202-HDS-174701)
OpenJDK 64-Bit Server VM (build 25.0-b70, mixed mode)

Q- Why am I seeing "build 25.0-b70" ? Does this "b70" has anything to
do with above "jdk8-b132" ?
That is the hotspot version number and build number as used in the GA
build of JDK 8. It is correct.

java version "1.8.0"
Java(TM) SE Runtime Environment (build 1.8.0-b132)
Java HotSpot(TM) Server VM (build 25.0-b70, mixed mode)

David
-

Thanks
Medi

Re: Confirming JDK 8 GA version

[David Holmes]

Just to further clarify things, the hotspot version is set in the build 
files:


jdk8/hotspot/make/hotspot_version

HS_MAJOR_VER=25
HS_MINOR_VER=0
HS_BUILD_NUMBER=70

so that is why you see it in your local build. The JDK version string, 
as Erik says, takes on values based on configure arguments (if passed) 
else defaults (that can include timestamps). The official Oracle JDK 
builds use a specific format for the JDK version string.


Also if you look in hotspot/.hgtags you will see the last two lines:

9f9179e8f0cfe74c08f3716cf3c38e21e1de4c4a hs25-b70
0c94c41dcd70e9a9b4d96e31275afd5a73daa72d jdk8-b132

confirming that JDK8-b132 corresponds to hotspot hs25-b70.

David

On 4/12/2014 6:50 PM, Erik Joelsson wrote:


On 2014-12-03 21:54, Medi Montaseri wrote:

Thanks David and Erik,

A bit more confused...I don't know what hotspot is.

Hotspot is the jvm, basically libjvm.so (and a few of the support libs)
and has historically been treated as a separate project from the rest of
the jdk. Because of that it has its own version string. When you run
java -version in jdk8, you see three lines. The last one that says "VM"
belongs to Hotspot. It ignores the input from configure.

I am aware of configure --with-build-number and --with-milestone and
indeed use themand that is why I get

bin/java -version
openjdk version "1.8.0-20141202"// note how this version
is stamped with a date
OpenJDK Runtime Environment (build 1.8.0-20141202-HDS-174701)
// note how this portion is also decorated with date, label and more
numbers
OpenJDK 64-Bit Server VM (build 25.0-b70, mixed mode)
   ///
what is this b70

The b70 makes me think that I am not working with the tip

Look at that line with "VM" and "b70" and compare it to the same line
from jdk8 GA. They are the exact same.

Let me ask the question another way...

If I pull the latest and "hg tags" says b132 as in


 unix> cd srcDir/jdk ; hg tags | head -2
tip 9107:687fd7c7986d
jdk8-b132   9106:43cb25339b55


Can I then be sure that I am indeed working with the tip...?

AFAIK yes.


David...you see how yours is showing the b132


java version "1.8.0"
Java(TM) SE Runtime Environment (build 1.8.0-b132)   // look
there is the b132 thing  -b132
Java HotSpot(TM) Server VM (build 25.0-b70, mixed mode)

Yes, this shows b132, in the line that is only defined by configure
arguments, and that you are setting to something else.

/Erik

Again, I need to make sure I have the latest and be able to query some
binary (java or javac) to show that it is the latest.

Thanks
Medi


From: Erik Joelsson [erik.joels...@oracle.com]
Sent: Wednesday, December 03, 2014 12:33 AM
To: David Holmes; Medi Montaseri; build-dev@openjdk.java.net
Subject: Re: Confirming JDK 8 GA version

The source/makefiles are not aware of the build number. It's assigned
from an external source at the time of building a promoted build. If you
want your version number to better match 8 GA, you will have to play
around with the configure arguments --with-build-number and possibly
--with-milestone.

Also, as David pointed out, in JDK 8, hotspot has it's own numbering.

/Erik

On 2014-12-03 06:13, David Holmes wrote:

On 3/12/2014 12:25 PM, Medi Montaseri wrote:

Hi,

Just wanted to confirm some issues:
I am interested in building the latest (tip) of jdk8 GA release.

1- jdk8 GA is different from jdk8u (u is for updates), I should get
the src from http://hg.openjdk.java.net/jdk8/jdk8
2- The latest tagged version of the src is
"jdk8-b132   9106:43cb25339b55"
3- I confirm that I have the latest src (for GA) by running "hg tags"
like
 unix> cd srcDir/jdk ; hg tags | head -2
tip 9107:687fd7c7986d
jdk8-b132   9106:43cb25339b55

I go ahead and build everything and then run "java -version" and I see

openjdk version "1.8.0-20141202"
OpenJDK Runtime Environment (build 1.8.0-20141202-HDS-174701)
OpenJDK 64-Bit Server VM (build 25.0-b70, mixed mode)

Q- Why am I seeing "build 25.0-b70" ? Does this "b70" has anything to
do with above "jdk8-b132" ?

That is the hotspot version number and build number as used in the GA
build of JDK 8. It is correct.

java version "1.8.0"
Java(TM) SE Runtime Environment (build 1.8.0-b132)
Java HotSpot(TM) Server VM (build 25.0-b70, mixed mode)

David
-


Thanks
Medi

Re: Adding Microbenchmarks to the JDK forest/trees (JEP-230)

[David Holmes]

Hi Staffan,

On 2/12/2014 10:08 AM, Staffan Friberg wrote:

Hi,

Hopefully this is the right list for this discussion.

As part of adding Microbenchmarks to the OpenJDK source tree, I'm trying
to understand how we best would add the benchmark sources to the
existing OpenJDK tree structure.


Is there a reason this needs to be inside the OpenJDK instead of a 
stand-alone project? If it ends up in its own repo and the repo is not 
needed by anything else, then it is already like a stand-alone project.


David


Since the microbenchmark suite will cover all parts of the JDK, covering
HotSpot, JDK libraries and Nashorn, it would be preferred to add the
microbenchmark directory as a new top level directory. Something similar
to the following structure. Having "benchmark" as the top-level
directory would allow us to later add different types of benchmarks
without colliding with the microbenchmark suite.

/
benchmark/microbenchmark/...
hotspot/...
jdk/...
nashorn/...

With this as the premise I can see the following 3 options for how this
could be added to the source code layout

1. Part of jdk-root repository
  * Only makes sense if we want to move in a direction with fewer
trees (and eventually a single tree)
2. Part of another already existing tree
  * Not sure if this is possible without converting and moving the
directory to a subdirectory of that tree
3. New tree in the forest/tree structure
  * Most logical option as it follows the current setup and structure


Anyone have any comments and/or concerns on the suggested directory
location and the tree structure in option 3.

Would the build-dev team be the right group to later help setup a new
tree if decided to be the right way to go?

Regards,
Staffan

Re: Adding Microbenchmarks to the JDK forest/trees (JEP-230)

[mark . reinhold]

2014/12/4 9:51 -0800, staffan.frib...@oracle.com:
> On 12/03/2014 02:58 AM, Magnus Ihse Bursie wrote:
>> ...
>> 
>> My suggestion is that the microbenchmarks are put in the top-level 
>> repo, if only for the reason that it seems fully possible to split 
>> them out to a separate repo some time in the future if it grows too 
>> much, but it seems much more unlikely that it will ever be moved back 
>> into the top-level repo if we realized it was a stupid idea to put it 
>> in a separate repo.
> 
> I like this idea, and agree that shifting in the opposite direction is 
> probably something that would be much more work than breaking it out if 
> size becomes an issue further down the road.
> 
> When moving a directory to a new sub-repository, is there any concern 
> about the diffs for that set of files still lingering in the top repo, 
> or can those be moved as well?

The files can be moved, but their earlier history will remain in the
top-level repo.

Given the forest structure that we have today, and the fact that this
set of tests could grow to a thousand or so files over time, I think it
makes more sense to place them in a new top-level repo of their own.

In the (very?) long term we might dramatically reduce the number of
repositories but that will be a huge change, and in the mean time adding
one more repo is a pretty minor change and also consistent with current
practice.

A pleasant property of the current root repo is that it's very small
(< 9MB) and easy to understand, containing mostly makefiles, build
utilities, and some metadata.  Placing tests in it would start turning
it into more of a grab-bag.

I don't think it makes sense to split the microbenchmarks across
different repos, as we already do with the unit and regression tests.

The latter types of tests primarily address the correctness of the code
in one repo, on the assumption that any code in other repos upon which
that code depends is correct.  In most cases, therefore, it's pretty
easy to tell whether a unit or regression test is specific to, say, the
jdk repo, the langtools repo, or some other repo.

Microbenchmarks, by contrast, address the performance of a particular
API, language construct, or other programmatic interface and also all of
the code upon which it depends, regardless of which repo it came from.
By nature they're more holistic, and so less strongly associated with
the code in any particular repo.

- Mark

Re: [9] RFR (XS) 8066508: JTReg tests timeout on slow devices when run using JPRT

[David Holmes]

Hi Chris,

Sorry I mis-directed you to send this one to build-dev, as it is a 
hotspot test/Makefile fix it should be reviewed by hotspot-dev now cc'd.


Fix looks good to me.

Thanks,
David

On 5/12/2014 6:37 AM, Chris Plummer wrote:

Please review the following fix to address JPRT timeout issues when
using -rtests to run hotspot JTReg tests on slow devices. The same logic
has been in place for jdk/test/Makefile for a while now, so I just
copied from there to hotspot/test/Makefile.

https://bugs.openjdk.java.net/browse/JDK-8066508
http://cr.openjdk.java.net/~cjplummer/8066508/webrev.00/

thanks,

Chris

Re: [9] RFR (XS) 8066507: JPRT is not capable of running jtreg tests located jdk/test

[David Holmes]

Looks good!

Thanks for fixing.

David

On 5/12/2014 6:37 AM, Chris Plummer wrote:

Please review the following fix to allow running jdk jtreg tests using
JPRT -rtests. This is the same fix that his been in place in
hotspot/test/Makefile for a while now, so I just copied from there to
jdk/test/Makefile.

https://bugs.openjdk.java.net/browse/JDK-8066507
http://cr.openjdk.java.net/~cjplummer/8066507/webrev.00/

thanks,

Chris

Re: [8u60] RFR: 8038189: Add cross-platform compact profiles support

[David Holmes]

Thanks Erik. Once 8u/dev is open for 8u60 I'll request push approval.

David

On 4/12/2014 6:53 PM, Erik Joelsson wrote:

Looks good to me.

/Erik

On 2014-12-04 04:16, David Holmes wrote:

Updated webrev:

http://cr.openjdk.java.net/~dholmes/8038189/webrev.jdk.v2/

Only changes are to profile-includes.txt.

Thanks,
David

On 2/12/2014 2:24 PM, David Holmes wrote:

Erik,

Many thanks for the makefile macro wizardry! I will incorporate, test
and return with an updated webreb.

David

On 1/12/2014 7:19 PM, Erik Joelsson wrote:

Hello David,

Most of it looks good, but profile-includes.txt could certainly benefit
from some reduction in duplication. (Aside from the extension of the
file itself, which I find a bit weird, but it's already there.) The two
common structures that I can see are:
1. Expanding debuginfo files for macosx.
2. Assigning/expanding libraries to the correct variable.

For macosx debuginfo expanding, I would define a macro to something
like
this:

# Expand the contents of the .dSYM directories on macosx.
# Param 1 - debug files list
# Param 2 - libraries list
define expand-debuginfo
   $(if $(and $(filter-out true, $(ZIP_DEBUGINFO_FILES)), $(filter
macosx, $(OPENJDK_TARGET_OS))), \
   $(foreach i, $1, $(addsuffix /Contents/Info.plist, $i)) \
   $(foreach i, $2, $(addsuffix /Contents/Resources/DWARF/$i,
$(filter $i.%, $1))), \
   $1)
endef

And use it like this:

PROFILE_1_DEBUG_FILES := $(call expand-debuginfo,
$(PROFILE_1_DEBUG_FILES), $(PROFILE_1_LIBRARIES))

Filtering out jsig can be done on the parameters at the macro call.

For the conditional addprefix of OPENJDK_TARGET_CPU_LEGACY_LIB, I would
do something like:

ifeq (, $(findstring $(OPENJDK_TARGET_OS), windows macosx))
   LIB_SUBDIR := $(OPENJDK_TARGET_CPU_LEGACY_LIB)/
else
   LIB_SUBDIR :=
endif

And then always assign PROFILE_*_JRE_LIB_FILES with $(addprefix
$(LIB_SUBDIR), ...). The conditional on Windows for assigning to
...BIN_FILES or ...LIB_FILES I would leave in place since I don't think
a macro solution would make it easier to understand.

/Erik


On 2014-12-01 01:09, David Holmes wrote:

Main bug is 8038189 but that is a closed bug, the open backport issue
is:

https://bugs.openjdk.java.net/browse/JDK-8066200

Compact Profiles support was added in 8, but only for the Linux
platform. I've now generalized this support to all the other
platforms. This is an 8u only change, targetting 8u60, as soon as the
jdk8u/dev starts accepting things for 8u60.

The changes are not that extensive - mostly generalizing the lists and
accounting for different platforms putting files into different places
(jre/lib/ vs jre/lib vs jre/bin). It isn't necessary to produce
detailed per-platform lists as files that don't exist simply don't get
copied; but when files are obviously platform specific then I add them
under suitable guards.

The biggest complexity comes from the debuginfo files, and in
particular unzipped debuginfo files. This accounts for the bulk of the
changes in profiles-includes.txt, as we try to identify the set of
debug files that might exist for each library (and OSX is the main
complication due to the .dSYM directory because the existing rules
only copy files not directories). Suggestions for reducing the
duplicated patterns would be appreciated.

Platform specific contents were determined in conjunction with
examination of what Jigsaw is using in JDK 9 for the base module.

I tested all the main platforms (Windows, Linux, Solaris and OSX) and
with/without zipping of debuginfo files. Note that Windows builds will
not work with unzipped debuginfo files due to 8025936, but I checked
that multiple debug info files were expanded into the right set of
targets.

webrevs:

http://cr.openjdk.java.net/~dholmes/8038189/webrev.top/

make/Main.gmk:
 - Remove the os-check that constrained profiles to linux

http://cr.openjdk.java.net/~dholmes/8038189/webrev.jdk/

make/CreateJars.gmk:
 - Check for empty variables (Solaris doesn't like them)
 - Fix # # comments
 - Add explicit targets for the beanless classes with $ in them (the %
substitution doesn't work on all platforms when $ is also present)

make/Images.gmk
 - Don't filter out server VM from compact profiles

make/Import.gmk
 - Add Info.plist to the exported files (for unzipped debuginfo files
- this is a general fix not profiles specific so may warrant its
own CR)

make/Profiles.gmk
 - Remove linux-only comment

make/Tools.gmk
 - Fix tool definitions to use $(PATH_SEP) and quote cp entries

make/profile-includes.txt
 - Bulk of changes as described above

make/profile-rtjar-includes.txt
 - Additional packages that exist on OSX only (but don't need to be
conditionally added)
 - NOTE: if AIX or other platform add platform specific packages not
already included by an enclosing package, then they will also need to
be added

Thanks,
David

Re: [9] RFR (XS) 8066507: JPRT is not capable of running jtreg tests located jdk/test

[Tim Bell]

Hi Chris:

Please review the following fix to allow running jdk jtreg tests using 
JPRT -rtests. This is the same fix that his been in place in 
hotspot/test/Makefile for a while now, so I just copied from there to 
jdk/test/Makefile.


https://bugs.openjdk.java.net/browse/JDK-8066507
http://cr.openjdk.java.net/~cjplummer/8066507/webrev.00/


Looks good to me.

Tim

[9] RFR (XS) 8066508: JTReg tests timeout on slow devices when run using JPRT

[Chris Plummer]

Please review the following fix to address JPRT timeout issues when 
using -rtests to run hotspot JTReg tests on slow devices. The same logic 
has been in place for jdk/test/Makefile for a while now, so I just 
copied from there to hotspot/test/Makefile.


https://bugs.openjdk.java.net/browse/JDK-8066508
http://cr.openjdk.java.net/~cjplummer/8066508/webrev.00/

thanks,

Chris

[9] RFR (XS) 8066507: JPRT is not capable of running jtreg tests located jdk/test

[Chris Plummer]

Please review the following fix to allow running jdk jtreg tests using 
JPRT -rtests. This is the same fix that his been in place in 
hotspot/test/Makefile for a while now, so I just copied from there to 
jdk/test/Makefile.


https://bugs.openjdk.java.net/browse/JDK-8066507
http://cr.openjdk.java.net/~cjplummer/8066507/webrev.00/

thanks,

Chris

Re: Adding Microbenchmarks to the JDK forest/trees (JEP-230)

[Staffan Friberg]

On 12/03/2014 02:58 AM, Magnus Ihse Bursie wrote:

On 2014-12-02 23:45, Christian Thalinger wrote:
On Dec 2, 2014, at 2:40 PM, Jonathan Gibbons 
 wrote:


Staffan,

That seems to put it on the low end for reasonably being its own 
repo, if you wanted that, at least, as indicated by the numbers.

Do we really want more repositories?
As long as the number of repositories are around a dozen, one more or 
less does not really matter. But our model will probably not scale 
well with hundreds of repos (e.g. if someone would suggest that every 
module should reside in a separate repo).


My suggestion is that the microbenchmarks are put in the top-level 
repo, if only for the reason that it seems fully possible to split 
them out to a separate repo some time in the future if it grows too 
much, but it seems much more unlikely that it will ever be moved back 
into the top-level repo if we realized it was a stupid idea to put it 
in a separate repo.


/Magnus
I like this idea, and agree that shifting in the opposite direction is 
probably something that would be much more work than breaking it out if 
size becomes an issue further down the road.


When moving a directory to a new sub-repository, is there any concern 
about the diffs for that set of files still lingering in the top repo, 
or can those be moved as well?


Thanks,
Staffan

RFR(XS): 8066589: Make importing sa-jdi.jar optional on its existance

[Volker Simonis]

Hi,

could you please review this tiny change which fixes the build on
platforms which don't built the SA agent after the recent modualrity
integrations:

http://cr.openjdk.java.net/~simonis/webrevs/8066589
https://bugs.openjdk.java.net/browse/JDK-8066589

I've tested that the fix works on AIX but I havn't had a chance to build Zero.

@Xerxes: maybe you could check if my suggested fix also solves your
build problems. I'm also no sure if the "ifneq ($(JVM_VARIANT_ZERO),
true)" clause also catches the ZEROSHARK case (altough I think it
should). If not we would need yet another "ifneq
($(JVM_VARIANT_ZEROSHARK), true)"

Thanks,
Volker

Re: RFR [JEP 220] Modular Run-Time Images - compact profiles footprint

[Alan Bateman]

On 04/12/2014 16:38, Naoto Sato wrote:
Quick question. Why is jdk.localedata module included in the compact 
profiles? The data there used to be in lib/ext/localedata.jar and 
lib/ext/cldrdata.jar which weren't included in those compact profiles.


Naoto
The "profiles" make target on JDK 8 (and JDK 9 before yesterday) always 
included localedata.jar in the images. I would expect this would be 
pruned or removed before deploying to an embedded device of course, same 
thing for the security providers that get included by default.


We've now moved to world where the runtime images are created from a set 
of modules and this should give a lot of flexibility once. We're just 
not there yet with locale data so we have to temporarily link in the big 
jdk.localedata module so that the profiles images have all locales.


-Alan.

Re: RFR [JEP 220] Modular Run-Time Images

[Magnus Ihse Bursie]

On 2014-11-20 22:39, Chris Hegarty wrote:

This is a review request for the changes for JEP 220: Modular Run-Time Images 
[1].

There are a number of individuals responsible for these changes. Some, possibly 
not all, are explicitly listed in the 'To' section of this mail, and they will 
help address any comments arising from this review request.

The new run-time image structure is defined in JEP 220 [1], and can be seen as 
a result of building the staging forest [2], or in the early access builds [3].

Webrevs:
http://cr.openjdk.java.net/~chegar/8061971/00/


I have thoroughly reviewed all the build changes, and they look good to 
me, with the following remarks.


It would have been nice to use the new "wrapper" based approach for all 
module-phase specific details -- this is done now for all phases except 
gensrc and rmic. But this is not necessary for the patch to work, and 
can be addressed later on.


We also need to update configure to accept a module-based JDK as a boot 
JDK. In the patch, this is done sort-of -- we can run a bootcycle build 
with the new module-based JDK, but we will not accept it as an initial 
boot JDK. But this too can be fixed later on.


And of course there is always some syntax issues. :-) In 
langtools/make/gensrc/Gensrc-jdk.*.gmk, there are several calls of the 
pattern:

$(eval $(call SetupVersionProperties,JAVAP_VERSION, ...))
$(eval $(call SetupCompileProperties,COMPILE_PROPERTIES, ...))
These should have a space following the comma to adhere to the 
recommended style. But once again, I'm okay with fixing this in a follow up.


/Magnus

Re: RFR [JEP 220] Modular Run-Time Images - fix zero build

[Xerxes Rånby]

Hi Chris!

The build changes introduced a new dependency that sa-jdi.jar is always 
built.


The sa-jdi.jar do not get built  for Zero and Itanium builds.
Zero is built using the --with-jvm-interpreter=cpp 
--with-jvm-variants=zero configure options.


The sa-jdi.jar also do not exist if you use the --with-import-hotspot= 
to import alternative OpenJDK JVM such as

a pre-compiled CACAO JVM or JamVM libjvm.so .

I would suggest to change the jdk/make/Import.gmk and 
jdk/make/gensrc/Gensrc-jdk.jdi.gmk to first check if sa-jdi.jar
exist before adding it to the SA_TARGETS and GENSRC_JDK_JDI something 
like this:


Index: openjdk-jdk9-b38/jdk/make/Import.gmk
===
--- openjdk-jdk9-b38.orig/jdk/make/Import.gmk   2014-11-17 17:34:13.830175424 
+0100
+++ openjdk-jdk9-b38/jdk/make/Import.gmk2014-11-18 09:10:25.420715300 
+0100
@@ -221,6 +221,8 @@
 # even if zip is already unpacked.
 $(SUPPORT_OUTPUTDIR)/gensrc/jdk.hotspot.agent/_the.sa.services: 
$(HOTSPOT_DIST)/lib/sa-jdi.jar
 
+# sa-jdi.jar do not exist for Itanium and zero

+if [ -a $(HOTSPOT_DIST)/lib/sa-jdi.jar ] ; \
 SA_TARGETS += $(JDK_OUTPUTDIR)/modules/jdk.hotspot.agent/_the.sa.jar.unpacked \
 $(SUPPORT_OUTPUTDIR)/gensrc/jdk.hotspot.agent/_the.sa.services
 
Index: openjdk-jdk9-b38/jdk/make/gensrc/Gensrc-jdk.jdi.gmk

===
--- openjdk-jdk9-b38.orig/jdk/make/gensrc/Gensrc-jdk.jdi.gmk2014-11-18 
08:57:26.504852865 +0100
+++ openjdk-jdk9-b38/jdk/make/gensrc/Gensrc-jdk.jdi.gmk 2014-11-18 
09:10:43.348804201 +0100
@@ -78,12 +78,10 @@
 
$(SUPPORT_OUTPUTDIR)/gensrc/jdk.jdi/META-INF/services/com.sun.jdi.connect.Connector
$(install-file)
 
+# sa-jdi.jar do not exist for Itanium and zero

+if [ -a $(HOTSPOT_DIST)/lib/sa-jdi.jar ] ; \
 GENSRC_JDK_JDI += 
$(SUPPORT_OUTPUTDIR)/gensrc/jdk.jdi/META-INF/services/com.sun.jdi.connect.Connector
 \
 
$(JDK_OUTPUTDIR)/modules/jdk.hotspot.agent/META-INF/services/com.sun.jdi.connect.Connector

 
Cheers

Xerxes




From: Chris Hegarty 
Subject: RFR [JEP 220] Modular Run-Time Images
Date: 20 November 2014 21:39:14 GMT
To: jigsaw-dev , jdk9-dev , build-dev , Alan Bateman , 
Alex Buckley , Chris Hegarty , Erik Joelsson , Jonathan Gibbons 
, Karen Kinnear , "Jim Laskey (Oracle)" , Magnus Ihse Bursie 
, Mandy Chung , Mark Reinhold , Paul Sandoz , "A. 
Sundararajan" 

This is a review request for the changes for JEP 220: Modular Run-Time Images 
[1].

There are a number of individuals responsible for these changes. Some, possibly 
not all, are explicitly listed in the 'To' section of this mail, and they will 
help address any comments arising from this review request.

The new run-time image structure is defined in JEP 220 [1], and can be seen as 
a result of building the staging forest [2], or in the early access builds [3].

Webrevs:
   http://cr.openjdk.java.net/~chegar/8061971/00/

Due to the significant impact of these changes, a JDK 9 promotion has been 
tentatively reserved for their integration. All comments are welcome, although 
given the nature of the changes then we might have to create separate issues in 
JIRA to address some of them later in jdk9/dev.

-Chris.

[1] http://openjdk.java.net/jeps/220
[2] http://hg.openjdk.java.net/jigsaw/m2/
[3] http://openjdk.java.net/projects/jigsaw/ea

Re: RFR [JEP 220] Modular Run-Time Images - compact profiles footprint

[Naoto Sato]

Quick question. Why is jdk.localedata module included in the compact 
profiles? The data there used to be in lib/ext/localedata.jar and 
lib/ext/cldrdata.jar which weren't included in those compact profiles.


Naoto

On 12/4/14, 5:48 AM, Alan Bateman wrote:

On 04/12/2014 12:58, Xerxes Rånby wrote:

The footprint of the compact profiles have been inflated by ~12Mb each
after the JEP 220 changes.

before
# du -s -h j2re-compact1-image
17.9Mj2re-compact1-image
# du -s -h j2re-compact2-image
28.7Mj2re-compact2-image
# du -s -h j2re-compact3-image
35.5Mj2re-compact3-image

after
# du -s -h jre-compact1
31.3Mjre-compact1
# du -s -h jre-compact2
41.5Mjre-compact2
# du -s -h jre-compact3
47.4Mjre-compact3

The attached file compact1.diff.tar.gz contains the list diff of the
files bundled in the
j2re-compact1-image/lib/rt.jar compared to jre-compact1/lib/*.jimage
most of the class diff is located in

com/sun/security/ntlm
com/sun/crypto
javax/crypto
sun/net/www/protocol/http/ntlm
sun/net/www/protocol/ftp
sun/net/www/protocol/mailto
sun/net/ftp
sun/net/smtp
sun/net/dns
sun/util/resources <- a lot of extra internationalized classes
sun/security/ssl
sun/security/ec
sun/security/pkcs11
sun/text/resources <- a lot of extra internationalized classes


The release file in the top directory of the runtime image gives a good
indication as to what is going on. If you look at the value of the
MODULES key then you'll the see the modules that are actually linked in.
For jre-compact1 then you should see a line like this:

MODULES="java.base jdk.localedata java.scripting java.logging
java.compact1 jdk.crypto.ec jdk.crypto.pkcs11"

The java.* modules are the modules that make up compact1, the jdk.*
modules are additional service providers linked into the image. The
service providers aren't strictly required to be there, we've just
chosen to include them so that "profiles" make target gives us images
that approximately correspond to what we had previously. If you want to
play around with leaving them out then look in make/Images.gmk and
COMPACT_EXTRA_MODULES. Going forward then I expect we will have a tool
that will allow for a lot more flexibility to create images with just
the modules that you want (and their transitive dependences of course).

So I think the bulk of the difference that you are seeing is explained
by the service providers and mostly jdk.localedata. That module is big
and contains all of non-US_en JRE locale data and all of the CLDR data.
We still need to figure out what how to split this, you might recall the
discussion on i18n-dev and jigsaw-dev where they was some recent
discussion on this. It's also listed in JEP 200 as an open issue.

So when comparing to JDK 8 or previous JDK 9 compact profile builds then
think of the new images has having the equivalent of both localedata.jar
and cldrdata.jar present. If you edit COMPACT_EXTRA_MODULES to remove
jdk.localdata.jar then it should make it easier to compare.

Another thing to point out is that rt.jar isn't everything in the legacy
image. You need to take account of jce.jar, jsse.jar and
ext/sunjce_provider.jar. That should explain the javax.crypto,
com.sun.crypto and sun.security.ssl packages in your list.

Another thing to mention is the java.base module currently contains a
few legacy items that we previously stripped out of the profiles builds
in JDK 8. We still need to figure out what to do with these. The ftp and
smtp protocol handlers come to mind, also the NTLM htto authentication
scheme. At one point we have a "compat" module for legacy stuff that
people might want for compatibility reasons. So expect some tweaking
here, we know people focused on footprint will not thank us for bringing
back legacy stuff.

A few final point to mention. (a) there are a few additional launchers
(and maybe debug info/diz files if build with those) that were not there
previously. This is a consequence of modularization where modules with
launchers and where the classes for those launchers were previously in
tools.jar. (b) there are some additional classes that didn't previously
exist in JDK 9, this is to support the new image format. I expect some
churn with those over the next few months.

So hopefully this helps to explain what you are seeing. I think folks
might be interested in see j2re-image vs. jre and j2sdk-image vs. jdk
sizes too.

-Alan.

Re: RFR [JEP 220] Modular Run-Time Images - compact profiles footprint

[Alan Bateman]

On 04/12/2014 12:58, Xerxes Rånby wrote:
The footprint of the compact profiles have been inflated by ~12Mb each 
after the JEP 220 changes.


before
# du -s -h j2re-compact1-image
17.9Mj2re-compact1-image
# du -s -h j2re-compact2-image
28.7Mj2re-compact2-image
# du -s -h j2re-compact3-image
35.5Mj2re-compact3-image

after
# du -s -h jre-compact1
31.3Mjre-compact1
# du -s -h jre-compact2
41.5Mjre-compact2
# du -s -h jre-compact3
47.4Mjre-compact3

The attached file compact1.diff.tar.gz contains the list diff of the 
files bundled in the

j2re-compact1-image/lib/rt.jar compared to jre-compact1/lib/*.jimage
most of the class diff is located in

com/sun/security/ntlm
com/sun/crypto
javax/crypto
sun/net/www/protocol/http/ntlm
sun/net/www/protocol/ftp
sun/net/www/protocol/mailto
sun/net/ftp
sun/net/smtp
sun/net/dns
sun/util/resources <- a lot of extra internationalized classes
sun/security/ssl
sun/security/ec
sun/security/pkcs11
sun/text/resources <- a lot of extra internationalized classes


The release file in the top directory of the runtime image gives a good 
indication as to what is going on. If you look at the value of the 
MODULES key then you'll the see the modules that are actually linked in. 
For jre-compact1 then you should see a line like this:


MODULES="java.base jdk.localedata java.scripting java.logging 
java.compact1 jdk.crypto.ec jdk.crypto.pkcs11"


The java.* modules are the modules that make up compact1, the jdk.* 
modules are additional service providers linked into the image. The 
service providers aren't strictly required to be there, we've just 
chosen to include them so that "profiles" make target gives us images 
that approximately correspond to what we had previously. If you want to 
play around with leaving them out then look in make/Images.gmk and 
COMPACT_EXTRA_MODULES. Going forward then I expect we will have a tool 
that will allow for a lot more flexibility to create images with just 
the modules that you want (and their transitive dependences of course).


So I think the bulk of the difference that you are seeing is explained 
by the service providers and mostly jdk.localedata. That module is big 
and contains all of non-US_en JRE locale data and all of the CLDR data. 
We still need to figure out what how to split this, you might recall the 
discussion on i18n-dev and jigsaw-dev where they was some recent 
discussion on this. It's also listed in JEP 200 as an open issue.


So when comparing to JDK 8 or previous JDK 9 compact profile builds then 
think of the new images has having the equivalent of both localedata.jar 
and cldrdata.jar present. If you edit COMPACT_EXTRA_MODULES to remove 
jdk.localdata.jar then it should make it easier to compare.


Another thing to point out is that rt.jar isn't everything in the legacy 
image. You need to take account of jce.jar, jsse.jar and 
ext/sunjce_provider.jar. That should explain the javax.crypto, 
com.sun.crypto and sun.security.ssl packages in your list.


Another thing to mention is the java.base module currently contains a 
few legacy items that we previously stripped out of the profiles builds 
in JDK 8. We still need to figure out what to do with these. The ftp and 
smtp protocol handlers come to mind, also the NTLM htto authentication 
scheme. At one point we have a "compat" module for legacy stuff that 
people might want for compatibility reasons. So expect some tweaking 
here, we know people focused on footprint will not thank us for bringing 
back legacy stuff.


A few final point to mention. (a) there are a few additional launchers 
(and maybe debug info/diz files if build with those) that were not there 
previously. This is a consequence of modularization where modules with 
launchers and where the classes for those launchers were previously in 
tools.jar. (b) there are some additional classes that didn't previously 
exist in JDK 9, this is to support the new image format. I expect some 
churn with those over the next few months.


So hopefully this helps to explain what you are seeing. I think folks 
might be interested in see j2re-image vs. jre and j2sdk-image vs. jdk 
sizes too.


-Alan.

Re: [8u60] RFR: 8038189: Add cross-platform compact profiles support

[Erik Joelsson]

Looks good to me.

/Erik

On 2014-12-04 04:16, David Holmes wrote:

Updated webrev:

http://cr.openjdk.java.net/~dholmes/8038189/webrev.jdk.v2/

Only changes are to profile-includes.txt.

Thanks,
David

On 2/12/2014 2:24 PM, David Holmes wrote:

Erik,

Many thanks for the makefile macro wizardry! I will incorporate, test
and return with an updated webreb.

David

On 1/12/2014 7:19 PM, Erik Joelsson wrote:

Hello David,

Most of it looks good, but profile-includes.txt could certainly benefit
from some reduction in duplication. (Aside from the extension of the
file itself, which I find a bit weird, but it's already there.) The two
common structures that I can see are:
1. Expanding debuginfo files for macosx.
2. Assigning/expanding libraries to the correct variable.

For macosx debuginfo expanding, I would define a macro to something 
like

this:

# Expand the contents of the .dSYM directories on macosx.
# Param 1 - debug files list
# Param 2 - libraries list
define expand-debuginfo
   $(if $(and $(filter-out true, $(ZIP_DEBUGINFO_FILES)), $(filter
macosx, $(OPENJDK_TARGET_OS))), \
   $(foreach i, $1, $(addsuffix /Contents/Info.plist, $i)) \
   $(foreach i, $2, $(addsuffix /Contents/Resources/DWARF/$i,
$(filter $i.%, $1))), \
   $1)
endef

And use it like this:

PROFILE_1_DEBUG_FILES := $(call expand-debuginfo,
$(PROFILE_1_DEBUG_FILES), $(PROFILE_1_LIBRARIES))

Filtering out jsig can be done on the parameters at the macro call.

For the conditional addprefix of OPENJDK_TARGET_CPU_LEGACY_LIB, I would
do something like:

ifeq (, $(findstring $(OPENJDK_TARGET_OS), windows macosx))
   LIB_SUBDIR := $(OPENJDK_TARGET_CPU_LEGACY_LIB)/
else
   LIB_SUBDIR :=
endif

And then always assign PROFILE_*_JRE_LIB_FILES with $(addprefix
$(LIB_SUBDIR), ...). The conditional on Windows for assigning to
...BIN_FILES or ...LIB_FILES I would leave in place since I don't think
a macro solution would make it easier to understand.

/Erik


On 2014-12-01 01:09, David Holmes wrote:

Main bug is 8038189 but that is a closed bug, the open backport issue
is:

https://bugs.openjdk.java.net/browse/JDK-8066200

Compact Profiles support was added in 8, but only for the Linux
platform. I've now generalized this support to all the other
platforms. This is an 8u only change, targetting 8u60, as soon as the
jdk8u/dev starts accepting things for 8u60.

The changes are not that extensive - mostly generalizing the lists and
accounting for different platforms putting files into different places
(jre/lib/ vs jre/lib vs jre/bin). It isn't necessary to produce
detailed per-platform lists as files that don't exist simply don't get
copied; but when files are obviously platform specific then I add them
under suitable guards.

The biggest complexity comes from the debuginfo files, and in
particular unzipped debuginfo files. This accounts for the bulk of the
changes in profiles-includes.txt, as we try to identify the set of
debug files that might exist for each library (and OSX is the main
complication due to the .dSYM directory because the existing rules
only copy files not directories). Suggestions for reducing the
duplicated patterns would be appreciated.

Platform specific contents were determined in conjunction with
examination of what Jigsaw is using in JDK 9 for the base module.

I tested all the main platforms (Windows, Linux, Solaris and OSX) and
with/without zipping of debuginfo files. Note that Windows builds will
not work with unzipped debuginfo files due to 8025936, but I checked
that multiple debug info files were expanded into the right set of
targets.

webrevs:

http://cr.openjdk.java.net/~dholmes/8038189/webrev.top/

make/Main.gmk:
 - Remove the os-check that constrained profiles to linux

http://cr.openjdk.java.net/~dholmes/8038189/webrev.jdk/

make/CreateJars.gmk:
 - Check for empty variables (Solaris doesn't like them)
 - Fix # # comments
 - Add explicit targets for the beanless classes with $ in them (the %
substitution doesn't work on all platforms when $ is also present)

make/Images.gmk
 - Don't filter out server VM from compact profiles

make/Import.gmk
 - Add Info.plist to the exported files (for unzipped debuginfo files
- this is a general fix not profiles specific so may warrant its 
own CR)


make/Profiles.gmk
 - Remove linux-only comment

make/Tools.gmk
 - Fix tool definitions to use $(PATH_SEP) and quote cp entries

make/profile-includes.txt
 - Bulk of changes as described above

make/profile-rtjar-includes.txt
 - Additional packages that exist on OSX only (but don't need to be
conditionally added)
 - NOTE: if AIX or other platform add platform specific packages not
already included by an enclosing package, then they will also need to
be added

Thanks,
David

Re: Confirming JDK 8 GA version

[Erik Joelsson]

On 2014-12-03 21:54, Medi Montaseri wrote:

Thanks David and Erik,

A bit more confused...I don't know what hotspot is.
Hotspot is the jvm, basically libjvm.so (and a few of the support libs) 
and has historically been treated as a separate project from the rest of 
the jdk. Because of that it has its own version string. When you run 
java -version in jdk8, you see three lines. The last one that says "VM" 
belongs to Hotspot. It ignores the input from configure.

I am aware of configure --with-build-number and --with-milestone and indeed use 
themand that is why I get

bin/java -version
openjdk version "1.8.0-20141202"// note how this version is stamped 
with a date
OpenJDK Runtime Environment (build 1.8.0-20141202-HDS-174701) // note 
how this portion is also decorated with date, label and more numbers
OpenJDK 64-Bit Server VM (build 25.0-b70, mixed mode)
   /// what is 
this b70

The b70 makes me think that I am not working with the tip
Look at that line with "VM" and "b70" and compare it to the same line 
from jdk8 GA. They are the exact same.

Let me ask the question another way...

If I pull the latest and "hg tags" says b132 as in


 unix> cd srcDir/jdk ; hg tags | head -2
tip 9107:687fd7c7986d
jdk8-b132   9106:43cb25339b55


Can I then be sure that I am indeed working with the tip...?

AFAIK yes.


David...you see how yours is showing the b132


java version "1.8.0"
Java(TM) SE Runtime Environment (build 1.8.0-b132)   // look there is the 
b132 thing  -b132
Java HotSpot(TM) Server VM (build 25.0-b70, mixed mode)
Yes, this shows b132, in the line that is only defined by configure 
arguments, and that you are setting to something else.


/Erik

Again, I need to make sure I have the latest and be able to query some binary 
(java or javac) to show that it is the latest.

Thanks
Medi


From: Erik Joelsson [erik.joels...@oracle.com]
Sent: Wednesday, December 03, 2014 12:33 AM
To: David Holmes; Medi Montaseri; build-dev@openjdk.java.net
Subject: Re: Confirming JDK 8 GA version

The source/makefiles are not aware of the build number. It's assigned
from an external source at the time of building a promoted build. If you
want your version number to better match 8 GA, you will have to play
around with the configure arguments --with-build-number and possibly
--with-milestone.

Also, as David pointed out, in JDK 8, hotspot has it's own numbering.

/Erik

On 2014-12-03 06:13, David Holmes wrote:

On 3/12/2014 12:25 PM, Medi Montaseri wrote:

Hi,

Just wanted to confirm some issues:
I am interested in building the latest (tip) of jdk8 GA release.

1- jdk8 GA is different from jdk8u (u is for updates), I should get
the src from http://hg.openjdk.java.net/jdk8/jdk8
2- The latest tagged version of the src is
"jdk8-b132   9106:43cb25339b55"
3- I confirm that I have the latest src (for GA) by running "hg tags"
like
 unix> cd srcDir/jdk ; hg tags | head -2
tip 9107:687fd7c7986d
jdk8-b132   9106:43cb25339b55

I go ahead and build everything and then run "java -version" and I see

openjdk version "1.8.0-20141202"
OpenJDK Runtime Environment (build 1.8.0-20141202-HDS-174701)
OpenJDK 64-Bit Server VM (build 25.0-b70, mixed mode)

Q- Why am I seeing "build 25.0-b70" ? Does this "b70" has anything to
do with above "jdk8-b132" ?

That is the hotspot version number and build number as used in the GA
build of JDK 8. It is correct.

java version "1.8.0"
Java(TM) SE Runtime Environment (build 1.8.0-b132)
Java HotSpot(TM) Server VM (build 25.0-b70, mixed mode)

David
-


Thanks
Medi