Re: JDK8 Preliminary Repository Layout

[Kelly O'Hair]

On Apr 28, 2011, at 2:48 AM, Fredrik Öhrström wrote:

> 2011/3/11 Kelly O'Hair 
> in the repository. If there are frequent pushes going on, either from too 
> much activity or too many developers,
> someone may experience a:
>   hg push# fails because you need to do a pull "too many heads message"
>   hg pull -u && hg merge && hg commit -m Merge#  Or hg fetch
>   hg push   # fails because you took too long and someone else pushed a new 
> one
>   hg pull -u && hg merge && hg commit -m Merge#  Or hg fetch
>   hg push   # fails because you took too long and someone else pushed a new 
> one
> 
> I suppose this is related to the fact that mercurial has developed over time.
> But today, using merge to solve this problem would be ill advised.
> I think it is much better to use the rebase extension to do 
> hg pull --rebase
> 
> This will move your outgoing changes to the tip and avoid the creation of an 
> unnecessary merge node.

This rebase extension (which is an extension that needs to be enabled) has not 
always existed, it's a nice addition
and will be very useful. But  it does come with come cautions, effectively 
a rebase will regenerate
changesets, and anything that regenerates changesets needs to be used with 
great care, just like rollbacks.
And hidden in the rebase, is a merge, potentially many merges, potentially 
merges where file
changes actually need to be resolved. With rebase, the merges just get folded 
into a new changeset.

There are dumb merge changesets, where no files were in conflict, which the 
rebase can easily fix,
but when there are file conflicts, you may run a risk here of mis-merging if 
you are not careful.
And once rebased, your new changeset might not be the same changeset that you 
originally created,
so great care is always advised with a 'hg pull', regardless of how you manage 
the merge.
Depending on how much you pulled over, re-builds and re-tests may be important.

> 
> Assuming that you have a large number of committers at work at the same time, 
> a simple solution
> is to have the committers add themselves to a queue, then they get a message 
> (IM,mail or otherwise)
> when they have exclusive access. When they are done, they relinquish their 
> exclusive acces or
> it will be revoked automatically after 5 minutes.

In my opinion, a collision between 2 developers in a small window of time (say 
3mins) should be
rare, and it's easily resolved. If it happens frequently, I would tend to 
question what is going on.
That many changesets being pushed in at the same time may be a sign of 
something seriously wrong
with development, the number of changesets per day should not be in the 1,000's.

Yes, I see John mentioned the stuffed animal Teamware tokens, representing 
locks on workspaces, I remember those. ;^)
With the old Teamware workspaces, the equivalent pull/merge was often very very 
slow, 30mins
sometimes 60mins. But with Mercurial, the time to push is measured in minutes, 
sometimes seconds.
I just don't see a need for locks. More important in my mind, is making sure we 
get good changeset pushes.

-kto

> 
> Fredrik Öhrström

Re: JDK8 Preliminary Repository Layout

[John Rose]

On Apr 28, 2011, at 2:48 AM, Fredrik Öhrström wrote:

> I think it is much better to use the rebase extension to do 
> hg pull --rebase

Repo history is simplified if individual developers (a) delay commits and 
merges and (b) minimize private merges.

For my part, I use mq to manage my bug fixes in patch form, all the way up to 
the final push (or jprt job).  If the push fails for some reason, I reapply my 
patch and remerge.  There are no stacked merges.

In most cases, any push requires at most one merge.  Maybe jprt will throw in 
another merge, sometimes.

> Assuming that you have a large number of committers at work at the same time, 
> a simple solution
> is to have the committers add themselves to a queue, then they get a message 
> (IM,mail or otherwise)
> when they have exclusive access. When they are done, they relinquish their 
> exclusive acces or
> it will be revoked automatically after 5 minutes.

If there is a critical section, it has to be long enough to allow relevant 
pre-integration testing.  For hotspot development, this means a jprt run.

For small fan-in situations (like Kelly noted), optimistic concurrency works 
very comfortably, without lock negotiation.

(Does anyone remember the days of using stuffed animals for source control 
locks?  I wonder what is the internet version of that.  Probably what Fredrik 
said.)

-- John

Re: JDK8 Preliminary Repository Layout

[Fredrik Öhrström]

2011/3/11 Kelly O'Hair 

> in the repository. If there are frequent pushes going on, either from too
> much activity or too many developers,
> someone may experience a:
>   hg push# fails because you need to do a pull "too many heads message"
>   hg pull -u && hg merge && hg commit -m Merge#  Or hg fetch
>   hg push   # fails because you took too long and someone else pushed a new
> one
>   hg pull -u && hg merge && hg commit -m Merge#  Or hg fetch
>   hg push   # fails because you took too long and someone else pushed a new
> one
>

I suppose this is related to the fact that mercurial has developed over
time.
But today, using merge to solve this problem would be ill advised.
I think it is much better to use the rebase extension to do
hg pull --rebase

This will move your outgoing changes to the tip and avoid the creation of an
unnecessary merge node.

Assuming that you have a large number of committers at work at the same
time, a simple solution
is to have the committers add themselves to a queue, then they get a message
(IM,mail or otherwise)
when they have exclusive access. When they are done, they relinquish their
exclusive acces or
it will be revoked automatically after 5 minutes.

Fredrik Öhrström

Re: JDK8 Preliminary Repository Layout

[Mike Swingler]

On Mar 11, 2011, at 10:07 AM, Kelly O'Hair wrote:

> On Mar 11, 2011, at 2:11 AM, Steve Poole wrote:
> 
>> Kelly - can you explain for us newbies why you have separate repositories?  
>> I'm sure I can list any number of reasons but it would be good to get your 
>> view.   It may sound like a dumb question but it does help in these sort of 
>> discussions to know some of the history :-)
> 
> This is probably tainted, but I will try and provide hopefully a honest view, 
> with some humor thrown in. ;^)
> 
> Prior to Mercurial, we used the Sun product Teamware and we had separate 
> workspaces (what Teamware called
> a repository) for: control, hotspot, j2se, deploy, install, etc. (deploy and 
> install were Sun plugins & installers).
> So that set a pattern. Teamware basically managed SCCS files, so as a 
> workspace grew, it did not scale well,
> and Teamware relied on NFS access to share these files (80,000+ files, when 
> you count SCCS s.* and * files).
> So this separation initially, in my view, was done for developer productivity.
> I don't have any history on why the other workspaces existed as separate 
> workspaces, but I just assume
> it was for the same reasons as hotspot, nobody wanted to part of the big j2se 
> gorilla in the room,
> and having your own workspace created more of a separate silo for that team 
> to work in I suspect.
> The control workspace was a small batch of makefiles that built all the 
> workspaces, used by Release
> Engineering mostly.
> 
> Note that Teamware allowed for partial workspaces, since it was only managing 
> SCCS and individual
> file edits, you could trim a Teamware workspace down to just the directories 
> you were working in, and
> still sync and push with subset workspaces. This flexibility was taken 
> advantage of by the j2se team
> to minimize the NFS traffic and improve productivity too. Mercurial doesn't 
> allow for subset repositories.
> 
> The hotspot team found that their smaller 5,000 file workspace was easier to 
> deal with, and in fact
> the VM was a natural interface boundary, easy to isolate, controlled APIs, 
> pre-built VMs could be
> dropped into a JDK, testing/experiments were easy. Hotspot was also mostly 
> C++ and native code.
> Later, a "Hotspot Express" delivery model was possible so that the same 
> sources could be delivered
> to completely separate JDK releases.
> The hotspot developers were happy, well, as happy as a hotspot developer can 
> be I suppose ;^)
> (The Serviceability Agent or SA was developed by the hotspot team and was/is 
> very tightly integrated
> with hotspot, so it became part of hotspot, not the j2se).
> 
> The j2se workspace was much larger, maybe 35,000 source files, it initially 
> included all the sources from the
> corba, jaxp. jaxws, and langtools repositories that exist now.
> This j2se workspace was very hard to deal with and many of the sources were 
> copy&paste from other projects
> that weren't even managed by the JDK team, new deliveries created lost fix 
> situations and an unreliable state.
> The build process was complicated because part of the workspace had the javac 
> sources, which had to be built
> first, then that used to build the sources all over again.
> 
> So just prior to OpenJDK, or about then, we decided to try and split up the 
> j2se workspace to better manage our build
> and source importing issues. The corba, jaxp, and jaxws workspaces were 
> created and those files were pulled
> from the j2se workspace, as was the javac and "language tools" sources into a 
> langtools workspace.
> The j2se workspace was then renamed "jdk".
> 
> That gave us the workspaces: corba, jaxp, jaxws, langtools, jdk, hotspot, ...
> 
> These Teamware workspaces eventually became what you see today as the 
> openjdk7 Mercurial repositories,
> but we had to push some files down into smaller closed repositories: 
> src/closed, test/closed, and make/closed
> for jdk, and src/closed, test/closed, and build/closed for hotspot. The fact 
> that hotspot had managed sources
> in a build directory was a thorn in our sides for a while and it was 
> eventually removed along with build/closed.
> Makefile logic is pretty much 100% open right now.
> 
> I'm not sure that the open sourcing influenced this, but note that corba, 
> jaxp, jaxws, and langtools are pure
> open source, and 100% Java (except for one .c file in corba initially). 
> Managing pure open Java projects is a
> joy if you ask me. ;^)
> 
> For langtools, the team wanted this separate repository and lobbied hard for 
> it as a productivity aid and also to allow
> them to use the NetBeans IDE on just their sources (NetBeans and some IDEs 
> had a hard time swallowing the entire
> j2se sources), but they also needed to try and ship a separate javac product 
> somewhere, I forget the details.
> Maybe some work with some outside developers, Jonathan Gibbons would remember.
> I'm sure if you asked him, there is no way they would want back into a larger 
> r

Re: JDK8 Preliminary Repository Layout

[Kelly O'Hair]

On Mar 11, 2011, at 2:11 AM, Steve Poole wrote:

> Kelly - can you explain for us newbies why you have separate repositories?  
> I'm sure I can list any number of reasons but it would be good to get your 
> view.   It may sound like a dumb question but it does help in these sort of 
> discussions to know some of the history :-)
> 


This is probably tainted, but I will try and provide hopefully a honest view, 
with some humor thrown in. ;^)

Prior to Mercurial, we used the Sun product Teamware and we had separate 
workspaces (what Teamware called
a repository) for: control, hotspot, j2se, deploy, install, etc. (deploy and 
install were Sun plugins & installers).
So that set a pattern. Teamware basically managed SCCS files, so as a workspace 
grew, it did not scale well,
and Teamware relied on NFS access to share these files (80,000+ files, when you 
count SCCS s.* and * files).
So this separation initially, in my view, was done for developer productivity.
I don't have any history on why the other workspaces existed as separate 
workspaces, but I just assume
it was for the same reasons as hotspot, nobody wanted to part of the big j2se 
gorilla in the room,
and having your own workspace created more of a separate silo for that team to 
work in I suspect.
The control workspace was a small batch of makefiles that built all the 
workspaces, used by Release
Engineering mostly.

Note that Teamware allowed for partial workspaces, since it was only managing 
SCCS and individual
file edits, you could trim a Teamware workspace down to just the directories 
you were working in, and
still sync and push with subset workspaces. This flexibility was taken 
advantage of by the j2se team
to minimize the NFS traffic and improve productivity too. Mercurial doesn't 
allow for subset repositories.

The hotspot team found that their smaller 5,000 file workspace was easier to 
deal with, and in fact
the VM was a natural interface boundary, easy to isolate, controlled APIs, 
pre-built VMs could be
dropped into a JDK, testing/experiments were easy. Hotspot was also mostly C++ 
and native code.
Later, a "Hotspot Express" delivery model was possible so that the same sources 
could be delivered
to completely separate JDK releases.
The hotspot developers were happy, well, as happy as a hotspot developer can be 
I suppose ;^)
(The Serviceability Agent or SA was developed by the hotspot team and was/is 
very tightly integrated
with hotspot, so it became part of hotspot, not the j2se).

The j2se workspace was much larger, maybe 35,000 source files, it initially 
included all the sources from the
corba, jaxp. jaxws, and langtools repositories that exist now.
This j2se workspace was very hard to deal with and many of the sources were 
copy&paste from other projects
that weren't even managed by the JDK team, new deliveries created lost fix 
situations and an unreliable state.
The build process was complicated because part of the workspace had the javac 
sources, which had to be built
first, then that used to build the sources all over again.

So just prior to OpenJDK, or about then, we decided to try and split up the 
j2se workspace to better manage our build
and source importing issues. The corba, jaxp, and jaxws workspaces were created 
and those files were pulled
from the j2se workspace, as was the javac and "language tools" sources into a 
langtools workspace.
The j2se workspace was then renamed "jdk".

That gave us the workspaces: corba, jaxp, jaxws, langtools, jdk, hotspot, ...

These Teamware workspaces eventually became what you see today as the openjdk7 
Mercurial repositories,
but we had to push some files down into smaller closed repositories: 
src/closed, test/closed, and make/closed
for jdk, and src/closed, test/closed, and build/closed for hotspot. The fact 
that hotspot had managed sources
in a build directory was a thorn in our sides for a while and it was eventually 
removed along with build/closed.
Makefile logic is pretty much 100% open right now.

I'm not sure that the open sourcing influenced this, but note that corba, jaxp, 
jaxws, and langtools are pure
open source, and 100% Java (except for one .c file in corba initially). 
Managing pure open Java projects is a
joy if you ask me. ;^)

For langtools, the team wanted this separate repository and lobbied hard for it 
as a productivity aid and also to allow
them to use the NetBeans IDE on just their sources (NetBeans and some IDEs had 
a hard time swallowing the entire
j2se sources), but they also needed to try and ship a separate javac product 
somewhere, I forget the details.
Maybe some work with some outside developers, Jonathan Gibbons would remember.
I'm sure if you asked him, there is no way they would want back into a larger 
repository.

The corba sources haven't changed much since then, makefile changes and all 
native code has been removed.
Originally, we wanted an ant script for a faster build and to allow for 
NetBeans/IDE use as it became pure
Java. That ha

Re: JDK8 Preliminary Repository Layout

[Jonathan Gibbons]

Having jdk/test and jdk/closed/test will make it harder[*] to run all 
the jdk tests in a single run using jtreg directly.


-- Jon

[*] Brit-speak for "impossible".


On 03/08/2011 06:32 PM, Kelly O'Hair wrote:


First, if we talk about the mercurial forests, it has nothing to do 
with the Mercurial Forest Extension.
What we really have is a set of nested repositories, sometimes called 
our "forest" of repositories.


This email is just about the actual layout of the repositories for jdk8.

The initial thinking at this time is that the openjdk8 open forest 
will look very much like openjdk7:


   openjdk8/
corba/
jaxp/
jaxws/
jdk/
hotspot/
langtools/

7 repositories total. Cloned from the openjdk7 repos so we will have 
all the openjdk7 history in the openjdk8 repositories.


Just for discussion sake, not that you can see what is behind the 
closed curtains, we are considering changing
the closed overlay a little, from the current jdk7 (bold is a closed 
repo):


  jdk7/
corba/
*deploy*/
jaxp/
jaxws/
jdk/
  src*/closed*/
  test*/closed*
  make*/closed*/
hotspot/
  src*/closed*/
  test*/closed*
*install*/
langtools/
*pubs*/

To something a little simplier like:

  jdk8/
corba/
*deploy*/
jaxp/
jaxws/
jdk/
*closed*/{src,test,make}
hotspot/
*closed*/{src,test,make}
*install*/
langtools/
*pubs*/

The existence of these closed repos should not be a surprise, and it 
should have no impact on the openjdk itself.
We are just trying to consolidate and have fewer repositories. Just 
thought it might be of interest.


But back to the openjdk8 forest.

Other ideas were considered:
  * Folding jaxp/jaxws into the root or jdk8/jdk repo
  * Separating out the jdk demos from the jdk repo to a separate 
"demos" repository
  * Separating out the client (awt/swing/etc) code from the jdk repo 
into a separate repo

  * Updating the corba sources changing it to an ant build

None of this seemed urgent to do out of the gate, or delay getting a 
preliminary jdk8 layout defined.


I know there is some interest in pulling the actual jaxp/jaxws sources 
back into their repos, that will
be discussed separately, we have multiple issues with that, but I am 
well aware of the pains that the

source drop zip files have created.

As always, we would like to get comments, or additional ideas.

Separate topics:
  * Forest Extension and it's replacement
  * Mercurial server update to 1.8 or newer
  * Build&Test system [1]
  * Open bug tracking system [2]

-kto

[1] 
http://mail.openjdk.java.net/pipermail/build-dev/2011-February/004112.html
[2] 
http://mail.openjdk.java.net/pipermail/web-discuss/2011-March/000153.html

Re: JDK8 Preliminary Repository Layout

[Dr Andrew John Hughes]

On 09:14 Thu 10 Mar , Kelly O'Hair wrote:
> 
> On Mar 9, 2011, at 4:48 PM, Dr Andrew John Hughes wrote:
> 
> >> 
> >> Other ideas were considered:
> >>  * Folding jaxp/jaxws into the root or jdk8/jdk repo
> > 
> > Sounds good.  jdk8/jdk would make more sense as jaxws depends on some 
> > classes that are in the jdk
> > tree (com.sun.net.httpserver) and we could then get rid of the Ant junk.  
> > It would be good to have
> > the code there too so we can once again track changes.  The recent security 
> > issue was a nightmare
> > due to the unavailability of the source code.
> 
> Humm "Ant junk" ok...  I'm not in love with it either, but it does provide a 
> very fast build of simple pure java code.

I haven't seen much noticeable difference and one would have thought that 
loading just javac would be quicker than
loading javac+ant classes+xml file parsing.

If you're comparing with the JDK build, then it's a bad comparison as it's 
designed so that javac is invoked
many times, sometimes on just a couple of files.  As a result, HotSpot never 
really gets to kick in.  To compare
like-with-like, you'd need to build all the JDK java files with one javac 
invocation, which would also avoid
some classes being built against bootstrap classes.

> But having separated out jaxp/jaxws/corba from the rest of the jdk, we are 
> not considering folding it all
> back in. The consideration was moving the use of the drop bundle, not forking 
> the source again.
> The separation of these sources from the basic jdk has been a major benefit 
> to the developers and was also meant
> to ease the fork issues of this code, that is and has been maintained as 
> separate open source projects and
> delivered into multiple projects, licensed differently, and re-packaged in 
> different ways.
> The jdk is downstream on these sources.
> There will be a separate discussion on this but it will focus on improving 
> the source drop complications.
> 

So where are the upstreams?  What is in these tarballs?

The problem is that you can't track down bugs because we just move from one 
random snapshot to another
with no history.  We need more transparency in this area.

> > 
> >>  * Separating out the jdk demos from the jdk repo to a separate "demos" 
> >> repository
> > 
> > Makes sense.  Having an option to disable them would be a good first step.
> 
> A 'do not build demos' option would be a fine RFE. Maybe do some preliminary 
> surgery to at least
> isolate the sources that are strictly 'demo or sample' material.  IF someone 
> has the time. ;^)
> 
> > 
> >>  * Separating out the client (awt/swing/etc) code from the jdk repo into a 
> >> separate repo
> > 
> > Why would we want to do this?  IME, there are lots of interdependencies 
> > with the other code and
> > this would make the build a nightmare.
> 
> It was suggested and talked about a little, nobody stepped forward to 
> champion it.
> I assume it would be done to categorize the sources and make the developers 
> lives easier
> when focusing on their own area.
> 

While making the build harder.

> > 
> >>  * Updating the corba sources changing it to an ant build
> > 
> > Please, no!  These Ant builds are already a nightmare and an order of 
> > magniture harder to debug than
> > the Makefiles.  Why you want to require a build system that requires an 
> > entire JDK setup (possibly
> > bringing another into the mix beyond the bootstrap JDK) is beyond me.
> > 
> 
> Ok. I had no idea is was that bad. Well, yes, ant script logging and 
> debugging is a bit bad, and the syntax
> is a little ...ah... bizarre... don't want to insult any xml people... :^(  
> The pains with ant scripts seemed worth
> it with regards to the speed it can compile/zip/jar all in one VM process. 
> And of course the potential NetBeans
> IDE usage benefits, just ask the langtools development team.
> The current corba makefile build can take 10-15 minutes on some systems, and 
> I would estimate that an ant build could
> cut that in half or maybe a quarter of that time. Especially on Windows 
> systems which aren't exactly exec demons.
> 

I don't know what the Oracle setup is but I would guess (especially from the 
'/java' stuff and other defaults)
that there is one well-defined setup.  I also guess there's only ever Oracle 
JDKs in the mix.

Out in the real world, there are all manner of possible Ant setups and they 
could be using any JDK to run on.
At one point, I think Kaffe got pulled into an OpenJDK build I was doing due to 
Ant.  In short, it brings in
a heap of dependencies and setup variances that just using make (which is 
required anyway) doesn't.

I guess all these problems come down to the fact that everything is designed 
around known sets of developers
with largely homogenous requirements rather than being designed for the 
plethora of possibilities that could
arise due to this now being a Free Software code base.  The numerous build 
questions on the list suggest
people don't always get too

Re: JDK8 Preliminary Repository Layout

[Kelly O'Hair]

On Mar 15, 2011, at 2:27 AM, Steve Poole wrote:

> Primarily then the objective of having multiple repositories is to improve 
> developer productivity and component stability.
> 

And also facilitate the ability to distribute some components to multiple 
projects outside the jdk or multiple jdks.

> I don't know how much cross-repo changes go on.   It would seem that if that 
> is minimal then the next logical step would be to
> remove the source for the repos you're not working on and just have the 
> binaries instead.Do you see that as a worthwhile goal?

When doing partial builds, there is a ALT_JDK_IMPORT_PATH (import jdk) setting 
which is supposed to point to
a jdk built image (one with a bin, lib, include, jre, etc. layout). This should 
be a very recent jdk7 build
image, and is normally the last promoted jdk build (which changes weekly with 
each promotion).
That is where previously built components come from, see the 
jdk/make/java/redist/Makefile for details.

This ALT_JDK_IMPORT_PATH setting has taken on various names over the years, you 
will also see
ALT_HOTSPOT_* variables in the jdk repository makefiles.

So for partial builds, you need a ALT_BOOTDIR (boot jdk) and also a 
ALT_JDK_IMPORT_PATH
to get the components that you are not building.

For OpenJDK, once you have a complete forest build, you can save the 
j2sdk-image directory somewhere,
point ALT_JDK_IMPORT_PATH at that area, and you should be good to go for doing 
partial builds
in the hotspot or jdk repositories. The langtools, jaxp, jaxws, and corba 
repositories are fairly standalone
and don't result in a complete built jdk image when built by themselves.

See:
http://blogs.sun.com/kto/entry/anatomy_of_the_jdk_build
http://blogs.sun.com/jjg/entry/building_javac_for_jdk7

-kto

Re: JDK8 Preliminary Repository Layout

[David Holmes]

Steve Poole said the following on 03/15/11 21:11:

On 15/03/11 10:44, David Holmes wrote:

Steve Poole said the following on 03/15/11 19:27:
I don't know how much cross-repo changes go on.   It would seem that 
if that is minimal then the next logical step would be to
remove the source for the repos you're not working on and just have 
the binaries instead.Do you see that as a worthwhile goal?


I'm not quite sure what you mean by this but working on different 
repos we often do just use the "binaries" for other parts. For hotspot 
for example my build process was the create the new libjvm.so file and 
simply drop into a JDK. I never needed to build the JDK to do that. I 
imagine the JDK folks do something similar and just drop an updated 
rt.jar into an existing JDK, or an updated libjava.so etc. Sometimes 
when I'm working on JDK changes I'll just use Xbootclasspath to test 
things out with an existing JDK.


Do you have to still check out all the repos to build the libjvm.so ?
I'd like to get to the position of not having to checkout the hotspot 
repo to build the class libraries and vice versa.


No I only checkout hotspot. That's the advantage of the "forest" 
approach - you only need to check out the repos you want to work on, 
everything else can be replaced in the build process either as an 
integral part of that build process (ie setting an import path for 
something), or by only building a sub component (eg hotspot) and 
manually dropping it into an existing JDK image.


David

Re: JDK8 Preliminary Repository Layout

[Alan Bateman]

David Holmes wrote:


I'm not quite sure what you mean by this but working on different 
repos we often do just use the "binaries" for other parts. For hotspot 
for example my build process was the create the new libjvm.so file and 
simply drop into a JDK. I never needed to build the JDK to do that. I 
imagine the JDK folks do something similar and just drop an updated 
rt.jar into an existing JDK, or an updated libjava.so etc. Sometimes 
when I'm working on JDK changes I'll just use Xbootclasspath to test 
things out with an existing JDK.
On the JDK side of the house then I think most people just build the 
"jdk" repository. This means they are doing a partial build with hotspot 
and the other components (jax* etc.) being imported by the build from 
the "import JDK", usually the last promoted build. So this is making use 
of pre-built binaries to avoid needing to build the world. Another thing 
is that I think most people just build the jdk "all" target and so don't 
build the complete image (no rt.jar etc.). Coupled with incremental 
building means you can work in a specific area with really good  
re-build times.


Steve wondered about "cross-repo" changes. There are some but probably 
only a small percentage of the changes requiring coordinated changes. 
The langtools area will periodically need to push changes to both 
langtools and the jdk repositories. Usually not a major issue becuase 
the changes go into the same forest. The really one is the 
serviceability area where they are changes to both the hotspot and the 
jdk repositories taking different routes into the master (although that 
is probably more of a process issue than a repository layout issue).


-Alan

Re: JDK8 Preliminary Repository Layout

[Steve Poole]

On 15/03/11 10:44, David Holmes wrote:

Steve,

Steve Poole said the following on 03/15/11 19:27:
I don't know how much cross-repo changes go on.   It would seem that 
if that is minimal then the next logical step would be to
remove the source for the repos you're not working on and just have 
the binaries instead.Do you see that as a worthwhile goal?


I'm not quite sure what you mean by this but working on different 
repos we often do just use the "binaries" for other parts. For hotspot 
for example my build process was the create the new libjvm.so file and 
simply drop into a JDK. I never needed to build the JDK to do that. I 
imagine the JDK folks do something similar and just drop an updated 
rt.jar into an existing JDK, or an updated libjava.so etc. Sometimes 
when I'm working on JDK changes I'll just use Xbootclasspath to test 
things out with an existing JDK.


Do you have to still check out all the repos to build the libjvm.so ?
I'd like to get to the position of not having to checkout the hotspot 
repo to build the class libraries and vice versa.



David Holmes

Re: JDK8 Preliminary Repository Layout

[David Holmes]

Steve,

Steve Poole said the following on 03/15/11 19:27:
I don't know how much cross-repo changes go on.   It would seem that if 
that is minimal then the next logical step would be to
remove the source for the repos you're not working on and just have the 
binaries instead.Do you see that as a worthwhile goal?


I'm not quite sure what you mean by this but working on different repos 
we often do just use the "binaries" for other parts. For hotspot for 
example my build process was the create the new libjvm.so file and 
simply drop into a JDK. I never needed to build the JDK to do that. I 
imagine the JDK folks do something similar and just drop an updated 
rt.jar into an existing JDK, or an updated libjava.so etc. Sometimes 
when I'm working on JDK changes I'll just use Xbootclasspath to test 
things out with an existing JDK.


David Holmes

Re: JDK8 Preliminary Repository Layout

[Steve Poole]

On 11/03/11 18:07, Kelly O'Hair wrote:

On Mar 11, 2011, at 2:11 AM, Steve Poole wrote:


Kelly - can you explain for us newbies why you have separate repositories?  I'm 
sure I can list any number of reasons but it would be good to get your view.   
It may sound like a dumb question but it does help in these sort of discussions 
to know some of the history :-)



This is probably tainted, but I will try and provide hopefully a honest view, 
with some humor thrown in. ;^)

Prior to Mercurial, we used the Sun product Teamware and we had separate 
workspaces (what Teamware called
a repository) for: control, hotspot, j2se, deploy, install, etc. (deploy and 
install were Sun plugins&  installers).
So that set a pattern. Teamware basically managed SCCS files, so as a workspace 
grew, it did not scale well,
and Teamware relied on NFS access to share these files (80,000+ files, when you 
count SCCS s.* and * files).
So this separation initially, in my view, was done for developer productivity.
I don't have any history on why the other workspaces existed as separate 
workspaces, but I just assume
it was for the same reasons as hotspot, nobody wanted to part of the big j2se 
gorilla in the room,
and having your own workspace created more of a separate silo for that team to 
work in I suspect.
The control workspace was a small batch of makefiles that built all the 
workspaces, used by Release
Engineering mostly.

Note that Teamware allowed for partial workspaces, since it was only managing 
SCCS and individual
file edits, you could trim a Teamware workspace down to just the directories 
you were working in, and
still sync and push with subset workspaces. This flexibility was taken 
advantage of by the j2se team
to minimize the NFS traffic and improve productivity too. Mercurial doesn't 
allow for subset repositories.

The hotspot team found that their smaller 5,000 file workspace was easier to 
deal with, and in fact
the VM was a natural interface boundary, easy to isolate, controlled APIs, 
pre-built VMs could be
dropped into a JDK, testing/experiments were easy. Hotspot was also mostly C++ 
and native code.
Later, a "Hotspot Express" delivery model was possible so that the same sources 
could be delivered
to completely separate JDK releases.
The hotspot developers were happy, well, as happy as a hotspot developer can be 
I suppose ;^)
(The Serviceability Agent or SA was developed by the hotspot team and was/is 
very tightly integrated
with hotspot, so it became part of hotspot, not the j2se).

The j2se workspace was much larger, maybe 35,000 source files, it initially 
included all the sources from the
corba, jaxp. jaxws, and langtools repositories that exist now.
This j2se workspace was very hard to deal with and many of the sources were 
copy&paste from other projects
that weren't even managed by the JDK team, new deliveries created lost fix 
situations and an unreliable state.
The build process was complicated because part of the workspace had the javac 
sources, which had to be built
first, then that used to build the sources all over again.

So just prior to OpenJDK, or about then, we decided to try and split up the 
j2se workspace to better manage our build
and source importing issues. The corba, jaxp, and jaxws workspaces were created 
and those files were pulled
from the j2se workspace, as was the javac and "language tools" sources into a 
langtools workspace.
The j2se workspace was then renamed "jdk".

That gave us the workspaces: corba, jaxp, jaxws, langtools, jdk, hotspot, ...

These Teamware workspaces eventually became what you see today as the openjdk7 
Mercurial repositories,
but we had to push some files down into smaller closed repositories: 
src/closed, test/closed, and make/closed
for jdk, and src/closed, test/closed, and build/closed for hotspot. The fact 
that hotspot had managed sources
in a build directory was a thorn in our sides for a while and it was eventually 
removed along with build/closed.
Makefile logic is pretty much 100% open right now.

I'm not sure that the open sourcing influenced this, but note that corba, jaxp, 
jaxws, and langtools are pure
open source, and 100% Java (except for one .c file in corba initially). 
Managing pure open Java projects is a
joy if you ask me. ;^)

For langtools, the team wanted this separate repository and lobbied hard for it 
as a productivity aid and also to allow
them to use the NetBeans IDE on just their sources (NetBeans and some IDEs had 
a hard time swallowing the entire
j2se sources), but they also needed to try and ship a separate javac product 
somewhere, I forget the details.
Maybe some work with some outside developers, Jonathan Gibbons would remember.
I'm sure if you asked him, there is no way they would want back into a larger 
repository.

The corba sources haven't changed much since then, makefile changes and all 
native code has been removed.
Originally, we wanted an ant script for a faster build and to allow for 
NetBeans/IDE use 

Re: JDK8 Preliminary Repository Layout

[Kelly O'Hair]

On Mar 11, 2011, at 1:51 AM, Alan Bateman wrote:

> Kelly O'Hair wrote:
>> 
>> I don't think that is the reason for the 1.4. But someone closer to this SA 
>> project would need to clarify this.
>> 
>> The reason was so that the SA could potentially operate on an older JDK as I 
>> recall.
>> I suspect 1.4 settings should have been changed to 1.5 in 1.6, and maybe 1.6 
>> in 1.7.
> Sundar or Swamy might remember, but I'm pretty sure that SA was deliberately 
> not updated to use generics in the jdk5 time-frame because the code was being 
> used to get SA working with 1.4.2. Ancient history now and a nice little 
> project for someone to do some clean-up. In any case, it's not good to 
> compile with -source 1.4 without also setting -bootclasspath to specify the 
> corresponding platform classes so another reason to re-visit this.

Yes now I vaguely recall this. This needs a CR filed. The 1.4.2 and even 1.5 
use is something that needs to be re-visited.

I have also thought that these java sources should be compiled by the langtools 
boot javac.jar, whether
they use -source or not, all the java code and generated class files should be 
going through the latest javac
in the jdk being built, when doing a full forest build.

-kto

> 
> -Alan

RE: JDK8 Preliminary Repository Layout

[Salter, Thomas A]

I want to cast my vote for the current JDK7 layout.  My effort with the source 
is exclusively to port it to a proprietary OS.  I tend to work off the bundled 
zip files, not Hg, so I'm concerned the lines between the forests.  The 
advantage to me of having CORBA, JAXWS and JAXP be distinct, identifiable trees 
is that that code never needs porting - there's no native C code and no 
platform-specific Java code.  I build them once and they just work.  Hotspot 
and JDK however are littered with native code and platform-dependent code, all 
of which I need to at least examine to see if it needs porting.

Tom Salter

Re: JDK8 Preliminary Repository Layout

[Jonathan Gibbons]

On 03/11/2011 04:22 AM, Johan Walles wrote:

2011-03-10 20:17, John Coomes skrev:

Johan Walles (johan.wal...@oracle.com) wrote:

The problem for many developers with the all-in-one repository solution
is the time it takes to clone everything (5-6 minutes).


That's the best case, when I'm on the corporate network, close to the
server.  Over my dsl line at home, it takes 24 minutes for everything,
vs. 1 min 10 secs for hotspot (the only part I need).


John, how would you feel about this situation?

* You are forced to clone everything.

* Cloning everything takes under a minute on your current (slow) DSL 
connection.


* You can still build and work with just your favorite component just 
like you can today.


  Regards //Johan


(Different Jon here)

I would be unhappy with proposals that force me to clone everything, 
since I typically have many copies of the langtools repo in flight, in 
various stages of development.  It would be excessive if all of these 
were full forests.  I've tried using "mq" and it doesn't work well for 
me -- in particular, it's a pain to switch between patches and have to 
rebuild all the corresponding binaries every time you switch.


-- Jon

Re: JDK8 Preliminary Repository Layout

[Johan Walles]

2011-03-10 20:17, John Coomes skrev:

Johan Walles (johan.wal...@oracle.com) wrote:

The problem for many developers with the all-in-one repository solution
is the time it takes to clone everything (5-6 minutes).


That's the best case, when I'm on the corporate network, close to the
server.  Over my dsl line at home, it takes 24 minutes for everything,
vs. 1 min 10 secs for hotspot (the only part I need).


John, how would you feel about this situation?

* You are forced to clone everything.

* Cloning everything takes under a minute on your current (slow) DSL 
connection.


* You can still build and work with just your favorite component just 
like you can today.


  Regards //Johan

Re: JDK8 Preliminary Repository Layout

[Steve Poole]

On 09/03/11 02:32, Kelly O'Hair wrote:


First, if we talk about the mercurial forests, it has nothing to do 
with the Mercurial Forest Extension.
What we really have is a set of nested repositories, sometimes called 
our "forest" of repositories.


This email is just about the actual layout of the repositories for jdk8.

The initial thinking at this time is that the openjdk8 open forest 
will look very much like openjdk7:


   openjdk8/
corba/
jaxp/
jaxws/
jdk/
hotspot/
langtools/

7 repositories total. Cloned from the openjdk7 repos so we will have 
all the openjdk7 history in the openjdk8 repositories.


Just for discussion sake, not that you can see what is behind the 
closed curtains, we are considering changing
the closed overlay a little, from the current jdk7 (bold is a closed 
repo):


  jdk7/
corba/
*deploy*/
jaxp/
jaxws/
jdk/
  src*/closed*/
  test*/closed*
  make*/closed*/
hotspot/
  src*/closed*/
  test*/closed*
*install*/
langtools/
*pubs*/

To something a little simplier like:

  jdk8/
corba/
*deploy*/
jaxp/
jaxws/
jdk/
*closed*/{src,test,make}
hotspot/
*closed*/{src,test,make}
*install*/
langtools/
*pubs*/

The existence of these closed repos should not be a surprise, and it 
should have no impact on the openjdk itself.
We are just trying to consolidate and have fewer repositories. Just 
thought it might be of interest.


But back to the openjdk8 forest.

Other ideas were considered:
  * Folding jaxp/jaxws into the root or jdk8/jdk repo
  * Separating out the jdk demos from the jdk repo to a separate 
"demos" repository
  * Separating out the client (awt/swing/etc) code from the jdk repo 
into a separate repo

  * Updating the corba sources changing it to an ant build

None of this seemed urgent to do out of the gate, or delay getting a 
preliminary jdk8 layout defined.


I know there is some interest in pulling the actual jaxp/jaxws sources 
back into their repos, that will
be discussed separately, we have multiple issues with that, but I am 
well aware of the pains that the

source drop zip files have created.

As always, we would like to get comments, or additional ideas.


Kelly - can you explain for us newbies why you have separate 
repositories?  I'm sure I can list any number of reasons but it would be 
good to get your view.   It may sound like a dumb question but it does 
help in these sort of discussions to know some of the history :-)





Separate topics:
  * Forest Extension and it's replacement
  * Mercurial server update to 1.8 or newer
  * Build&Test system [1]
  * Open bug tracking system [2]

-kto

[1] 
http://mail.openjdk.java.net/pipermail/build-dev/2011-February/004112.html
[2] 
http://mail.openjdk.java.net/pipermail/web-discuss/2011-March/000153.html

Re: JDK8 Preliminary Repository Layout

[Alan Bateman]

Kelly O'Hair wrote:


I don't think that is the reason for the 1.4. But someone closer to 
this SA project would need to clarify this.


The reason was so that the SA could potentially operate on an older 
JDK as I recall.
I suspect 1.4 settings should have been changed to 1.5 in 1.6, and 
maybe 1.6 in 1.7.
Sundar or Swamy might remember, but I'm pretty sure that SA was 
deliberately not updated to use generics in the jdk5 time-frame because 
the code was being used to get SA working with 1.4.2. Ancient history 
now and a nice little project for someone to do some clean-up. In any 
case, it's not good to compile with -source 1.4 without also setting 
-bootclasspath to specify the corresponding platform classes so another 
reason to re-visit this.


-Alan

Re: JDK8 Preliminary Repository Layout

[Johan Walles]

2011-03-10 20:17, John Coomes skrev:

Johan Walles (johan.wal...@oracle.com) wrote:

2011-03-10 10:33, Anthony Petrov skrev:

Hi Andrew,

On 3/10/2011 3:48 AM, Dr Andrew John Hughes wrote:

[snip]

Hey, I'd just make it all one repository as they all interdepend on
each other


One huge "all-in-all" repository is great for integrators, porters, or
maintainers, but it isn't that convenient for developers working on
specific areas of JDK. Primarily because of the unnecessarily increased
build time.


Build time wouldn't be an issue if you could build a single component by
itself by just cd:ing into the component directory and do "make" there.


So I pull the entire tree, make some changes, do a build.


Right, so now you've built Hotspot (let's say that's your component).


I want to
sync with the latest changes in my component (whatever it is), but
have to pull everything.


OK, now you have the latest and greatest sources for all components, 
including Hotspot.



To save build time, I build just my
component.


Your Hotspot binary is now up to date with the source changes you pulled in.


But then my built bits no longer correspond to my source
tree.  Debugging becomes a new challenge.


I think I missed something earlier, but why would the Hotspot binary you 
just built not correspond to the sources you just built it from?


  Regards /Johan

SA and JDK ( was Re: JDK8 Preliminary Repository Layout)

[David Holmes]

Andrew,

Just picking up on your SA comments and adding in the servicability folk 
and bcc'ing the build folk ...


Dr Andrew John Hughes said the following on 03/11/11 10:46:

Well HotSpot is one thing I think works well as a separate repository.
It allows us to have a stable branch for OpenJDK6 for example.  The only
real thing that stops it being independent is the servicability agent,
which, IME, would be better held in the JDK.  That would also mean that
the existing rules for building java classes could be used, rather than
reinventing it all in the HotSpot makefiles as at present.


The SA is part of Hotspot because the SA is a Java interface to the 
Hotspot internals. Hence the SA Java files have to be updated when the 
hotspot source files are, hence they must be in the same repo or we'd 
have huge coordination problems. These files reside in sa-jdi.jar


The SA classes are dependent on the com.sun.jdi interfaces/classes, so 
yes changes to JDI might require changes to the SA. But the SA is just a 
JDI client in that regard.



I've already run across one place where this is true.  HotSpot builds
the servicability agent with -source 1.4 -target 1.4.  Why? Because
of incompatibilies between its implementation and the com.sun.jdi
interfaces which require Comparable not Comparable.


My understanding is that the SA classes define their own Enum type which 
conflicts with the java.lang.Enum added in 1.5. To avoid all the 
warnings this would generate, plus the raw type warnings as the SA is 
not generified, it is compiled with -source 1.4.


David

Re: JDK8 Preliminary Repository Layout

[Kelly O'Hair]

On Mar 10, 2011, at 4:46 PM, Dr Andrew John Hughes wrote:

> I've already run across one place where this is true.  HotSpot builds
> the servicability agent with -source 1.4 -target 1.4.  Why? Because
> of incompatibilies between its implementation and the com.sun.jdi
> interfaces which require Comparable not Comparable.

I don't think that is the reason for the 1.4. But someone closer to this SA 
project would need to clarify this.

The reason was so that the SA could potentially operate on an older JDK as I 
recall.
I suspect 1.4 settings should have been changed to 1.5 in 1.6, and maybe 1.6 in 
1.7.

-kto

Re: JDK8 Preliminary Repository Layout

[Dr Andrew John Hughes]

On 19:27 Wed 09 Mar , Phil Race wrote:
> Andrew,
> 
> Whilst almost everything you wrote is something I agree with (like getting
> jcheck out there, not adding additional build tools/complexity), the one
> thing I quite like right now comes up here I'd like to keep is the 
> separate repos.
> Its not just hotspot,  but I've been working on JDK for a long time now,
> since before there was a hotspot and last year I built hotspot for the 
> first time,
> and even then only because I was working on the VS2010 compiler upgrade.
> 
> So I like that hotspot and jaxp/jaxws can be pre-built and pulled from an
> import JDK in a way that doesn't slow down my builds, bloat the source
> I need to pull via hg etc etc.
> 

Well HotSpot is one thing I think works well as a separate repository.
It allows us to have a stable branch for OpenJDK6 for example.  The only
real thing that stops it being independent is the servicability agent,
which, IME, would be better held in the JDK.  That would also mean that
the existing rules for building java classes could be used, rather than
reinventing it all in the HotSpot makefiles as at present.

Likewise, I think langtools works as a separate entity.  The other three
(corba, jaxp, jaxws) only really exist separately because no-one on the
OpenJDK project takes care of them.  JAXP and JAXWS get irregular build
drops, but I don't think I've seen anything more than build fixes for
CORBA.  They would be less work if maintaining in the JDK directory
as they'd share the build infrastructure.  They are hardly huge (unlike
HotSpot vs. JDK) and there are dependencies between the JDK and these.

I probably should provide some context.  During this week, I've been
looking at getting OpenJDK bootstrapping well when OpenJDK is not the
bootstrap JDK (i.e. sun.* and com.sun.* classes are not available).
To cut a long story short, it's a week later and I'm still only at
HotSpot.  So far, I've found dependencies in all but JAXP that mean
that the build has to be patched to reference other repositories
(mainly JDK but also langtools in one case).  JAXWS has to be able to
find com.sun.net.httpserver, annotation/javadoc classes and the JAXP
classes, while CORBA needs the sun.tools stuff and HotSpot needs
com.sun.jdi for the servicability agent.

Now you may think this isn't relevant and expect people to just use
OpenJDK to build.  Well, this also becomes a problem if one of these
APIs changes.  You're building stuff in CORBA, JAXWS and HotSpot against
a different API than the one they will eventually run against, hiding
issues which could be exposed at build time until run time.  

I've already run across one place where this is true.  HotSpot builds
the servicability agent with -source 1.4 -target 1.4.  Why? Because
of incompatibilies between its implementation and the com.sun.jdi
interfaces which require Comparable not Comparable.

> Maybe this doesn't matter to people who want to pull a full forest but
> its great for those working on specific components. Which I believe is
> most developers.
> 

It's hard to generalise, especially with an open source project. These
mailing lists show that there will be quite a few newbies who follow
the build guides and try downloading the entire forest.  As was mentioned
elsewhere in this discussion, the tarballs would be better for this
scenario as you don't really need the repos unless you're going to create
a patch (and even then, there are ways to do that without them).

My concern isn't the pull, but simply what makes best sense in terms
of maintenance, both of the code and the build system.  Having independent
repositories also means duplicate build code and cross-repo dependencies.
With HotSpot and langtools it makes sense as you have independent entities
(the VM and javac et. al.).  I don't think it makes sense with code that
has to then be imported into the JDK build.

> -phil.
> 
> 
> On 3/9/2011 4:48 PM, Dr Andrew John Hughes wrote:
> > On 18:32 Tue 08 Mar , Kelly O'Hair wrote:
> >> First, if we talk about the mercurial forests, it has nothing to do with 
> >> the Mercurial Forest Extension.
> >> What we really have is a set of nested repositories, sometimes called our 
> >> "forest" of repositories.
> >>
> >> This email is just about the actual layout of the repositories for jdk8.
> >>
> >> The initial thinking at this time is that the openjdk8 open forest will 
> >> look very much like openjdk7:
> >>
> >> openjdk8/
> >>  corba/
> >>  jaxp/
> >>  jaxws/
> >>  jdk/
> >>  hotspot/
> >>  langtools/
> >>
> >> 7 repositories total. Cloned from the openjdk7 repos so we will have all 
> >> the openjdk7 history in the openjdk8 repositories.
> >>
> >> Just for discussion sake, not that you can see what is behind the closed 
> >> curtains, we are considering changing
> >> the closed overlay a little, from the current jdk7 (bold is a closed repo):
> >>
> >>jdk7/
> >>  corba/
> >>  deploy/
> >>  jaxp/
> >>  j

Re: JDK8 Preliminary Repository Layout

[Kelly O'Hair]

On Mar 10, 2011, at 3:10 PM, David Holmes wrote:

> Kelly O'Hair said the following on 03/11/11 09:00:
>> On Mar 10, 2011, at 2:50 PM, David Holmes wrote:
>>> Lana Steuck said the following on 03/11/11 07:03:
 On 03/10/2011 08:46 AM, Kelly O'Hair wrote:
> Interesting point, we will need to decide which projects need jdk8 
> forests. I imagine some will not, and we may be
> doing a little trimming down on the number of team forests.
 Makes sense. There are some team forests (at least one that I know of) 
 that are not being used, so we can look into removing those and maybe 
 combining those team forests which have more interdependencies into one 
 forest (provided that the team members agree).
>>> I've never been a fan of the "all projects are complete forests". It just 
>>> increases the conceptual complexity of the repositories and the workload of 
>>> keeping all the forests sync'ed up. It should be easy to add additional 
>>> repositories to a project as the need arises, without requiring all 
>>> repositories.
>>> 
>> But all too often, regressions in one part of the forest only manifest 
>> themselves when building the entire forest.
>> So it is the responsibility of the Developer and Integrator to insure the 
>> entire forest builds and works prior
>> to integrations into the 'master' (jdk7/jdk7) forest.
>> Integrators are required to sync up with master, test build, and integrate 
>> the entire forest, not partials.
> 
> Developers (other than those working on the build system) rarely build 
> forests. Hotspot engineers build hotspot, T&L engineers build jdk etc. The 
> team integrators certainly have to combine with the "master" forest at the 
> next level. But keeping a bunch of unmodified team repos up to date with the 
> "master" is just pure overhead.

Completely disagree. They may be unmodified by one team, but someone IS 
modifying them, they are ALL
very much moving targets.
But if the Integrators wanted to do it differently, with the same reliability, 
they certainly have that option.
I have always assumed these full forests existed for the sanity of the 
Integrators and the protection of the
master always being able to build after an integration.
So the Integrator having a place to keep a full forest on the server, that they 
controlled when it got sync'd
up with master, and a place to point at and tell a team member, "this forest 
does not build after your change was pushed"
seemed like a benefit to the Integrator.

Not many people are "Integrators", it's a a bit of a thankless task sometimes, 
be nice to your Integrators! ;^)

> 
> Maybe hotspot needs a full forest (I don't think so), but certainly 
> hotspot_rt, hotspot_comp and hotspot_gc do not, in my opinion.

The sub-team repos  hotspot_rt, hotspot_comp and hotspot_gc aren't a major 
concern to me, they never
integrate directly into the master. If you want these to be partial, I'd talk 
to the person on your team that
can change that.

> 
>> By having that full forest available for the team, you can at least be sure 
>> that anyone on the team doing full
>> forest builds is dealing with the same sources.
>> Developers can choose to only use partials, that is certainly always 
>> possible.
> 
> If a team operates in such a way that full forest builds are needed then the 
> team will use a full forest. Most of the teams do not need that. Having the 
> forest just creates work for the gatekeeper.

If by "gatekeeper" you mean the person that pushes your sub-team repos to the 
team repos, hey, that's
your team's decision.

---
And there have been several times in the past, granted not many, where jdk and 
hotspot changes needed to
made by a hotspot developer. Ask the serviceability team. It's rare, but having 
them there does have value
to some, rather than a moving target, they can rely on a state that is defined 
by their team.

-kto

> 
> I said all this when the forest of forests was first devised. As I said never 
> been a fan, don't see the need for it.

> Cheers,
> David
> 
>> -kto

Re: JDK8 Preliminary Repository Layout

[David Holmes]

Kelly O'Hair said the following on 03/11/11 09:00:

On Mar 10, 2011, at 2:50 PM, David Holmes wrote:


Lana Steuck said the following on 03/11/11 07:03:

On 03/10/2011 08:46 AM, Kelly O'Hair wrote:

Interesting point, we will need to decide which projects need jdk8 forests. I 
imagine some will not, and we may be
doing a little trimming down on the number of team forests.

Makes sense. There are some team forests (at least one that I know of) that are 
not being used, so we can look into removing those and maybe combining those 
team forests which have more interdependencies into one forest (provided that 
the team members agree).

I've never been a fan of the "all projects are complete forests". It just 
increases the conceptual complexity of the repositories and the workload of keeping all 
the forests sync'ed up. It should be easy to add additional repositories to a project as 
the need arises, without requiring all repositories.



But all too often, regressions in one part of the forest only manifest 
themselves when building the entire forest.
So it is the responsibility of the Developer and Integrator to insure the 
entire forest builds and works prior
to integrations into the 'master' (jdk7/jdk7) forest.
Integrators are required to sync up with master, test build, and integrate the 
entire forest, not partials.


Developers (other than those working on the build system) rarely build 
forests. Hotspot engineers build hotspot, T&L engineers build jdk etc. 
The team integrators certainly have to combine with the "master" forest 
at the next level. But keeping a bunch of unmodified team repos up to 
date with the "master" is just pure overhead.


Maybe hotspot needs a full forest (I don't think so), but certainly 
hotspot_rt, hotspot_comp and hotspot_gc do not, in my opinion.



By having that full forest available for the team, you can at least be sure 
that anyone on the team doing full
forest builds is dealing with the same sources.
Developers can choose to only use partials, that is certainly always possible.


If a team operates in such a way that full forest builds are needed then 
the team will use a full forest. Most of the teams do not need that. 
Having the forest just creates work for the gatekeeper.


I said all this when the forest of forests was first devised. As I said 
never been a fan, don't see the need for it.


Cheers,
David


-kto

Re: JDK8 Preliminary Repository Layout

[Kelly O'Hair]

On Mar 10, 2011, at 2:50 PM, David Holmes wrote:

> Lana Steuck said the following on 03/11/11 07:03:
>> On 03/10/2011 08:46 AM, Kelly O'Hair wrote:
>>> 
>>> Interesting point, we will need to decide which projects need jdk8 forests. 
>>> I imagine some will not, and we may be
>>> doing a little trimming down on the number of team forests.
>> Makes sense. There are some team forests (at least one that I know of) that 
>> are not being used, so we can look into removing those and maybe combining 
>> those team forests which have more interdependencies into one forest 
>> (provided that the team members agree).
> 
> I've never been a fan of the "all projects are complete forests". It just 
> increases the conceptual complexity of the repositories and the workload of 
> keeping all the forests sync'ed up. It should be easy to add additional 
> repositories to a project as the need arises, without requiring all 
> repositories.
> 

But all too often, regressions in one part of the forest only manifest 
themselves when building the entire forest.
So it is the responsibility of the Developer and Integrator to insure the 
entire forest builds and works prior
to integrations into the 'master' (jdk7/jdk7) forest.
Integrators are required to sync up with master, test build, and integrate the 
entire forest, not partials.

By having that full forest available for the team, you can at least be sure 
that anyone on the team doing full
forest builds is dealing with the same sources.
Developers can choose to only use partials, that is certainly always possible.

-kto

Re: JDK8 Preliminary Repository Layout

[David Holmes]

Lana Steuck said the following on 03/11/11 07:03:

On 03/10/2011 08:46 AM, Kelly O'Hair wrote:


Interesting point, we will need to decide which projects need jdk8 
forests. I imagine some will not, and we may be

doing a little trimming down on the number of team forests.


Makes sense. There are some team forests (at least one that I know of) 
that are not being used, so we can look into removing those and maybe 
combining those team forests which have more interdependencies into one 
forest (provided that the team members agree).


I've never been a fan of the "all projects are complete forests". It 
just increases the conceptual complexity of the repositories and the 
workload of keeping all the forests sync'ed up. It should be easy to add 
additional repositories to a project as the need arises, without 
requiring all repositories.


David

Re: JDK8 Preliminary Repository Layout

[Lana Steuck]

On 03/10/2011 08:46 AM, Kelly O'Hair wrote:


Interesting point, we will need to decide which projects need jdk8 
forests. I imagine some will not, and we may be

doing a little trimming down on the number of team forests.


Makes sense. There are some team forests (at least one that I know of) 
that are not being used, so we can look into removing those and maybe 
combining those team forests which have more interdependencies into one 
forest (provided that the team members agree).


Regards,
Lana

Re: JDK8 Preliminary Repository Layout

[Kelly O'Hair]

On Mar 10, 2011, at 11:17 AM, John Coomes wrote:

> Johan Walles (johan.wal...@oracle.com) wrote:
>> 2011-03-10 10:33, Anthony Petrov skrev:
>>> Hi Andrew,
>>> 
>>> On 3/10/2011 3:48 AM, Dr Andrew John Hughes wrote:
>> [snip]
 Hey, I'd just make it all one repository as they all interdepend on
 each other
>>> 
>>> One huge "all-in-all" repository is great for integrators, porters, or
>>> maintainers, but it isn't that convenient for developers working on
>>> specific areas of JDK. Primarily because of the unnecessarily increased
>>> build time.
>> 
>> Build time wouldn't be an issue if you could build a single component by 
>> itself by just cd:ing into the component directory and do "make" there.
> 
> So I pull the entire tree, make some changes, do a build.  I want to
> sync with the latest changes in my component (whatever it is), but
> have to pull everything.  To save build time, I build just my
> component.  But then my built bits no longer correspond to my source
> tree.  Debugging becomes a new challenge.
> 
>> The problem for many developers with the all-in-one repository solution 
>> is the time it takes to clone everything (5-6 minutes).
> 
> That's the best case, when I'm on the corporate network, close to the
> server.  Over my dsl line at home, it takes 24 minutes for everything,
> vs. 1 min 10 secs for hotspot (the only part I need).

DSL?  Is that the bundle of wires I see in those comcast cable commerials? ;^)

-kto

> 
> -John
> 
>>> Since it's always possible to make a clone of the whole 'forest', and
>>> hence get all the repos virtually at once, I don't really see any
>>> problems with making some parts of the class (and native) libraries
>>> separate repositories.
>>> 
>>> --
>>> best regards,
>>> Anthony
> 

Re: JDK8 Preliminary Repository Layout

[John Coomes]

Johan Walles (johan.wal...@oracle.com) wrote:
> 2011-03-10 10:33, Anthony Petrov skrev:
> > Hi Andrew,
> >
> > On 3/10/2011 3:48 AM, Dr Andrew John Hughes wrote:
> [snip]
> >> Hey, I'd just make it all one repository as they all interdepend on
> >> each other
> >
> > One huge "all-in-all" repository is great for integrators, porters, or
> > maintainers, but it isn't that convenient for developers working on
> > specific areas of JDK. Primarily because of the unnecessarily increased
> > build time.
> 
> Build time wouldn't be an issue if you could build a single component by 
> itself by just cd:ing into the component directory and do "make" there.

So I pull the entire tree, make some changes, do a build.  I want to
sync with the latest changes in my component (whatever it is), but
have to pull everything.  To save build time, I build just my
component.  But then my built bits no longer correspond to my source
tree.  Debugging becomes a new challenge.

> The problem for many developers with the all-in-one repository solution 
> is the time it takes to clone everything (5-6 minutes).

That's the best case, when I'm on the corporate network, close to the
server.  Over my dsl line at home, it takes 24 minutes for everything,
vs. 1 min 10 secs for hotspot (the only part I need).

-John

> > Since it's always possible to make a clone of the whole 'forest', and
> > hence get all the repos virtually at once, I don't really see any
> > problems with making some parts of the class (and native) libraries
> > separate repositories.
> >
> > --
> > best regards,
> > Anthony

Re: JDK8 Preliminary Repository Layout

[Kelly O'Hair]

On Mar 9, 2011, at 4:48 PM, Dr Andrew John Hughes wrote:

>> 
>> Other ideas were considered:
>>  * Folding jaxp/jaxws into the root or jdk8/jdk repo
> 
> Sounds good.  jdk8/jdk would make more sense as jaxws depends on some classes 
> that are in the jdk
> tree (com.sun.net.httpserver) and we could then get rid of the Ant junk.  It 
> would be good to have
> the code there too so we can once again track changes.  The recent security 
> issue was a nightmare
> due to the unavailability of the source code.

Humm "Ant junk" ok...  I'm not in love with it either, but it does provide a 
very fast build of simple pure java code.
But having separated out jaxp/jaxws/corba from the rest of the jdk, we are not 
considering folding it all
back in. The consideration was moving the use of the drop bundle, not forking 
the source again.
The separation of these sources from the basic jdk has been a major benefit to 
the developers and was also meant
to ease the fork issues of this code, that is and has been maintained as 
separate open source projects and
delivered into multiple projects, licensed differently, and re-packaged in 
different ways.
The jdk is downstream on these sources.
There will be a separate discussion on this but it will focus on improving the 
source drop complications.

> 
>>  * Separating out the jdk demos from the jdk repo to a separate "demos" 
>> repository
> 
> Makes sense.  Having an option to disable them would be a good first step.

A 'do not build demos' option would be a fine RFE. Maybe do some preliminary 
surgery to at least
isolate the sources that are strictly 'demo or sample' material.  IF someone 
has the time. ;^)

> 
>>  * Separating out the client (awt/swing/etc) code from the jdk repo into a 
>> separate repo
> 
> Why would we want to do this?  IME, there are lots of interdependencies with 
> the other code and
> this would make the build a nightmare.

It was suggested and talked about a little, nobody stepped forward to champion 
it.
I assume it would be done to categorize the sources and make the developers 
lives easier
when focusing on their own area.

> 
>>  * Updating the corba sources changing it to an ant build
> 
> Please, no!  These Ant builds are already a nightmare and an order of 
> magniture harder to debug than
> the Makefiles.  Why you want to require a build system that requires an 
> entire JDK setup (possibly
> bringing another into the mix beyond the bootstrap JDK) is beyond me.
> 

Ok. I had no idea is was that bad. Well, yes, ant script logging and debugging 
is a bit bad, and the syntax
is a little ...ah... bizarre... don't want to insult any xml people... :^(  The 
pains with ant scripts seemed worth
it with regards to the speed it can compile/zip/jar all in one VM process. And 
of course the potential NetBeans
IDE usage benefits, just ask the langtools development team.
The current corba makefile build can take 10-15 minutes on some systems, and I 
would estimate that an ant build could
cut that in half or maybe a quarter of that time. Especially on Windows systems 
which aren't exactly exec demons.

But regardless, this won't be happening, corba will remain 'as is' unless we 
decide to upgrade the entire corba
source base, in which case I would advocate using the default build mechanism 
recommended and used by the
Corba team. I would like to defer to the development team that has to work on 
this code and the upstream
project that has what is considered the master sources.

> What would make much more sense is to just add corba into the jdk tree.  It 
> actually requires sun.tools
> classes to compile rmic anyway so it would make much more sense there.

No, we won't be doing that as far as I know. Unless the Corba team wants to 
lobby for that, and I suspect they won't.

> 
> Hey, I'd just make it all one repository as they all interdepend on each 
> other, but the HotSpot folks
> probably wouldn't like that... ;-)   Moving the HotSpot servicability agent 
> into the JDK, where the
> interfaces it uses lives, might be a good idea though.  I'm currently working 
> on a patch which fixes
> up a mismatch between the HotSpot SA implementation and the interfaces in the 
> JDK which has existed
> for who knows how long (it's certainly in OpenJDK6)...

When we initially started the OpenJDK project, one single repository was 
suggested. It was quickly rejected.

The comments about the Serviceability Agent is puzzling, it has a very tight 
interface between it and the Hotspot
VM data structures, or did as I recall. What JDK interfaces are you talking 
about?

> 
>> 
>> None of this seemed urgent to do out of the gate, or delay getting a 
>> preliminary jdk8 layout defined.
>> 
> 
> No, I agree with that.  Splitting off OpenJDK8 already seems overdue.
> 
>> I know there is some interest in pulling the actual jaxp/jaxws sources back 
>> into their repos, that will
>> be discussed separately, we have multiple issues with that, but I am well 
>> aware of the pains 

Re: JDK8 Preliminary Repository Layout

[Kelly O'Hair]

On Mar 9, 2011, at 4:48 PM, Dr Andrew John Hughes wrote:

> What I would ask is do the projects get this as well?  Specifially, I'd like 
> an icedtea/jdk8 at the same time please.

Interesting point, we will need to decide which projects need jdk8 forests. I 
imagine some will not, and we may be
doing a little trimming down on the number of team forests.

-kto

Re: JDK8 Preliminary Repository Layout

[Paul Hohensee]

We could rename 'closed' to be 'oracle'.  You could then have the oracle 
build

be the common code plus the 'oracle' repo.  Add an 'icedtea' repo and you
could have the icedtea build be the common code plus the 'icedtea' 
repo.  Etc.,

for other versions/vendors.

Paul

On 3/9/11 7:48 PM, Dr Andrew John Hughes wrote:

On 18:32 Tue 08 Mar , Kelly O'Hair wrote:

First, if we talk about the mercurial forests, it has nothing to do with the 
Mercurial Forest Extension.
What we really have is a set of nested repositories, sometimes called our 
"forest" of repositories.

This email is just about the actual layout of the repositories for jdk8.

The initial thinking at this time is that the openjdk8 open forest will look 
very much like openjdk7:

openjdk8/
 corba/
 jaxp/
 jaxws/
 jdk/
 hotspot/
 langtools/

7 repositories total. Cloned from the openjdk7 repos so we will have all the 
openjdk7 history in the openjdk8 repositories.

Just for discussion sake, not that you can see what is behind the closed 
curtains, we are considering changing
the closed overlay a little, from the current jdk7 (bold is a closed repo):

   jdk7/
 corba/
 deploy/
 jaxp/
 jaxws/
 jdk/
   src/closed/
   test/closed
   make/closed/
 hotspot/
   src/closed/
   test/closed
 install/
 langtools/
 pubs/

To something a little simplier like:

   jdk8/
 corba/
 deploy/
 jaxp/
 jaxws/
 jdk/
   closed/{src,test,make}
 hotspot/
   closed/{src,test,make}
 install/
 langtools/
 pubs/

The existence of these closed repos should not be a surprise, and it should 
have no impact on the openjdk itself.
We are just trying to consolidate and have fewer repositories. Just thought it 
might be of interest.


Not surprised.  Also don't really care where you put them as I don't seem to 
have them either way ;-)

What I would ask is do the projects get this as well?  Specifially, I'd like an 
icedtea/jdk8 at the same time please.


But back to the openjdk8 forest.

Other ideas were considered:
   * Folding jaxp/jaxws into the root or jdk8/jdk repo

Sounds good.  jdk8/jdk would make more sense as jaxws depends on some classes 
that are in the jdk
tree (com.sun.net.httpserver) and we could then get rid of the Ant junk.  It 
would be good to have
the code there too so we can once again track changes.  The recent security 
issue was a nightmare
due to the unavailability of the source code.


   * Separating out the jdk demos from the jdk repo to a separate "demos" 
repository

Makes sense.  Having an option to disable them would be a good first step.


   * Separating out the client (awt/swing/etc) code from the jdk repo into a 
separate repo

Why would we want to do this?  IME, there are lots of interdependencies with 
the other code and
this would make the build a nightmare.


   * Updating the corba sources changing it to an ant build

Please, no!  These Ant builds are already a nightmare and an order of magniture 
harder to debug than
the Makefiles.  Why you want to require a build system that requires an entire 
JDK setup (possibly
bringing another into the mix beyond the bootstrap JDK) is beyond me.

What would make much more sense is to just add corba into the jdk tree.  It 
actually requires sun.tools
classes to compile rmic anyway so it would make much more sense there.

Hey, I'd just make it all one repository as they all interdepend on each other, 
but the HotSpot folks
probably wouldn't like that... ;-)   Moving the HotSpot servicability agent 
into the JDK, where the
interfaces it uses lives, might be a good idea though.  I'm currently working 
on a patch which fixes
up a mismatch between the HotSpot SA implementation and the interfaces in the 
JDK which has existed
for who knows how long (it's certainly in OpenJDK6)...


None of this seemed urgent to do out of the gate, or delay getting a 
preliminary jdk8 layout defined.


No, I agree with that.  Splitting off OpenJDK8 already seems overdue.


I know there is some interest in pulling the actual jaxp/jaxws sources back 
into their repos, that will
be discussed separately, we have multiple issues with that, but I am well aware 
of the pains that the
source drop zip files have created.


The problem is less technical and more social; there is no obvious way to 
interact with the JAXP and JAXWS
side of things.  We just get these zips of code with no idea of what changes 
are in there or why.

Basically, they shouldn't work either like they do now or did before, but like 
everything else in OpenJDK
with visible commits.  Hey, maybe we could even have some mailing list 
discussion if we're very lucky...


As always, we would like to get comments, or additional ideas.

Separate topics:
   * Forest Extension and it's replacement

Do we really need one?  Current status quo (get_source.sh) works fine.


   * Mercurial server update to 1.8 or newer

I guess this is related to the 

Re: JDK8 Preliminary Repository Layout

[Maurizio Cimadamore]

On 10/03/11 09:49, Johan Walles wrote:
The problem for many developers with the all-in-one repository 
solution is the time it takes to clone everything (5-6 minutes). 
I think another problem with the all-in-one repository solution is that 
it increases chances for conflicting changes (i.e. when I was about to 
commit I find that some other change has been pushed in an unrelated 
component directory, which mean I now have to pull the change and do a 
merge before pushing).


Maurizio

Re: JDK8 Preliminary Repository Layout

[Johan Walles]

2011-03-10 10:33, Anthony Petrov skrev:

Hi Andrew,

On 3/10/2011 3:48 AM, Dr Andrew John Hughes wrote:

[snip]

Hey, I'd just make it all one repository as they all interdepend on
each other


One huge "all-in-all" repository is great for integrators, porters, or
maintainers, but it isn't that convenient for developers working on
specific areas of JDK. Primarily because of the unnecessarily increased
build time.


Build time wouldn't be an issue if you could build a single component by 
itself by just cd:ing into the component directory and do "make" there.


The problem for many developers with the all-in-one repository solution 
is the time it takes to clone everything (5-6 minutes).


  Regards //Johan


Since it's always possible to make a clone of the whole 'forest', and
hence get all the repos virtually at once, I don't really see any
problems with making some parts of the class (and native) libraries
separate repositories.

--
best regards,
Anthony

Re: JDK8 Preliminary Repository Layout

[Anthony Petrov]

Hi Andrew,

On 3/10/2011 3:48 AM, Dr Andrew John Hughes wrote:

   * Separating out the client (awt/swing/etc) code from the jdk repo into a 
separate repo


Why would we want to do this?  IME, there are lots of interdependencies with 
the other code and
this would make the build a nightmare.


Not at all. AWT/Swing/2D have a lot of inter-dependencies with each 
other, but no deep dependencies with other parts of JDK. I believe the 
Sound libraries are very similar in this regard as well. Separating them 
out makes a lot of sense to speed up the build time for those primarily 
working on bugs and enhancements in these areas only.


What particular problems do you anticipate with their separation?



Hey, I'd just make it all one repository as they all interdepend on each other


One huge "all-in-all" repository is great for integrators, porters, or 
maintainers, but it isn't that convenient for developers working on 
specific areas of JDK. Primarily because of the unnecessarily increased 
build time.


Since it's always possible to make a clone of the whole 'forest', and 
hence get all the repos virtually at once, I don't really see any 
problems with making some parts of the class (and native) libraries 
separate repositories.


--
best regards,
Anthony

Re: JDK8 Preliminary Repository Layout

[Phil Race]

Andrew,

Whilst almost everything you wrote is something I agree with (like getting
jcheck out there, not adding additional build tools/complexity), the one
thing I quite like right now comes up here I'd like to keep is the 
separate repos.

Its not just hotspot,  but I've been working on JDK for a long time now,
since before there was a hotspot and last year I built hotspot for the 
first time,

and even then only because I was working on the VS2010 compiler upgrade.

So I like that hotspot and jaxp/jaxws can be pre-built and pulled from an
import JDK in a way that doesn't slow down my builds, bloat the source
I need to pull via hg etc etc.

Maybe this doesn't matter to people who want to pull a full forest but
its great for those working on specific components. Which I believe is
most developers.

-phil.


On 3/9/2011 4:48 PM, Dr Andrew John Hughes wrote:

On 18:32 Tue 08 Mar , Kelly O'Hair wrote:

First, if we talk about the mercurial forests, it has nothing to do with the 
Mercurial Forest Extension.
What we really have is a set of nested repositories, sometimes called our 
"forest" of repositories.

This email is just about the actual layout of the repositories for jdk8.

The initial thinking at this time is that the openjdk8 open forest will look 
very much like openjdk7:

openjdk8/
 corba/
 jaxp/
 jaxws/
 jdk/
 hotspot/
 langtools/

7 repositories total. Cloned from the openjdk7 repos so we will have all the 
openjdk7 history in the openjdk8 repositories.

Just for discussion sake, not that you can see what is behind the closed 
curtains, we are considering changing
the closed overlay a little, from the current jdk7 (bold is a closed repo):

   jdk7/
 corba/
 deploy/
 jaxp/
 jaxws/
 jdk/
   src/closed/
   test/closed
   make/closed/
 hotspot/
   src/closed/
   test/closed
 install/
 langtools/
 pubs/

To something a little simplier like:

   jdk8/
 corba/
 deploy/
 jaxp/
 jaxws/
 jdk/
   closed/{src,test,make}
 hotspot/
   closed/{src,test,make}
 install/
 langtools/
 pubs/

The existence of these closed repos should not be a surprise, and it should 
have no impact on the openjdk itself.
We are just trying to consolidate and have fewer repositories. Just thought it 
might be of interest.


Not surprised.  Also don't really care where you put them as I don't seem to 
have them either way ;-)

What I would ask is do the projects get this as well?  Specifially, I'd like an 
icedtea/jdk8 at the same time please.


But back to the openjdk8 forest.

Other ideas were considered:
   * Folding jaxp/jaxws into the root or jdk8/jdk repo

Sounds good.  jdk8/jdk would make more sense as jaxws depends on some classes 
that are in the jdk
tree (com.sun.net.httpserver) and we could then get rid of the Ant junk.  It 
would be good to have
the code there too so we can once again track changes.  The recent security 
issue was a nightmare
due to the unavailability of the source code.


   * Separating out the jdk demos from the jdk repo to a separate "demos" 
repository

Makes sense.  Having an option to disable them would be a good first step.


   * Separating out the client (awt/swing/etc) code from the jdk repo into a 
separate repo

Why would we want to do this?  IME, there are lots of interdependencies with 
the other code and
this would make the build a nightmare.


   * Updating the corba sources changing it to an ant build

Please, no!  These Ant builds are already a nightmare and an order of magniture 
harder to debug than
the Makefiles.  Why you want to require a build system that requires an entire 
JDK setup (possibly
bringing another into the mix beyond the bootstrap JDK) is beyond me.

What would make much more sense is to just add corba into the jdk tree.  It 
actually requires sun.tools
classes to compile rmic anyway so it would make much more sense there.

Hey, I'd just make it all one repository as they all interdepend on each other, 
but the HotSpot folks
probably wouldn't like that... ;-)   Moving the HotSpot servicability agent 
into the JDK, where the
interfaces it uses lives, might be a good idea though.  I'm currently working 
on a patch which fixes
up a mismatch between the HotSpot SA implementation and the interfaces in the 
JDK which has existed
for who knows how long (it's certainly in OpenJDK6)...


None of this seemed urgent to do out of the gate, or delay getting a 
preliminary jdk8 layout defined.


No, I agree with that.  Splitting off OpenJDK8 already seems overdue.


I know there is some interest in pulling the actual jaxp/jaxws sources back 
into their repos, that will
be discussed separately, we have multiple issues with that, but I am well aware 
of the pains that the
source drop zip files have created.


The problem is less technical and more social; there is no obvious way to 
interact with the JAXP and JAXWS
side of things.  We just get these zips

Re: JDK8 Preliminary Repository Layout

[Dr Andrew John Hughes]

On 18:32 Tue 08 Mar , Kelly O'Hair wrote:
> 
> First, if we talk about the mercurial forests, it has nothing to do with the 
> Mercurial Forest Extension.
> What we really have is a set of nested repositories, sometimes called our 
> "forest" of repositories.
> 
> This email is just about the actual layout of the repositories for jdk8.
> 
> The initial thinking at this time is that the openjdk8 open forest will look 
> very much like openjdk7:
> 
>openjdk8/
> corba/
> jaxp/
> jaxws/
> jdk/
> hotspot/
> langtools/
> 
> 7 repositories total. Cloned from the openjdk7 repos so we will have all the 
> openjdk7 history in the openjdk8 repositories.
> 
> Just for discussion sake, not that you can see what is behind the closed 
> curtains, we are considering changing
> the closed overlay a little, from the current jdk7 (bold is a closed repo):
> 
>   jdk7/
> corba/
> deploy/
> jaxp/
> jaxws/
> jdk/
>   src/closed/
>   test/closed
>   make/closed/
> hotspot/
>   src/closed/
>   test/closed
> install/
> langtools/
> pubs/
> 
> To something a little simplier like:
> 
>   jdk8/
> corba/
> deploy/
> jaxp/
> jaxws/
> jdk/
>   closed/{src,test,make}
> hotspot/
>   closed/{src,test,make}
> install/
> langtools/
> pubs/
> 
> The existence of these closed repos should not be a surprise, and it should 
> have no impact on the openjdk itself.
> We are just trying to consolidate and have fewer repositories. Just thought 
> it might be of interest.
> 

Not surprised.  Also don't really care where you put them as I don't seem to 
have them either way ;-)

What I would ask is do the projects get this as well?  Specifially, I'd like an 
icedtea/jdk8 at the same time please.

> But back to the openjdk8 forest.
> 
> Other ideas were considered:
>   * Folding jaxp/jaxws into the root or jdk8/jdk repo

Sounds good.  jdk8/jdk would make more sense as jaxws depends on some classes 
that are in the jdk
tree (com.sun.net.httpserver) and we could then get rid of the Ant junk.  It 
would be good to have
the code there too so we can once again track changes.  The recent security 
issue was a nightmare
due to the unavailability of the source code.

>   * Separating out the jdk demos from the jdk repo to a separate "demos" 
> repository

Makes sense.  Having an option to disable them would be a good first step.

>   * Separating out the client (awt/swing/etc) code from the jdk repo into a 
> separate repo

Why would we want to do this?  IME, there are lots of interdependencies with 
the other code and
this would make the build a nightmare.

>   * Updating the corba sources changing it to an ant build

Please, no!  These Ant builds are already a nightmare and an order of magniture 
harder to debug than
the Makefiles.  Why you want to require a build system that requires an entire 
JDK setup (possibly
bringing another into the mix beyond the bootstrap JDK) is beyond me.

What would make much more sense is to just add corba into the jdk tree.  It 
actually requires sun.tools
classes to compile rmic anyway so it would make much more sense there.

Hey, I'd just make it all one repository as they all interdepend on each other, 
but the HotSpot folks
probably wouldn't like that... ;-)   Moving the HotSpot servicability agent 
into the JDK, where the
interfaces it uses lives, might be a good idea though.  I'm currently working 
on a patch which fixes
up a mismatch between the HotSpot SA implementation and the interfaces in the 
JDK which has existed
for who knows how long (it's certainly in OpenJDK6)...

> 
> None of this seemed urgent to do out of the gate, or delay getting a 
> preliminary jdk8 layout defined.
> 

No, I agree with that.  Splitting off OpenJDK8 already seems overdue.

> I know there is some interest in pulling the actual jaxp/jaxws sources back 
> into their repos, that will
> be discussed separately, we have multiple issues with that, but I am well 
> aware of the pains that the
> source drop zip files have created.
> 

The problem is less technical and more social; there is no obvious way to 
interact with the JAXP and JAXWS
side of things.  We just get these zips of code with no idea of what changes 
are in there or why.

Basically, they shouldn't work either like they do now or did before, but like 
everything else in OpenJDK
with visible commits.  Hey, maybe we could even have some mailing list 
discussion if we're very lucky...

> As always, we would like to get comments, or additional ideas.
> 
> Separate topics:
>   * Forest Extension and it's replacement

Do we really need one?  Current status quo (get_source.sh) works fine.

>   * Mercurial server update to 1.8 or newer

I guess this is related to the forest issue.

>   * Build&Test system [1]
>   * Open bug tracking system [2]
> 

You missed off open sourcing that jcheck thing... that's the biggest problem I 
have with
the Mercurial server.  :

JDK8 Preliminary Repository Layout

[Kelly O'Hair]

First, if we talk about the mercurial forests, it has nothing to do with the 
Mercurial Forest Extension.
What we really have is a set of nested repositories, sometimes called our 
"forest" of repositories.

This email is just about the actual layout of the repositories for jdk8.

The initial thinking at this time is that the openjdk8 open forest will look 
very much like openjdk7:

   openjdk8/
corba/
jaxp/
jaxws/
jdk/
hotspot/
langtools/

7 repositories total. Cloned from the openjdk7 repos so we will have all the 
openjdk7 history in the openjdk8 repositories.

Just for discussion sake, not that you can see what is behind the closed 
curtains, we are considering changing
the closed overlay a little, from the current jdk7 (bold is a closed repo):

  jdk7/
corba/
deploy/
jaxp/
jaxws/
jdk/
  src/closed/
  test/closed
  make/closed/
hotspot/
  src/closed/
  test/closed
install/
langtools/
pubs/

To something a little simplier like:

  jdk8/
corba/
deploy/
jaxp/
jaxws/
jdk/
  closed/{src,test,make}
hotspot/
  closed/{src,test,make}
install/
langtools/
pubs/

The existence of these closed repos should not be a surprise, and it should 
have no impact on the openjdk itself.
We are just trying to consolidate and have fewer repositories. Just thought it 
might be of interest.

But back to the openjdk8 forest.

Other ideas were considered:
  * Folding jaxp/jaxws into the root or jdk8/jdk repo
  * Separating out the jdk demos from the jdk repo to a separate "demos" 
repository
  * Separating out the client (awt/swing/etc) code from the jdk repo into a 
separate repo
  * Updating the corba sources changing it to an ant build

None of this seemed urgent to do out of the gate, or delay getting a 
preliminary jdk8 layout defined.

I know there is some interest in pulling the actual jaxp/jaxws sources back 
into their repos, that will
be discussed separately, we have multiple issues with that, but I am well aware 
of the pains that the
source drop zip files have created.

As always, we would like to get comments, or additional ideas.

Separate topics:
  * Forest Extension and it's replacement
  * Mercurial server update to 1.8 or newer
  * Build&Test system [1]
  * Open bug tracking system [2]

-kto

[1]   http://mail.openjdk.java.net/pipermail/build-dev/2011-February/004112.html
[2]   http://mail.openjdk.java.net/pipermail/web-discuss/2011-March/000153.html