Re: JDK-8025705
I have a different issue with closed and custom source that maybe can be addressed while you're looking at this area. As a licensee, I build with at least some of the closed source, but I also need to customize the build. Ideally, I'd like to be able to insert my custom make files and still have the closed files used. The solution might be as simple as having a configure definition for the location of my custom files. Chaining to the closed files would be my responsibility (though I'm not sure how I would change the closed files, if I needed to do that.) For JDK8, we took the path of least resistance and just modified the make files in much the same way we had for earlier releases. Tom Salter - Date: Tue, 29 Apr 2014 14:52:32 +1000 From: David Holmes david.hol...@oracle.com To: Keith McGuigan kmcgui...@twitter.com, mark.reinh...@oracle.com Cc: build-dev build-dev@openjdk.java.net Subject: Re: JDK-8025705 Message-ID: 535f3010.5070...@oracle.com Content-Type: text/plain; charset=ISO-8859-1; format=flowed Hi Keith, As requested via email I'm responding on your response to Mark. On 25/04/2014 8:10 AM, Keith McGuigan wrote: Hi Mark, Well first off, the existence of src/closed triggers OPENJDK to be unset, so even having such directories (or subrepos) present turns on all the logic that looks for files in src/closed which probably don't exist in a non-Oracle distribution. Yes that is the default logic from configure. It's not perfect but it sets OPENJDK correctly for the majority of people that only have the OpenJDK sources. As I said early on part of the problem you have is that you chose to use src/closed for your own customizations and that conflicts with the far-from-perfect use of src/closed existence checks. Obviously we don't have exclusive rights to that name, but it doesn't seem a prudent choice given the current imperfect separation. That aside configure supports building an OpenJDK configuration, --enable-openjdk-only, to allow for building the OpenJDK even when you also have custom sources present**. That said I think some recent changes regarding licensee builds may have broken that somewhat as any existence check for src/closed outside of ifndef OPENJDK would trigger. ** In that logic everywhere you see closed it should read custom. :( Building OPENJDK does have other side-effects as you have mentioned elsewhere, specifically the version information used. But presumably you are customizing that information anyway. Plus if the code were changed to use !ORACLEJDK you'd still get the default OpenJDK version information. So it seems to me that version string customization is something you will need to manage explicitly either way. So having an ORACLEJDK test instead around that code means we can control it with a single switch somewhere in a top-level Makefile (maybe even a makefile in make/closed). This would let us use the src/closed mechanism and at the same time unset ORACLEJDK (OPENJDK would also be unset). I'm not convinced this really makes a significant difference to things, and I am convinced it doesn't help us along the path to a proper separation. But if you and others think differently I'm not going to waste more energy fighting it. If we need to have a private modification to futz with ORACLEJDK (to turn it off), that's not so bad -- it'd be just in one place so maintenance would be easier. But it's possible that we can put this logic in a make/closed file that gets conditionally included, in which case we wouldn't even have to worry about that. As to how to get custom code compiled, I think we have a couple options to try. In hotspot the code is just magically picked up and treated as if it were sitting in the non-closed directory, augmenting or overriding the open sources (depending on the filename). To the best of my understanding this is all handled with the lists of src dirs to build. If you add a custom src dir then you can build those custom sources. If you add a custom src dir and remove the equivalent open src dir, then you can effectively override. Of course in a custom makefile you can do whatever you like to pick and choose files to compile etc. If we could do something like that in the jdk makefiles, that would be ideal (IMO). But it may not be feasible to do this -- I'd have to investigate. Another, probably more realistic alternative is to add a couple generic hooks (as few as possible) in the open source to make/closed makefiles which may or may not exist depending on which distribution you are building. Something like: -include make/closed/custom.gmk somewhere in a top-level Makefile which would include custom makefile if it exists, and silently skip over it if it is not there. Such logic exists already in many places in the build system. But we did not pro-actively add a custom include for every single file; rather we
Re: JDK-8025705
Hi Tom, On 30/04/2014 12:39 AM, Salter, Thomas A wrote: I have a different issue with closed and custom source that maybe can be addressed while you're looking at this area. As a licensee, I build with at least some of the closed source, but I also need to customize the build. Ideally, I'd like to be able to insert my custom make files and still have the closed files used. The solution might be as simple as having a configure definition for the location of my custom files. Chaining to the closed files would be my responsibility (though I'm not sure how I would change the closed files, if I needed to do that.) Probably not a discussion for an OpenJDK list but there is only a single notion of a custom file location, and that is closed in your case. So if you want to further customize you would need to do it within that closed directory structure. David For JDK8, we took the path of least resistance and just modified the make files in much the same way we had for earlier releases. Tom Salter - Date: Tue, 29 Apr 2014 14:52:32 +1000 From: David Holmes david.hol...@oracle.com To: Keith McGuigan kmcgui...@twitter.com, mark.reinh...@oracle.com Cc: build-dev build-dev@openjdk.java.net Subject: Re: JDK-8025705 Message-ID: 535f3010.5070...@oracle.com Content-Type: text/plain; charset=ISO-8859-1; format=flowed Hi Keith, As requested via email I'm responding on your response to Mark. On 25/04/2014 8:10 AM, Keith McGuigan wrote: Hi Mark, Well first off, the existence of src/closed triggers OPENJDK to be unset, so even having such directories (or subrepos) present turns on all the logic that looks for files in src/closed which probably don't exist in a non-Oracle distribution. Yes that is the default logic from configure. It's not perfect but it sets OPENJDK correctly for the majority of people that only have the OpenJDK sources. As I said early on part of the problem you have is that you chose to use src/closed for your own customizations and that conflicts with the far-from-perfect use of src/closed existence checks. Obviously we don't have exclusive rights to that name, but it doesn't seem a prudent choice given the current imperfect separation. That aside configure supports building an OpenJDK configuration, --enable-openjdk-only, to allow for building the OpenJDK even when you also have custom sources present**. That said I think some recent changes regarding licensee builds may have broken that somewhat as any existence check for src/closed outside of ifndef OPENJDK would trigger. ** In that logic everywhere you see closed it should read custom. :( Building OPENJDK does have other side-effects as you have mentioned elsewhere, specifically the version information used. But presumably you are customizing that information anyway. Plus if the code were changed to use !ORACLEJDK you'd still get the default OpenJDK version information. So it seems to me that version string customization is something you will need to manage explicitly either way. So having an ORACLEJDK test instead around that code means we can control it with a single switch somewhere in a top-level Makefile (maybe even a makefile in make/closed). This would let us use the src/closed mechanism and at the same time unset ORACLEJDK (OPENJDK would also be unset). I'm not convinced this really makes a significant difference to things, and I am convinced it doesn't help us along the path to a proper separation. But if you and others think differently I'm not going to waste more energy fighting it. If we need to have a private modification to futz with ORACLEJDK (to turn it off), that's not so bad -- it'd be just in one place so maintenance would be easier. But it's possible that we can put this logic in a make/closed file that gets conditionally included, in which case we wouldn't even have to worry about that. As to how to get custom code compiled, I think we have a couple options to try. In hotspot the code is just magically picked up and treated as if it were sitting in the non-closed directory, augmenting or overriding the open sources (depending on the filename). To the best of my understanding this is all handled with the lists of src dirs to build. If you add a custom src dir then you can build those custom sources. If you add a custom src dir and remove the equivalent open src dir, then you can effectively override. Of course in a custom makefile you can do whatever you like to pick and choose files to compile etc. If we could do something like that in the jdk makefiles, that would be ideal (IMO). But it may not be feasible to do this -- I'd have to investigate. Another, probably more realistic alternative is to add a couple generic hooks (as few as possible) in the open source to make/closed makefiles which may or may not exist depending on which distribution you are building. Something like: -include make/closed/custom.gmk somewhere
Re: JDK-8025705
Hi Keith, As requested via email I'm responding on your response to Mark. On 25/04/2014 8:10 AM, Keith McGuigan wrote: Hi Mark, Well first off, the existence of src/closed triggers OPENJDK to be unset, so even having such directories (or subrepos) present turns on all the logic that looks for files in src/closed which probably don't exist in a non-Oracle distribution. Yes that is the default logic from configure. It's not perfect but it sets OPENJDK correctly for the majority of people that only have the OpenJDK sources. As I said early on part of the problem you have is that you chose to use src/closed for your own customizations and that conflicts with the far-from-perfect use of src/closed existence checks. Obviously we don't have exclusive rights to that name, but it doesn't seem a prudent choice given the current imperfect separation. That aside configure supports building an OpenJDK configuration, --enable-openjdk-only, to allow for building the OpenJDK even when you also have custom sources present**. That said I think some recent changes regarding licensee builds may have broken that somewhat as any existence check for src/closed outside of ifndef OPENJDK would trigger. ** In that logic everywhere you see closed it should read custom. :( Building OPENJDK does have other side-effects as you have mentioned elsewhere, specifically the version information used. But presumably you are customizing that information anyway. Plus if the code were changed to use !ORACLEJDK you'd still get the default OpenJDK version information. So it seems to me that version string customization is something you will need to manage explicitly either way. So having an ORACLEJDK test instead around that code means we can control it with a single switch somewhere in a top-level Makefile (maybe even a makefile in make/closed). This would let us use the src/closed mechanism and at the same time unset ORACLEJDK (OPENJDK would also be unset). I'm not convinced this really makes a significant difference to things, and I am convinced it doesn't help us along the path to a proper separation. But if you and others think differently I'm not going to waste more energy fighting it. If we need to have a private modification to futz with ORACLEJDK (to turn it off), that's not so bad -- it'd be just in one place so maintenance would be easier. But it's possible that we can put this logic in a make/closed file that gets conditionally included, in which case we wouldn't even have to worry about that. As to how to get custom code compiled, I think we have a couple options to try. In hotspot the code is just magically picked up and treated as if it were sitting in the non-closed directory, augmenting or overriding the open sources (depending on the filename). To the best of my understanding this is all handled with the lists of src dirs to build. If you add a custom src dir then you can build those custom sources. If you add a custom src dir and remove the equivalent open src dir, then you can effectively override. Of course in a custom makefile you can do whatever you like to pick and choose files to compile etc. If we could do something like that in the jdk makefiles, that would be ideal (IMO). But it may not be feasible to do this -- I'd have to investigate. Another, probably more realistic alternative is to add a couple generic hooks (as few as possible) in the open source to make/closed makefiles which may or may not exist depending on which distribution you are building. Something like: -include make/closed/custom.gmk somewhere in a top-level Makefile which would include custom makefile if it exists, and silently skip over it if it is not there. Such logic exists already in many places in the build system. But we did not pro-actively add a custom include for every single file; rather we have added the includes as needed - knowing that this may need expansion in the future, but being content to wait until there was something that would actually test the use of the new hook. That said (and I have said this a few times over the past couple of years) the hook placement may not be flexible enough nor suit all possible customizations. Sometimes we need to include at the start to control the logic in the open file; sometimes at the end, to override definitions in the open file; and sometimes in the middle because that's the only way we could get it to work. :( Also note that a number of the customizations are applied at configure time, through custom .m4 file hooks, not necessarily via the .gmk files. We use CUSTOM_CONFIG_DIR to allow customization of the configure process. It would be up to the customizer to provide that file and have it include whatever is needed in order to build the custom code. For example, I could put ORACLEJDK= in there perhaps to turn off the Oracle extensions in the Makefile, as well as adding whatever other logic I need. Oracle's version of
Re: JDK-8025705
Keith, Given ifndef OPENJDK currently really means ifdef ORACLEJDK all I'm saying is that you should be setting OPENJDK=true to exclude much of the ORACLEJDK stuff - which is what you wanted. Don't let the default value for it be set based on whether src/closed is found - force it to true! There is some OracleJDK stuff explicitly referenced via src/closed whatever. For those things they should either be inside ifndef OPENJDK or else there should an existence test. This should also exclude them from your build. For your custom sources that you want to include in the build, then you have to write and maintain the rules for that in your custom makefiles - that's why we have the custom makefiles (yeah it ain't complete or perfect but it can be worked on). I don't see that adding ORACLEJDK as a variable makes any practical difference, or enables anything not currently doable by setting OPENJDK. David - On 24/04/2014 10:34 PM, Keith McGuigan wrote: On Wed, Apr 23, 2014 at 8:51 PM, David Holmes david.hol...@oracle.com mailto:david.hol...@oracle.com wrote: On 23/04/2014 9:23 PM, Keith McGuigan wrote: Yes, I did consider using some ifeq tricks like that -- but they are rather ugly and unreadable and have the same problem that you want to avoid: adding distribution-specific code into the open-source makefiles. I see no short-term fix for this beyond what I suggested, as an alternate to putting in the ORACLEJDK variable. But you would have to maintain that change in your private repo - I see no way around that. So let me get this clear then. There is Oracle code in the makefiles. That code interferes with creating an alternate distribution based upon OpenJDK using the alternative source mechanism. These things we agree upon, yes? What you telling me is that you will prevent me from adding a harmless macro to the OpenJDK makefiles that will simply identify the Oracle closed, proprietary code so that I (and anyone who is not Oracle or it's licensees) can skip over those parts of the makefile. And instead your solution is that I need to do all of this in a private repository and maintain this code myself, going forward forever, until Oracle gets around to removing the proprietary code from the makefiles (code that has been present for 7+ years and doesn't look to be going anywhere). All so that you can avoid seeing a macro that won't do you any harm or add to your burden in any way. Do I have this all correct? is this how cooperative, open development occurs in OpenJDK? Oracle gets to use features to make a custom distribution, but no one else can? This hardly seems fair. -- - Keith Given that: ifndef OPENJDK actually, implicitly means ifdef ORACLEJDK then all non-Oracle builds must presently declare themselves to be OPENJDK implementations. To which they can add specific customizations. My goal here is to have the public OpenJDK makefiles be in a state such that custom distribution code can be added (in make/closed, src/closed, or some such alternative location) without having to perform surgery on the Makefiles and maintain the private changes. The mechanism is already in place,it's just some leftover OracleJDK that hasn't made it out of the open makefiles yet. If we could just cordon that off somehow, then anyone could make a custom distribution by augmenting OpenJDK with 'closed' style repositories -- without having to maintain private, unrelated edits to jdk Makefiles. You simply need to leave OPENJDK set to true to achieve that cordoning off. Even if we move all OracleJDK specific stuff out of the open makefiles a completely clean separation may not be possible: - the customization hooks can not be everywhere and different customizations may have different requirements - if there are chunks of OpenJDK code (ie code currently part of both OpenJDK and Oracle JDK) that are not wanted in your custom build then you will still need to maintain private makefile changes to exclude them. To fix this will need additional modularization of the build with feature-selection (though not in a way that would violate the platform specifications). David - On Wed, Apr 23, 2014 at 12:10 AM, David Holmes david.hol...@oracle.com mailto:david.hol...@oracle.com mailto:david.holmes@oracle.__com mailto:david.hol...@oracle.com wrote: Hi Keith, Okay ... so you don't set OPENJDK and thus from the make logic perspective you are implicitly ORACLE_JDK. So first question: why don't you set OPENJDK and then add blocks guarded by MY_JDK (or whatever) for your custom stuff? Second, the way to get a disjunction is to
Re: JDK-8025705
I've been following this discussion for a while now and more and more I agree with Keith. The current situation where ifndef OPENJDK essentially means OracleJDK really is broken. This is something I and Magnus have noted several times but not yet had time to fokus on fixing. Mostly becuase there hasn't been any urgent need for it (no external party has expressed enough interest in utilizing the features we have for including extra source), until now. Yes, it's bad if we pollute OpenJDK makefiles with references to Oracle, but the damage really is already done since we already have all this Oracle specific stuff in them, just hidden under a not-open label. I think we should indeed make it explicit rather than pretending it's not there. I can confirm to anyone that isn't intimately familiar with the makefiles that the difference in maintenance for an external party would potentially be much less if we made the suggested changes. On 2014-04-25 00:10, Keith McGuigan wrote: Hi Mark, Well first off, the existence of src/closed triggers OPENJDK to be unset, so even having such directories (or subrepos) present turns on all the logic that looks for files in src/closed which probably don't exist in a non-Oracle distribution. So having an ORACLEJDK test instead around that code means we can control it with a single switch somewhere in a top-level Makefile (maybe even a makefile in make/closed). This would let us use the src/closed mechanism and at the same time unset ORACLEJDK (OPENJDK would also be unset). If we need to have a private modification to futz with ORACLEJDK (to turn it off), that's not so bad -- it'd be just in one place so maintenance would be easier. But it's possible that we can put this logic in a make/closed file that gets conditionally included, in which case we wouldn't even have to worry about that. As to how to get custom code compiled, I think we have a couple options to try. In hotspot the code is just magically picked up and treated as if it were sitting in the non-closed directory, augmenting or overriding the open sources (depending on the filename). If we could do something like that in the jdk makefiles, that would be ideal (IMO). But it may not be feasible to do this -- I'd have to investigate. Another, probably more realistic alternative is to add a couple generic hooks (as few as possible) in the open source to make/closed makefiles which may or may not exist depending on which distribution you are building. Something like: -include make/closed/custom.gmk somewhere in a top-level Makefile which would include custom makefile if it exists, and silently skip over it if it is not there. It would be up to the customizer to provide that file and have it include whatever is needed in order to build the custom code. For example, I could put ORACLEJDK= in there perhaps to turn off the Oracle extensions in the Makefile, as well as adding whatever other logic I need. Oracle's version of make/closed/custom.gmk maybe just sets ORACLEJDK=true and that's it (for now, once the artifacts are removed from the Makefiles you wouldn't need ORACLEJDK at all). We have some of those hooks in JDK makefiles already. In jdk8 we just had -include but in jdk9 we made it a bit more fancy with a macro that the closed implementor has to provide an implementation for (to give more freedome on the closed directory layout). It's not uniformly added everywhere yet but the pattern is established and if an open makefile is missing $(eval $(call IncludeCustomExtension, , Main.gmk)), feel free to add it. There is also some support for adding extra source directories. See ADD_SRC macro in spec.gmk.in and JavaCompilation.gmk. This is currently not used and might need some tweaking, but would be nice to see expanded upon. /Erik My vision of this is that I'd like to get to the point where OpenJDK makefiles work fine as is, but will automatically pick up new logic and new code from make/closed and src/closed if those directories exist (and contain the hook files). This gives custom code a nice place on the side where to live, with fewer worries about having to merge whenever anything in OpenJDK changes. Of course it wouldn't solve all the problems, but it would be a very nice start. On Thu, Apr 24, 2014 at 12:43 PM, mark.reinh...@oracle.com wrote: 2014/4/22 21:23 -0700, kmcgui...@twitter.com: Yes, I did consider using some ifeq tricks like that -- but they are rather ugly and unreadable and have the same problem that you want to avoid: adding distribution-specific code into the open-source makefiles. My goal here is to have the public OpenJDK makefiles be in a state such that custom distribution code can be added (in make/closed, src/closed, or some such alternative location) without having to perform surgery on the Makefiles and maintain the private changes. The mechanism is already in place,it's just some leftover OracleJDK that hasn't made it out of the open
Re: JDK-8025705
On 04/25/2014 08:48 AM, Erik Joelsson wrote: Yes, it's bad if we pollute OpenJDK makefiles with references to Oracle, but the damage really is already done since we already have all this Oracle specific stuff in them, just hidden under a not-open label. I think we should indeed make it explicit rather than pretending it's not there. It might actually be better from the POV of the non-Oracle community to expose as much of the closed build as possible. At the moment we can't tell that we might break things we can't see: at least if they were visible we'd have a clue. Andrew.
Re: JDK-8025705
On Wed, Apr 23, 2014 at 8:51 PM, David Holmes david.hol...@oracle.comwrote: On 23/04/2014 9:23 PM, Keith McGuigan wrote: Yes, I did consider using some ifeq tricks like that -- but they are rather ugly and unreadable and have the same problem that you want to avoid: adding distribution-specific code into the open-source makefiles. I see no short-term fix for this beyond what I suggested, as an alternate to putting in the ORACLEJDK variable. But you would have to maintain that change in your private repo - I see no way around that. So let me get this clear then. There is Oracle code in the makefiles. That code interferes with creating an alternate distribution based upon OpenJDK using the alternative source mechanism. These things we agree upon, yes? What you telling me is that you will prevent me from adding a harmless macro to the OpenJDK makefiles that will simply identify the Oracle closed, proprietary code so that I (and anyone who is not Oracle or it's licensees) can skip over those parts of the makefile. And instead your solution is that I need to do all of this in a private repository and maintain this code myself, going forward forever, until Oracle gets around to removing the proprietary code from the makefiles (code that has been present for 7+ years and doesn't look to be going anywhere). All so that you can avoid seeing a macro that won't do you any harm or add to your burden in any way. Do I have this all correct? is this how cooperative, open development occurs in OpenJDK? Oracle gets to use features to make a custom distribution, but no one else can? This hardly seems fair. -- - Keith Given that: ifndef OPENJDK actually, implicitly means ifdef ORACLEJDK then all non-Oracle builds must presently declare themselves to be OPENJDK implementations. To which they can add specific customizations. My goal here is to have the public OpenJDK makefiles be in a state such that custom distribution code can be added (in make/closed, src/closed, or some such alternative location) without having to perform surgery on the Makefiles and maintain the private changes. The mechanism is already in place,it's just some leftover OracleJDK that hasn't made it out of the open makefiles yet. If we could just cordon that off somehow, then anyone could make a custom distribution by augmenting OpenJDK with 'closed' style repositories -- without having to maintain private, unrelated edits to jdk Makefiles. You simply need to leave OPENJDK set to true to achieve that cordoning off. Even if we move all OracleJDK specific stuff out of the open makefiles a completely clean separation may not be possible: - the customization hooks can not be everywhere and different customizations may have different requirements - if there are chunks of OpenJDK code (ie code currently part of both OpenJDK and Oracle JDK) that are not wanted in your custom build then you will still need to maintain private makefile changes to exclude them. To fix this will need additional modularization of the build with feature-selection (though not in a way that would violate the platform specifications). David - On Wed, Apr 23, 2014 at 12:10 AM, David Holmes david.hol...@oracle.com mailto:david.hol...@oracle.com wrote: Hi Keith, Okay ... so you don't set OPENJDK and thus from the make logic perspective you are implicitly ORACLE_JDK. So first question: why don't you set OPENJDK and then add blocks guarded by MY_JDK (or whatever) for your custom stuff? Second, the way to get a disjunction is to use the text functions eg (untested but you should get the gist): // OR ifeq (true, findstring( $(OPENJDK) $(MYJDK), true) // not-OR ifneq (true, findstring( $(OPENJDK) $(MYJDK), true) It's not as trivial as || etc but not too horrendously ugly :) Does this help? David On 22/04/2014 11:10 PM, Keith McGuigan wrote: Hi David, Most of the problem resides in jdk/make, in the following files: make/CompileDemos.gmk make/CompileJavaClasses.gmk make/CopyFiles.gmk make/CopyIntoClasses.gmk make/CreateSecurityJars.gmk make/gensrc/GensrcIcons.gmk make/Images.gmk make/lib/Awt2dLibraries.gmk Biggest offender is problem CopyFiles.gmk (but CreateSecurityJars.gmk has a bit too). Basically in each situation where there's a ifndef OPENJDK, it encloses a block of code that access something in src/closed or make/closed. I did initially try to set a new variable in our build in an attempt to replace these areas with something like: ifndef OPENJDK ifndef PRIVATEJDK ... but there's apparently no convenient way of doing that in makefiles (conjunctions and disjunctions at the preprocessing level aren't available -- and the
Re: JDK-8025705
2014/4/22 21:23 -0700, kmcgui...@twitter.com: Yes, I did consider using some ifeq tricks like that -- but they are rather ugly and unreadable and have the same problem that you want to avoid: adding distribution-specific code into the open-source makefiles. My goal here is to have the public OpenJDK makefiles be in a state such that custom distribution code can be added (in make/closed, src/closed, or some such alternative location) without having to perform surgery on the Makefiles and maintain the private changes. The mechanism is already in place,it's just some leftover OracleJDK that hasn't made it out of the open makefiles yet. If we could just cordon that off somehow, then anyone could make a custom distribution by augmenting OpenJDK with 'closed' style repositories -- without having to maintain private, unrelated edits to jdk Makefiles. I'm confused. Even if we replace `ifndef OPENJDK` with `ifdef ORACLEJDK`, how does this help you? In those cases where you want an open Makefile to refer to code in Twitter's internal src/closed directory aren't you still going to have to create and maintain your own patches to the open Makefile? Just trying to understand the problem here ... - Mark
Re: JDK-8025705
Hi Mark, Well first off, the existence of src/closed triggers OPENJDK to be unset, so even having such directories (or subrepos) present turns on all the logic that looks for files in src/closed which probably don't exist in a non-Oracle distribution. So having an ORACLEJDK test instead around that code means we can control it with a single switch somewhere in a top-level Makefile (maybe even a makefile in make/closed). This would let us use the src/closed mechanism and at the same time unset ORACLEJDK (OPENJDK would also be unset). If we need to have a private modification to futz with ORACLEJDK (to turn it off), that's not so bad -- it'd be just in one place so maintenance would be easier. But it's possible that we can put this logic in a make/closed file that gets conditionally included, in which case we wouldn't even have to worry about that. As to how to get custom code compiled, I think we have a couple options to try. In hotspot the code is just magically picked up and treated as if it were sitting in the non-closed directory, augmenting or overriding the open sources (depending on the filename). If we could do something like that in the jdk makefiles, that would be ideal (IMO). But it may not be feasible to do this -- I'd have to investigate. Another, probably more realistic alternative is to add a couple generic hooks (as few as possible) in the open source to make/closed makefiles which may or may not exist depending on which distribution you are building. Something like: -include make/closed/custom.gmk somewhere in a top-level Makefile which would include custom makefile if it exists, and silently skip over it if it is not there. It would be up to the customizer to provide that file and have it include whatever is needed in order to build the custom code. For example, I could put ORACLEJDK= in there perhaps to turn off the Oracle extensions in the Makefile, as well as adding whatever other logic I need. Oracle's version of make/closed/custom.gmk maybe just sets ORACLEJDK=true and that's it (for now, once the artifacts are removed from the Makefiles you wouldn't need ORACLEJDK at all). My vision of this is that I'd like to get to the point where OpenJDK makefiles work fine as is, but will automatically pick up new logic and new code from make/closed and src/closed if those directories exist (and contain the hook files). This gives custom code a nice place on the side where to live, with fewer worries about having to merge whenever anything in OpenJDK changes. Of course it wouldn't solve all the problems, but it would be a very nice start. On Thu, Apr 24, 2014 at 12:43 PM, mark.reinh...@oracle.com wrote: 2014/4/22 21:23 -0700, kmcgui...@twitter.com: Yes, I did consider using some ifeq tricks like that -- but they are rather ugly and unreadable and have the same problem that you want to avoid: adding distribution-specific code into the open-source makefiles. My goal here is to have the public OpenJDK makefiles be in a state such that custom distribution code can be added (in make/closed, src/closed, or some such alternative location) without having to perform surgery on the Makefiles and maintain the private changes. The mechanism is already in place,it's just some leftover OracleJDK that hasn't made it out of the open makefiles yet. If we could just cordon that off somehow, then anyone could make a custom distribution by augmenting OpenJDK with 'closed' style repositories -- without having to maintain private, unrelated edits to jdk Makefiles. I'm confused. Even if we replace `ifndef OPENJDK` with `ifdef ORACLEJDK`, how does this help you? In those cases where you want an open Makefile to refer to code in Twitter's internal src/closed directory aren't you still going to have to create and maintain your own patches to the open Makefile? Just trying to understand the problem here ... - Mark -- [image: twitter-icon-large.png] Keith McGuigan @kamggg kmcgui...@twitter.com
Re: JDK-8025705
Yes, I did consider using some ifeq tricks like that -- but they are rather ugly and unreadable and have the same problem that you want to avoid: adding distribution-specific code into the open-source makefiles. My goal here is to have the public OpenJDK makefiles be in a state such that custom distribution code can be added (in make/closed, src/closed, or some such alternative location) without having to perform surgery on the Makefiles and maintain the private changes. The mechanism is already in place,it's just some leftover OracleJDK that hasn't made it out of the open makefiles yet. If we could just cordon that off somehow, then anyone could make a custom distribution by augmenting OpenJDK with 'closed' style repositories -- without having to maintain private, unrelated edits to jdk Makefiles. On Wed, Apr 23, 2014 at 12:10 AM, David Holmes david.hol...@oracle.comwrote: Hi Keith, Okay ... so you don't set OPENJDK and thus from the make logic perspective you are implicitly ORACLE_JDK. So first question: why don't you set OPENJDK and then add blocks guarded by MY_JDK (or whatever) for your custom stuff? Second, the way to get a disjunction is to use the text functions eg (untested but you should get the gist): // OR ifeq (true, findstring( $(OPENJDK) $(MYJDK), true) // not-OR ifneq (true, findstring( $(OPENJDK) $(MYJDK), true) It's not as trivial as || etc but not too horrendously ugly :) Does this help? David On 22/04/2014 11:10 PM, Keith McGuigan wrote: Hi David, Most of the problem resides in jdk/make, in the following files: make/CompileDemos.gmk make/CompileJavaClasses.gmk make/CopyFiles.gmk make/CopyIntoClasses.gmk make/CreateSecurityJars.gmk make/gensrc/GensrcIcons.gmk make/Images.gmk make/lib/Awt2dLibraries.gmk Biggest offender is problem CopyFiles.gmk (but CreateSecurityJars.gmk has a bit too). Basically in each situation where there's a ifndef OPENJDK, it encloses a block of code that access something in src/closed or make/closed. I did initially try to set a new variable in our build in an attempt to replace these areas with something like: ifndef OPENJDK ifndef PRIVATEJDK ... but there's apparently no convenient way of doing that in makefiles (conjunctions and disjunctions at the preprocessing level aren't available -- and the workarounds are rather goofy). Duplicating the OPENJDK only code quickly became unreasonable too -- a few of the code blocks are one-liners, but there's a bunch that are much more involved clauses. On Tue, Apr 22, 2014 at 8:23 AM, David Holmes david.hol...@oracle.com mailto:david.hol...@oracle.com wrote: Hi Keith, Sorry I have very limited cycles right now, and just had a 4 day Easter break with anther long weekend ahead :) You are right that the src/closed - CUSTOM_SRC_DIR is somewhat tangential to your issue. The existence checks I suggested would be a check for whatever file/directory is enough to indicate the feature is present. Most uses of OPENJDK are really used to indicate !ORACLE_JDK, so introducing a third variation doesn't really fit. Can you give a concrete example of something that highlights this problem for you and how your proposal addresses it? I may get a better sense of things with specifics rather than trying to generalize - because I don't see a general solution without a lot of refactoring. Thanks, David On 22/04/2014 2:42 PM, Keith McGuigan wrote: Hi Mark, et al., The sad reality of the situation is that there is indeed Oracle-specific code in the OpenJDK makefiles, and this code interferes with our customization of the JDK. Adding temporary signposts to allow us (and others) to avoid this code will not make things worse. It doesn't have to be a distribution name -- we call it whatever you like: TO_BE_REMOVED, KEITH_IS_A_PITA, whatever -- just something to latch onto to deactivate that code when it is not needed. This would provide immediate relief to customizing distributors and give Oracle engineers time to phase that code into closed makefiles (at which time the signposts can be removed completely). Taking all this code out wholesale instead would be great, and this is something I am totally willing to tackle and put the effort in on if I was in a position to do so. Unfortunately, since this is not fully open-source, I can't do the refactoring needed to move this code into the closed directories. And I though I could easily strip the code from OpenJDK, this would totally muck with Oracle distribution so any patch I would submit would surely be immediately rejected. Considering that the code is question has been in
Re: JDK-8025705
On 23/04/2014 9:23 PM, Keith McGuigan wrote: Yes, I did consider using some ifeq tricks like that -- but they are rather ugly and unreadable and have the same problem that you want to avoid: adding distribution-specific code into the open-source makefiles. I see no short-term fix for this beyond what I suggested, as an alternate to putting in the ORACLEJDK variable. But you would have to maintain that change in your private repo - I see no way around that. Given that: ifndef OPENJDK actually, implicitly means ifdef ORACLEJDK then all non-Oracle builds must presently declare themselves to be OPENJDK implementations. To which they can add specific customizations. My goal here is to have the public OpenJDK makefiles be in a state such that custom distribution code can be added (in make/closed, src/closed, or some such alternative location) without having to perform surgery on the Makefiles and maintain the private changes. The mechanism is already in place,it's just some leftover OracleJDK that hasn't made it out of the open makefiles yet. If we could just cordon that off somehow, then anyone could make a custom distribution by augmenting OpenJDK with 'closed' style repositories -- without having to maintain private, unrelated edits to jdk Makefiles. You simply need to leave OPENJDK set to true to achieve that cordoning off. Even if we move all OracleJDK specific stuff out of the open makefiles a completely clean separation may not be possible: - the customization hooks can not be everywhere and different customizations may have different requirements - if there are chunks of OpenJDK code (ie code currently part of both OpenJDK and Oracle JDK) that are not wanted in your custom build then you will still need to maintain private makefile changes to exclude them. To fix this will need additional modularization of the build with feature-selection (though not in a way that would violate the platform specifications). David - On Wed, Apr 23, 2014 at 12:10 AM, David Holmes david.hol...@oracle.com mailto:david.hol...@oracle.com wrote: Hi Keith, Okay ... so you don't set OPENJDK and thus from the make logic perspective you are implicitly ORACLE_JDK. So first question: why don't you set OPENJDK and then add blocks guarded by MY_JDK (or whatever) for your custom stuff? Second, the way to get a disjunction is to use the text functions eg (untested but you should get the gist): // OR ifeq (true, findstring( $(OPENJDK) $(MYJDK), true) // not-OR ifneq (true, findstring( $(OPENJDK) $(MYJDK), true) It's not as trivial as || etc but not too horrendously ugly :) Does this help? David On 22/04/2014 11:10 PM, Keith McGuigan wrote: Hi David, Most of the problem resides in jdk/make, in the following files: make/CompileDemos.gmk make/CompileJavaClasses.gmk make/CopyFiles.gmk make/CopyIntoClasses.gmk make/CreateSecurityJars.gmk make/gensrc/GensrcIcons.gmk make/Images.gmk make/lib/Awt2dLibraries.gmk Biggest offender is problem CopyFiles.gmk (but CreateSecurityJars.gmk has a bit too). Basically in each situation where there's a ifndef OPENJDK, it encloses a block of code that access something in src/closed or make/closed. I did initially try to set a new variable in our build in an attempt to replace these areas with something like: ifndef OPENJDK ifndef PRIVATEJDK ... but there's apparently no convenient way of doing that in makefiles (conjunctions and disjunctions at the preprocessing level aren't available -- and the workarounds are rather goofy). Duplicating the OPENJDK only code quickly became unreasonable too -- a few of the code blocks are one-liners, but there's a bunch that are much more involved clauses. On Tue, Apr 22, 2014 at 8:23 AM, David Holmes david.hol...@oracle.com mailto:david.hol...@oracle.com mailto:david.holmes@oracle.__com mailto:david.hol...@oracle.com wrote: Hi Keith, Sorry I have very limited cycles right now, and just had a 4 day Easter break with anther long weekend ahead :) You are right that the src/closed - CUSTOM_SRC_DIR is somewhat tangential to your issue. The existence checks I suggested would be a check for whatever file/directory is enough to indicate the feature is present. Most uses of OPENJDK are really used to indicate !ORACLE_JDK, so introducing a third variation doesn't really fit. Can you give a concrete example of something that highlights this problem for you and how your proposal addresses it? I may get a better sense of things with
Re: JDK-8025705
Hi Keith, Sorry I have very limited cycles right now, and just had a 4 day Easter break with anther long weekend ahead :) You are right that the src/closed - CUSTOM_SRC_DIR is somewhat tangential to your issue. The existence checks I suggested would be a check for whatever file/directory is enough to indicate the feature is present. Most uses of OPENJDK are really used to indicate !ORACLE_JDK, so introducing a third variation doesn't really fit. Can you give a concrete example of something that highlights this problem for you and how your proposal addresses it? I may get a better sense of things with specifics rather than trying to generalize - because I don't see a general solution without a lot of refactoring. Thanks, David On 22/04/2014 2:42 PM, Keith McGuigan wrote: Hi Mark, et al., The sad reality of the situation is that there is indeed Oracle-specific code in the OpenJDK makefiles, and this code interferes with our customization of the JDK. Adding temporary signposts to allow us (and others) to avoid this code will not make things worse. It doesn't have to be a distribution name -- we call it whatever you like: TO_BE_REMOVED, KEITH_IS_A_PITA, whatever -- just something to latch onto to deactivate that code when it is not needed. This would provide immediate relief to customizing distributors and give Oracle engineers time to phase that code into closed makefiles (at which time the signposts can be removed completely). Taking all this code out wholesale instead would be great, and this is something I am totally willing to tackle and put the effort in on if I was in a position to do so. Unfortunately, since this is not fully open-source, I can't do the refactoring needed to move this code into the closed directories. And I though I could easily strip the code from OpenJDK, this would totally muck with Oracle distribution so any patch I would submit would surely be immediately rejected. Considering that the code is question has been in OpenJDK for about 8 years now, I think it's safe to assume that it's not a high priority item for Oracle engineers to get this fixed. Which is totally fine, IMO -- it's very much a tenant of open source development that he who has the itch ought to be the one to scratch it, and different entities are expected to have different sets of priorities. No doubt I'm probably the first one to complain about this :) Unfortunately, I'm also in the unfortunate position of having an itch (and willing fingernails), but an inability to scratch it. So, where do we go from here and how can I help in this effort? I really do want to help, as this is an immediate problem for me and I can't afford to wait years for it to get fixed. I still think that signposts are a very reasonable compromise given that: (1) It is something that can be done independently and doesn't require Oracle engineering resources (other than reviews and shepherding) (2) It does not interfere with efforts to move closed code out of OpenJDK makefiles (3) it can be done quickly (4) It does not avoid the Makefile-checking for existence of required files/directories (which reduces build-brittleness) Mark/Dave, if I can't convince you that we should take this path, can you please suggest an alternative design? I'm not picky -- if we can come up with something else that works then let's do it and I'll start on it right away. -- - Keith (itchy) On Mon, Apr 21, 2014 at 8:23 PM, mark.reinh...@oracle.com mailto:mark.reinh...@oracle.com wrote: 2014/4/16 14:52 -0700, david.hol...@oracle.com mailto:david.hol...@oracle.com: src/closed is Oracle's custom source location (hotspot calls it alt_src). If we never saw src/closed in the makefiles, only CUSTOM_SRC_DIR, and guarded with an existence test for a specific directory/file, then that should address your problem. That would be a first step in moving things to the custom makefiles where they belong. I'm opposed to the ORACLEJDK variable because I want to maintain the pressure to get this fixed properly. If we hack around it then it will never get cleaned up. I think it's wrong, in principle, for OpenJDK source code to contain identifiers naming specific vendors of JDK implementations. We're not quite there at the moment, but let's please not add any more. - Mark
Re: JDK-8025705
Hi David, Most of the problem resides in jdk/make, in the following files: make/CompileDemos.gmk make/CompileJavaClasses.gmk make/CopyFiles.gmk make/CopyIntoClasses.gmk make/CreateSecurityJars.gmk make/gensrc/GensrcIcons.gmk make/Images.gmk make/lib/Awt2dLibraries.gmk Biggest offender is problem CopyFiles.gmk (but CreateSecurityJars.gmk has a bit too). Basically in each situation where there's a ifndef OPENJDK, it encloses a block of code that access something in src/closed or make/closed. I did initially try to set a new variable in our build in an attempt to replace these areas with something like: ifndef OPENJDK ifndef PRIVATEJDK ... but there's apparently no convenient way of doing that in makefiles (conjunctions and disjunctions at the preprocessing level aren't available -- and the workarounds are rather goofy). Duplicating the OPENJDK only code quickly became unreasonable too -- a few of the code blocks are one-liners, but there's a bunch that are much more involved clauses. On Tue, Apr 22, 2014 at 8:23 AM, David Holmes david.hol...@oracle.comwrote: Hi Keith, Sorry I have very limited cycles right now, and just had a 4 day Easter break with anther long weekend ahead :) You are right that the src/closed - CUSTOM_SRC_DIR is somewhat tangential to your issue. The existence checks I suggested would be a check for whatever file/directory is enough to indicate the feature is present. Most uses of OPENJDK are really used to indicate !ORACLE_JDK, so introducing a third variation doesn't really fit. Can you give a concrete example of something that highlights this problem for you and how your proposal addresses it? I may get a better sense of things with specifics rather than trying to generalize - because I don't see a general solution without a lot of refactoring. Thanks, David On 22/04/2014 2:42 PM, Keith McGuigan wrote: Hi Mark, et al., The sad reality of the situation is that there is indeed Oracle-specific code in the OpenJDK makefiles, and this code interferes with our customization of the JDK. Adding temporary signposts to allow us (and others) to avoid this code will not make things worse. It doesn't have to be a distribution name -- we call it whatever you like: TO_BE_REMOVED, KEITH_IS_A_PITA, whatever -- just something to latch onto to deactivate that code when it is not needed. This would provide immediate relief to customizing distributors and give Oracle engineers time to phase that code into closed makefiles (at which time the signposts can be removed completely). Taking all this code out wholesale instead would be great, and this is something I am totally willing to tackle and put the effort in on if I was in a position to do so. Unfortunately, since this is not fully open-source, I can't do the refactoring needed to move this code into the closed directories. And I though I could easily strip the code from OpenJDK, this would totally muck with Oracle distribution so any patch I would submit would surely be immediately rejected. Considering that the code is question has been in OpenJDK for about 8 years now, I think it's safe to assume that it's not a high priority item for Oracle engineers to get this fixed. Which is totally fine, IMO -- it's very much a tenant of open source development that he who has the itch ought to be the one to scratch it, and different entities are expected to have different sets of priorities. No doubt I'm probably the first one to complain about this :) Unfortunately, I'm also in the unfortunate position of having an itch (and willing fingernails), but an inability to scratch it. So, where do we go from here and how can I help in this effort? I really do want to help, as this is an immediate problem for me and I can't afford to wait years for it to get fixed. I still think that signposts are a very reasonable compromise given that: (1) It is something that can be done independently and doesn't require Oracle engineering resources (other than reviews and shepherding) (2) It does not interfere with efforts to move closed code out of OpenJDK makefiles (3) it can be done quickly (4) It does not avoid the Makefile-checking for existence of required files/directories (which reduces build-brittleness) Mark/Dave, if I can't convince you that we should take this path, can you please suggest an alternative design? I'm not picky -- if we can come up with something else that works then let's do it and I'll start on it right away. -- - Keith (itchy) On Mon, Apr 21, 2014 at 8:23 PM, mark.reinh...@oracle.com mailto:mark.reinh...@oracle.com wrote: 2014/4/16 14:52 -0700, david.hol...@oracle.com mailto:david.hol...@oracle.com: src/closed is Oracle's custom source location (hotspot calls it alt_src). If we never saw src/closed in the makefiles, only CUSTOM_SRC_DIR, and guarded with an existence test for a specific directory/file, then that
Re: JDK-8025705
Hi Keith, Okay ... so you don't set OPENJDK and thus from the make logic perspective you are implicitly ORACLE_JDK. So first question: why don't you set OPENJDK and then add blocks guarded by MY_JDK (or whatever) for your custom stuff? Second, the way to get a disjunction is to use the text functions eg (untested but you should get the gist): // OR ifeq (true, findstring( $(OPENJDK) $(MYJDK), true) // not-OR ifneq (true, findstring( $(OPENJDK) $(MYJDK), true) It's not as trivial as || etc but not too horrendously ugly :) Does this help? David On 22/04/2014 11:10 PM, Keith McGuigan wrote: Hi David, Most of the problem resides in jdk/make, in the following files: make/CompileDemos.gmk make/CompileJavaClasses.gmk make/CopyFiles.gmk make/CopyIntoClasses.gmk make/CreateSecurityJars.gmk make/gensrc/GensrcIcons.gmk make/Images.gmk make/lib/Awt2dLibraries.gmk Biggest offender is problem CopyFiles.gmk (but CreateSecurityJars.gmk has a bit too). Basically in each situation where there's a ifndef OPENJDK, it encloses a block of code that access something in src/closed or make/closed. I did initially try to set a new variable in our build in an attempt to replace these areas with something like: ifndef OPENJDK ifndef PRIVATEJDK ... but there's apparently no convenient way of doing that in makefiles (conjunctions and disjunctions at the preprocessing level aren't available -- and the workarounds are rather goofy). Duplicating the OPENJDK only code quickly became unreasonable too -- a few of the code blocks are one-liners, but there's a bunch that are much more involved clauses. On Tue, Apr 22, 2014 at 8:23 AM, David Holmes david.hol...@oracle.com mailto:david.hol...@oracle.com wrote: Hi Keith, Sorry I have very limited cycles right now, and just had a 4 day Easter break with anther long weekend ahead :) You are right that the src/closed - CUSTOM_SRC_DIR is somewhat tangential to your issue. The existence checks I suggested would be a check for whatever file/directory is enough to indicate the feature is present. Most uses of OPENJDK are really used to indicate !ORACLE_JDK, so introducing a third variation doesn't really fit. Can you give a concrete example of something that highlights this problem for you and how your proposal addresses it? I may get a better sense of things with specifics rather than trying to generalize - because I don't see a general solution without a lot of refactoring. Thanks, David On 22/04/2014 2:42 PM, Keith McGuigan wrote: Hi Mark, et al., The sad reality of the situation is that there is indeed Oracle-specific code in the OpenJDK makefiles, and this code interferes with our customization of the JDK. Adding temporary signposts to allow us (and others) to avoid this code will not make things worse. It doesn't have to be a distribution name -- we call it whatever you like: TO_BE_REMOVED, KEITH_IS_A_PITA, whatever -- just something to latch onto to deactivate that code when it is not needed. This would provide immediate relief to customizing distributors and give Oracle engineers time to phase that code into closed makefiles (at which time the signposts can be removed completely). Taking all this code out wholesale instead would be great, and this is something I am totally willing to tackle and put the effort in on if I was in a position to do so. Unfortunately, since this is not fully open-source, I can't do the refactoring needed to move this code into the closed directories. And I though I could easily strip the code from OpenJDK, this would totally muck with Oracle distribution so any patch I would submit would surely be immediately rejected. Considering that the code is question has been in OpenJDK for about 8 years now, I think it's safe to assume that it's not a high priority item for Oracle engineers to get this fixed. Which is totally fine, IMO -- it's very much a tenant of open source development that he who has the itch ought to be the one to scratch it, and different entities are expected to have different sets of priorities. No doubt I'm probably the first one to complain about this :) Unfortunately, I'm also in the unfortunate position of having an itch (and willing fingernails), but an inability to scratch it. So, where do we go from here and how can I help in this effort? I really do want to help, as this is an immediate problem for me and I can't afford to wait years for it to get fixed. I still think that signposts are a very reasonable compromise given that: (1) It is something that
Re: JDK-8025705
2014/4/16 14:52 -0700, david.hol...@oracle.com: src/closed is Oracle's custom source location (hotspot calls it alt_src). If we never saw src/closed in the makefiles, only CUSTOM_SRC_DIR, and guarded with an existence test for a specific directory/file, then that should address your problem. That would be a first step in moving things to the custom makefiles where they belong. I'm opposed to the ORACLEJDK variable because I want to maintain the pressure to get this fixed properly. If we hack around it then it will never get cleaned up. I think it's wrong, in principle, for OpenJDK source code to contain identifiers naming specific vendors of JDK implementations. We're not quite there at the moment, but let's please not add any more. - Mark
Re: JDK-8025705
Hi Mark, et al., The sad reality of the situation is that there is indeed Oracle-specific code in the OpenJDK makefiles, and this code interferes with our customization of the JDK. Adding temporary signposts to allow us (and others) to avoid this code will not make things worse. It doesn't have to be a distribution name -- we call it whatever you like: TO_BE_REMOVED, KEITH_IS_A_PITA, whatever -- just something to latch onto to deactivate that code when it is not needed. This would provide immediate relief to customizing distributors and give Oracle engineers time to phase that code into closed makefiles (at which time the signposts can be removed completely). Taking all this code out wholesale instead would be great, and this is something I am totally willing to tackle and put the effort in on if I was in a position to do so. Unfortunately, since this is not fully open-source, I can't do the refactoring needed to move this code into the closed directories. And I though I could easily strip the code from OpenJDK, this would totally muck with Oracle distribution so any patch I would submit would surely be immediately rejected. Considering that the code is question has been in OpenJDK for about 8 years now, I think it's safe to assume that it's not a high priority item for Oracle engineers to get this fixed. Which is totally fine, IMO -- it's very much a tenant of open source development that he who has the itch ought to be the one to scratch it, and different entities are expected to have different sets of priorities. No doubt I'm probably the first one to complain about this :) Unfortunately, I'm also in the unfortunate position of having an itch (and willing fingernails), but an inability to scratch it. So, where do we go from here and how can I help in this effort? I really do want to help, as this is an immediate problem for me and I can't afford to wait years for it to get fixed. I still think that signposts are a very reasonable compromise given that: (1) It is something that can be done independently and doesn't require Oracle engineering resources (other than reviews and shepherding) (2) It does not interfere with efforts to move closed code out of OpenJDK makefiles (3) it can be done quickly (4) It does not avoid the Makefile-checking for existence of required files/directories (which reduces build-brittleness) Mark/Dave, if I can't convince you that we should take this path, can you please suggest an alternative design? I'm not picky -- if we can come up with something else that works then let's do it and I'll start on it right away. -- - Keith (itchy) On Mon, Apr 21, 2014 at 8:23 PM, mark.reinh...@oracle.com wrote: 2014/4/16 14:52 -0700, david.hol...@oracle.com: src/closed is Oracle's custom source location (hotspot calls it alt_src). If we never saw src/closed in the makefiles, only CUSTOM_SRC_DIR, and guarded with an existence test for a specific directory/file, then that should address your problem. That would be a first step in moving things to the custom makefiles where they belong. I'm opposed to the ORACLEJDK variable because I want to maintain the pressure to get this fixed properly. If we hack around it then it will never get cleaned up. I think it's wrong, in principle, for OpenJDK source code to contain identifiers naming specific vendors of JDK implementations. We're not quite there at the moment, but let's please not add any more. - Mark
Re: JDK-8025705
(moving discussion to build-dev since this isn't directly part of the makefile rewrite project) Hello Keith, I certainly feel your pain in dealing with this, it's currently a mess. I'm not as opposed to the ORACLEJDK variable as David is, but I'm also not sure it will correctly express things in all current situations. In many cases, specific individual variables would probably make more sense. Also note that we have several references to OPENJDK in the Oracle closed makefiles that would need to be updated at the same time. I agree that we need to move away from explicitly writing src/closed. We should definitely move as much of the Oracle specific parts of the build to closed makefiles, even though it's sometimes tricky. I don't think just having existence checks is enough. We do internally support the configure flag --enable-openjdk-only, which forces the build to ignore the non openjdk parts. It's not 100% functioning today, but I think it should be. This is also about consistency checking during the build. If parts of the build is optional just depending on existence of files, then a misspelled reference or other mistake can make those parts being silently excluded. At least for the common build scenarios/configurations I would like the build to know what needs to be built, which means we need explicit variables to control it. /Erik On 2014-04-17 06:52, David Holmes wrote: Hi Keith, src/closed is Oracle's custom source location (hotspot calls it alt_src). If we never saw src/closed in the makefiles, only CUSTOM_SRC_DIR, and guarded with an existence test for a specific directory/file, then that should address your problem. That would be a first step in moving things to the custom makefiles where they belong. I'm opposed to the ORACLEJDK variable because I want to maintain the pressure to get this fixed properly. If we hack around it then it will never get cleaned up. I see 68 uses of src/closed across 14 files in the JDK repo. That seems tractable. I think there are three things to be done here: 1. Replace all uses of src/closed with CUSTOM_SRC_DIR (similar to CUSTOM_MAKE_DIR) which in turn is set via configure 2. Guard all uses of CUSTOM_SRC_DIR in open makefiles with an existence check 3. Move all uses of CUSTOM_SRC_DIR to our closed makefiles Steps 1 and 2 can happen now. Step 3 is long term goal. --- The other problem I see with the OPENJDK, ORACLE_JDK, OTHER_JDK approach is that you actually have to deal with the permutations. Something currently flagged for OPENJDK really means !ORACLE_JDK - or does it? It actually depends on what sources a given licensee has. Even for your custom build you might want some OPENJDK items and not others. I'm not sure there is a general solution, but using OPENJDK in combination with CUSTOM_SRC_DIR is, I think, more flexible than trying to define discrete variables that represent build targets. David On 17/04/2014 1:31 PM, Keith McGuigan wrote: On Wed, Apr 16, 2014 at 9:15 PM, David Holmes david.hol...@oracle.com mailto:david.hol...@oracle.com wrote: Hi Keith, On 17/04/2014 7:13 AM, Keith McGuigan wrote: Hello, I just added a comment to this bug -- see there for the details, but in short I'd like to update a number of tests in the makefiles that check OPENJDK and change them to check instead of the inverse definition of some new variable, such as ORACLEJDK. Please no! It's bad enough this is implicitly in the build without making it explicit! As I mentioned, I agree that moving all this to closed makefiles is the best solution (and is something that we could push for even if we took this partial step), but doing at least this step would be a vast improvement from our point of view and is much easier to implement, especially for someone like me who cannot do the make/closed refactoring. Would file existence tests suffice? There should be a CUSTOM_SRC variable for src/closed as there is CUSTOM_MAKE for make/closed. It's not really feasible for the jdk makefiles. Almost each location where there is an OPENJDK test, when it is discovered that this isn't OpenJDK, it ends up referring to files in src/closed (which for us don't exist). In Hotspot it's only a few makefiles, so not too bad there, but jdk is a different story. But really, there's three situations here, OpenJDK, OracleJDK, and OtherJDK/custom, which can't be encoded using one boolean makefile variable. We really need at least one more here. Why is ORACLEJDK so abhorent? This would simply non-OpenJDK (i.e., src/closed builds), non-Oracle builds for those who are making their own distributions using the src/closed mechanism. As you can guess, that is something we are doing here at Twitter :) Hopefully you use src/custom (or whatever) not src/closed, as otherwise there's no way to tell the
Re: JDK-8025705
Hi Erik, Sorry I posted this to the wrong list. I agree that using CUSTOM_SRC_DIR, configured via 'configure', is a good idea (Dave's #1). I'm not so sure that file existence checks are the best idea, though, for the reasons Erik points out -- change a filename and all of sudden things just stop working because that happened to be the file that the Makefile was looking for as a trigger for a particular feature. Probably better would be individual feature flags that are (again) enabled or disabled via configure. This would allow build-time checking of the existence of files and would not be as brittle. But how many flags would that be? 68 seems a bit much (though I'm sure we'd get some overlap in features). Maybe something in-between, where we check for some top-level directory and turn features off or on based upon the existence of that? If there's a reasonable solution that I can start working on now, that'd be great -- anything we do in this direction (even a partial solution) would help reduce our ongoing maintenance costs and would be worth doing IMO. -- - Keith On Thu, Apr 17, 2014 at 3:28 AM, Erik Joelsson erik.joels...@oracle.comwrote: (moving discussion to build-dev since this isn't directly part of the makefile rewrite project) Hello Keith, I certainly feel your pain in dealing with this, it's currently a mess. I'm not as opposed to the ORACLEJDK variable as David is, but I'm also not sure it will correctly express things in all current situations. In many cases, specific individual variables would probably make more sense. Also note that we have several references to OPENJDK in the Oracle closed makefiles that would need to be updated at the same time. I agree that we need to move away from explicitly writing src/closed. We should definitely move as much of the Oracle specific parts of the build to closed makefiles, even though it's sometimes tricky. I don't think just having existence checks is enough. We do internally support the configure flag --enable-openjdk-only, which forces the build to ignore the non openjdk parts. It's not 100% functioning today, but I think it should be. This is also about consistency checking during the build. If parts of the build is optional just depending on existence of files, then a misspelled reference or other mistake can make those parts being silently excluded. At least for the common build scenarios/configurations I would like the build to know what needs to be built, which means we need explicit variables to control it. /Erik On 2014-04-17 06:52, David Holmes wrote: Hi Keith, src/closed is Oracle's custom source location (hotspot calls it alt_src). If we never saw src/closed in the makefiles, only CUSTOM_SRC_DIR, and guarded with an existence test for a specific directory/file, then that should address your problem. That would be a first step in moving things to the custom makefiles where they belong. I'm opposed to the ORACLEJDK variable because I want to maintain the pressure to get this fixed properly. If we hack around it then it will never get cleaned up. I see 68 uses of src/closed across 14 files in the JDK repo. That seems tractable. I think there are three things to be done here: 1. Replace all uses of src/closed with CUSTOM_SRC_DIR (similar to CUSTOM_MAKE_DIR) which in turn is set via configure 2. Guard all uses of CUSTOM_SRC_DIR in open makefiles with an existence check 3. Move all uses of CUSTOM_SRC_DIR to our closed makefiles Steps 1 and 2 can happen now. Step 3 is long term goal. --- The other problem I see with the OPENJDK, ORACLE_JDK, OTHER_JDK approach is that you actually have to deal with the permutations. Something currently flagged for OPENJDK really means !ORACLE_JDK - or does it? It actually depends on what sources a given licensee has. Even for your custom build you might want some OPENJDK items and not others. I'm not sure there is a general solution, but using OPENJDK in combination with CUSTOM_SRC_DIR is, I think, more flexible than trying to define discrete variables that represent build targets. David On 17/04/2014 1:31 PM, Keith McGuigan wrote: On Wed, Apr 16, 2014 at 9:15 PM, David Holmes david.hol...@oracle.com mailto:david.hol...@oracle.com wrote: Hi Keith, On 17/04/2014 7:13 AM, Keith McGuigan wrote: Hello, I just added a comment to this bug -- see there for the details, but in short I'd like to update a number of tests in the makefiles that check OPENJDK and change them to check instead of the inverse definition of some new variable, such as ORACLEJDK. Please no! It's bad enough this is implicitly in the build without making it explicit! As I mentioned, I agree that moving all this to closed makefiles is the best solution (and is something that we could push for even if we took this partial step), but doing at least this step would
Re: JDK-8025705
On Thu, Apr 17, 2014 at 12:52 AM, David Holmes david.hol...@oracle.comwrote: Hi Keith, src/closed is Oracle's custom source location (hotspot calls it alt_src). If we never saw src/closed in the makefiles, only CUSTOM_SRC_DIR, and guarded with an existence test for a specific directory/file, then that should address your problem. That would be a first step in moving things to the custom makefiles where they belong. Though parameterizing the custom src dir would be nice, it's disjoint from the problem that I'm trying to solve (unless I'm misunderstanding you). The specific existence tests would do it, but that seems a rather slapdash solution. I'm opposed to the ORACLEJDK variable because I want to maintain the pressure to get this fixed properly. If we hack around it then it will never get cleaned up. Perfect is the enemy of good. A suboptimal solution here would still provide immediate benefits to third-party distributions. If there's some way we can take a step in this direction in the short-term, that would be great, especially since it make take a while to fix this permanently. I see 68 uses of src/closed across 14 files in the JDK repo. That seems tractable. Oh sure, it's tractable to make these changes -- what I was pointing out is that as of now (and until this is fixed), a non-OpenJDK, non-Oracle distribution needs to make private changes to work around this, and maintain those changes going forward, which is at the very least, unsavory. I think there are three things to be done here: 1. Replace all uses of src/closed with CUSTOM_SRC_DIR (similar to CUSTOM_MAKE_DIR) which in turn is set via configure 2. Guard all uses of CUSTOM_SRC_DIR in open makefiles with an existence check An existence check for the particular file that is being used? 3. Move all uses of CUSTOM_SRC_DIR to our closed makefiles Steps 1 and 2 can happen now. Step 3 is long term goal. --- The other problem I see with the OPENJDK, ORACLE_JDK, OTHER_JDK approach is that you actually have to deal with the permutations. Something currently flagged for OPENJDK really means !ORACLE_JDK - or does it? It actually depends on what sources a given licensee has. Even for your custom build you might want some OPENJDK items and not others. I'm not sure there is a general solution, but using OPENJDK in combination with CUSTOM_SRC_DIR is, I think, more flexible than trying to define discrete variables that represent build targets. I guess I still don't see what benefit using CUSTOM_SRC_DIR does other than move the literal src/closed (and make/closed) out of the makefiles and into configure. Wouldn't the existence of custom code still negate OPENJDK? Maybe I misunderstand what OPENJDK means -- I've been thinking of it as meaning that the build is a pure open build with no extra custom code. -- - Keith
