Re: Java9: 90 broken by -source/-target
 Aug 31, 2017 le 06:42:28PM +0200, Emmanuel Bourg êcrit: > Le 31/08/2017 à 14:20, Chris West a écrit : > > I don't think we should touch the compiler, or the peanut gallery will > scream in horror and complain that Debian's OpenJDK isn't Java. Let's > fix Ant first and see what's left. I think all the remaining problems hit either: * -Dant.build.javac.source: https://bugs.debian.org/873969 , or * https://bugs.debian.org/873977 But, who knows what we'll find with those fixed. > If a package calls javac or javadoc > directly (like junit [1]) a bug > can be raised right now. Done. It was only about 30.
Java9: 90 broken by -source/-target
The Java 9 games continue! There are around 90 packages which fail today, but which will succeed if the -source and -target flags are fixed. Here are the build logs / package list: https://rbuild.fau.xxx/2017-08-30/regress/ This is after the current round of ant and Maven changes. There's not a super strong correlation in the results. Some will be fixed by asking ant to force the javadoc source option to 1.6, like the javac option. e.g. https://rbuild.fau.xxx/2017-08-30/regress/opticalraytracer.log I did this test with this minimal patch: https://github.com/FauxFaux/debjdk9/blob/6298e520cb315d93d03872aa8e70b7b964f5e1eb/jdk/allow-old-source-versions.diff The System.err logging is lost in most cases, unfortunately; I suspect build systems typically drop it, or javac is being run as part of some abnormal ant build, or... We could... fiddle the version iff we're running inside dpkg-buildpackage, using environment detection again? Pretty ew. Not sure it would work for Ubuntu. Are we going to fix these all by hand? i.e. shall I raise 90 bugs, and then start changing packages? Or are we going to fix this in the compiler? Or does someone have an ant/environent suggestion? Cheers.
openjdk-9: FYI: Swing generics changes
As I can't think of anywhere else to complain... I've been looking at Java 9 build logs again[1]. These packages are broken by the addition of generics to a small number of swing classes: * electric * freeplane * jajuk * jsymphonic * libjaudiotagger * sweethome3d * zeroc-ice There may be others. I've raised a bug against this with Oracle, but I doubt they will look at it before the release. The bulk of the text is: https://paste.debian.net/982594/ There was some discussion on the mailing list three years ago, and a couple of people's broken code was fixed in the JDK (e.g. netbeans); I guess I missed the boat on this one: http://mail.openjdk.java.net/pipermail/jdk9-dev/2014-September/001295.html The errors are typically "cannot cast .. TreeNode" or related to "Enumeration<..> children()". This really doesn't feel like the kind of thing we should patch, and is rather ugly to fix in each package (although essentially simple). Sadness. :( In happier news, an AWS i3.8xlarge (32 cores, 244GB RAM, NVMe) can build nearly every default-jdk dependency (~1260 packages) in two hours; for <$3 on the spot market. -- Chris. 1: https://cryptpad.fr/code/#/1/edit/VH6giqhAylzK7Sa+aJiB5A/c01Z3XEbE0HCk2wOz4Xz8kKE/
openjdk-9-jdk rebuild: Third pass
Progress! * doko has uploaded openjdk-9 b177, which turns off some of the module checks, and fixes some other things * I've patched maven-javadoc-plugin (#866929), and doclint (#866908) locally, and both patches are probably okay * I've also hacked up javac to fixup -source and -target to 1.6, as we've still got issues with our hundreds of build systems. I'm not sure we can actually ship this patch[1], so we might additionally have a load of work fixing build systems to do. Now, of 1925 packages, we have a 30% failure rate. That's much more like it! Of the failures: 16% are getting "cannot find symbol" errors: * many of these (e.g. maven2-core) hit this only during generating javadocs, has something happened to javadoc classpath generation in a build system, or is there something wrong with the javadoc command? * the rest are probably caused by module changes, e.g. javax.activation vanishing See the `symbol` folder in the build logs. 15% are hitting encoding issues, despite the jh_build fix in #866845 See the `encoding` folder in the build logs. <10% of packages (each) seem to be failing due to: * `cast` becoming illegal (this might be fixable with a lower -source) * `deps` not matching (packaging bug?) * `javadoc` failing, probably some overlap with the symbol errors above * `keyword` not available in this language version (lower -source in some cases) * obvious problems with `modules` * `tests` failing * `version` some error messages about versions, but probably not relevant as javac is fixing it ... leaving just 125 packages with `unknown` problems, e.g. * build script tries to check jdk version and fails horribly (glugen) * assuming presence of tools.jar (aspectj-maven-plugin) * bugs in the builder * failures in the builder that probably wouldn't happen on the buildd * packages that are broken anyway (diffoscope) * eclipse * name clashes (maven-war-plugin) Doesn't look like anything more to fix in the toolchain, so, er... Time for a mass bug filing (~300)? Perhaps less useful as the majority of the packages are maintained by people reading this mail. Build logs: https://rbuild.fau.xxx/2017-07-07/ My personal favourites from this round: * The doclint change is in a file named WorkArounds.java[2], which is a copy-paste of another file, added in a commit named "Merge", very shortly before the "implement modules" megacommit. * libbluray is broken due to the addition of an abstract method (i.e. no default) to the java.io.FileSystem[3] interface, which I never liked. Chris. 1: https://github.com/FauxFaux/debjdk9/blob/master/jdk/allow-old-source-versions.diff 2: https://github.com/FauxFaux/jdk9-flat/blob/f3edcfbd7946e21be277edf33e7b1d8000516c36/langtools/src/jdk.javadoc/share/classes/jdk/javadoc/internal/doclets/toolkit/WorkArounds.java#L114 3: https://github.com/FauxFaux/jdk9-flat/blame/f3edcfbd7946e21be277edf33e7b1d8000516c36/jdk/src/java.base/share/classes/java/io/FileSystem.java#L215
Re: openjdk-9-jdk rebuild: Second pass
On Tue, Jul 04, 2017 at 10:28:27AM +0100, Chris West wrote: > > > 16% are broken due to probably intentional changes in Java 9, e.g. > > > * modules changing accessiblity of things (~90 packages) > > I now believe that the accessibility checks are being turned off by default. > This was one of the conditions of the first JCP vote failing. So it's possible > that many of these will go away with a new (possibly as-yet unreleased) > openjdk > build. I have been unable to find a relevant upstream commit, although > there's plenty it could be in, e.g. > http://hg.openjdk.java.net/jdk9/jdk9/jdk/rev/f8b19df2115a [two weeks ago] > > Discussion: > http://mail.openjdk.java.net/pipermail/jigsaw-dev/2017-May/012673.html > http://openjdk.java.net/projects/jigsaw/spec/minutes/2017-05-22 > > https://londonjavacommunity.wordpress.com/2017/06/28/our-yes-vote-on-the-2nd-go-around-for-jsr-376-java-platform-module-system/ > > Agreement on relaxing Strong encapsulation as a default (means fewer > > apps will break out of the box, but get a warning instead). > http://zulu.org/forum/thread/discussion-on-zulu-9-pre-release-13/ says that is has landed, so will be picked up by an update of the openjdk-9 package to build >=176 (2017-06-29).
Re: openjdk-9-jdk rebuild: Second pass
On Tue, Jul 04, 2017 at 09:42:23AM +0200, Emmanuel Bourg wrote: > > 7% are broken because invalid ASCII in the code is now an error; > >it appears that it was only a warning on Java 8. > > This is now fixed for the packages built with javahelper and jh_build > (#866845). I wonder if these can all be fixed by fiddling the version, and whether doing so would be a good idea. For -source of 1.8(?), the error is a warning. It's not to do with the version of javac in use, as I previously claimed. Doing so is probably harder than fixing the locale everywhere, though. Changing the -source argument will also allow us to have libraries use _ as an identifier, for now, so that may be an easier fix? Also, I've been told that, with future Java releaseses, old -source values will continue to be dropped; so Java 10 will drop -source 1.6, and Java 11 will drop -source 1.7, etc. > > 16% are broken due to probably intentional changes in Java 9, e.g. > > * modules changing accessiblity of things (~90 packages) I now believe that the accessibility checks are being turned off by default. This was one of the conditions of the first JCP vote failing. So it's possible that many of these will go away with a new (possibly as-yet unreleased) openjdk build. I have been unable to find a relevant upstream commit, although there's plenty it could be in, e.g. http://hg.openjdk.java.net/jdk9/jdk9/jdk/rev/f8b19df2115a [two weeks ago] Discussion: http://mail.openjdk.java.net/pipermail/jigsaw-dev/2017-May/012673.html http://openjdk.java.net/projects/jigsaw/spec/minutes/2017-05-22 https://londonjavacommunity.wordpress.com/2017/06/28/our-yes-vote-on-the-2nd-go-around-for-jsr-376-java-platform-module-system/ > Agreement on relaxing Strong encapsulation as a default (means fewer > apps will break out of the box, but get a warning instead). Chris.
openjdk-9-jdk rebuild: Second pass
Progress! * Better tools[2], better scripts[3], better machine -> more packages. * Emmanuel fixed maven/guice (#866411), and hacked ant (1.9.9-4) to fixup the -source and -target versions. (Thanks!) * I totally hacked up[1] maven-compiler-plugin to do the same. Of 1261 packages compiled, 57% failed. Of those failures: 23% think javadoc(1) doesn't exist: http://bugs.debian.org/bug=866929 I definitely think this should be fixed in the toolchain; as the above bug. 18% still are trying to use old -source/-target: https://paste.debian.net/974407/ Perhaps some cases ant doesn't catch it, as it doesn't look like a Debian build environment? e.g. svnkit: https://sources.debian.net/src/svnkit/1.8.14-1/debian/rules/#L9 https://paste.debian.net/974409/ If we can't fix enough of these in ant/maven, we might have to fix them in javac... 11% (plus probably a lot of the above 23%) are failing doclint after otherwise compiling successfully. doclint should be disabled, but isn't: http://bugs.debian.org/866908 Hopefully everyone agrees we should fix this in the toolchain, as the above bug. 7% are broken because invalid ASCII in the code is now an error; it appears that it was only a warning on Java 8. e.g. https://reproducible.debian.net/naga has: [...] warning: unmappable character for encoding ASCII ... which is now: [...] error: unmappable character (0xC3) for encoding US-ASCII Logs: https://rbuild.fau.xxx/2017-07-02/ascii/ Maybe many of these may be fixed by defaulting the encoding, where not specified, to UTF-8? Not t many to fix otherwise... And, on a more positive note: 16% are broken due to probably intentional changes in Java 9, e.g. * introduction of java.lang.Module * modules changing accessiblity of things (~90 packages) * _ becoming a keyword (~15 packages) * apparent generics changes causing new casting problems(?) These are in the `modules`, `keyword` and `cast` directories of the build logs: https://rbuild.fau.xxx/2017-07-02/ 6% are broken due to tests or dependencies missing. I don't know why these are failing on my build env, but not on jenkins, and I suggest you ignore them. They are in `deps` and `tests` in https://rbuild.fau.xxx/2017-07-02/ This leaves just 19% of these packages (135) with random failures for which I haven't bothered to find a classification: https://rbuild.fau.xxx/2017-07-02/unknown/ Chris. 1: https://github.com/FauxFaux/debjdk9/blob/master/compiler-plugin.patch 2: https://paste.debian.net/974394/ 3: https://github.com/FauxFaux/debjdk9
Looking at an archive rebuild with opendjk-9-jdk
I rebuilt ~300 packages[1] which build-depend on default-jdk in a hacked-up[2] "chroot" which uses openjdk-9-jdk, in place of openjdk-8-jdk. (i.e. custom java-common build + some Provides: hacks.) Most things fail: 87% failures. ~41% are hitting a bug in Maven, e.g. argparse4j: dh_auto_build -O--buildsystem=maven /usr/lib/jvm/default-java/bin/java -noverify -cp /usr/[...] [ERROR] Error executing Maven. [ERROR] java.lang.IllegalStateException: Unable to load cache item [ERROR] Caused by: Unable to load cache item [ERROR] Caused by: Could not initialize class com.google.inject.internal.cglib.core.$ReflectUtils There's an old mailing list post about this, but it doesn't seem to have a resolution beyond "works for me!": https://www.mail-archive.com/jigsaw-dev@openjdk.java.net/msg06073.html Someone more aware of what versions of things we're running, and how they were built, should investigate this. ~30% are using an outdated -source or -target, e.g. dom4j: [javac] error: Source option 1.3 is no longer supported. Use 1.6 or later. [javac] error: Target option 1.3 is no longer supported. Use 1.6 or later. We discussed on IRC perhaps patching javac (or ant/maven) to just use 1.6 here, and pray. Annoying to try, as neither ant nor maven build in my "chroot". Does anyone have a less horrifying suggestion, beyond patching 30% of the libraries in Debian? ~8% are hitting a bug in gradle, e.g. gant: Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make protected java.lang.Package[] java.lang.ClassLoader.getPackages() accessible: module java.base does not "opens java.lang" to unnamed module @14fc5f04 https://github.com/gradle/gradle/issues/1095 suggests it's fixed in 4.1 nightly, as of a fortnight ago. I guess we just wait for upstream to fix this one, and the new gradle to make it into sid. The remaining failures are much more random: * eclipse stuff has weird failures, e.g. eclipse-cdt fails to find some files[3] * broken dependencies (i.e. probably not caused by jdk9, or caused by my build env), e.g. dogtag-pki * javadoc is more picky about some things, e.g. imports of sealed packages (hacked ant), [javadoc] /build/classycle-1.4.2/src/classycle/ant/ReportTask.java:97: error: no summary or caption for table * bnd actually fails due to a jdk9 reason, it uses _ as an identifier! * Full list (without reasons): https://paste.debian.net/973904/ I suggest we attempt to address these three major issues before trying a bigger rebuild. Chris. 1: Processed dose output: https://paste.debian.net/973901/ 2: https://github.com/FauxFaux/debjdk9 3: https://paste.debian.net/973906/
maven-debian-helper=2.0~exp2 rebuild
I did a rebuild of a bunch of packages with maven-debian-helper 2.0~exp2 installed, to see what would break. The resultant build logs are in https://rbuild.fau.xxx/2015-11-25-mdh-2-2/ , and the failures are sorted in the sorted/ subdirectory. Summary: 38 install-plugin-missing 10 other-missing-plugins 8 antrun-missing (plugin) 6 other-compile-failure 5 other-missing-artifacts 5 compiler-api-missing 4 multimodule-project-dir 3 test-failures 3 scm-publish-plugin 3 maven2-settings-debian 3 builder-failure 2 invalid-plugin-descriptor Missing plugins could be one of two things: 1) I believe maven-debian-helper Requires maven2 Recommends maven-install-plugin, which the builders may previously have been installing, and this round has not been? That is, the packages were possibly broken anyway, and it just happens that this build has caught them. 2) It is believed that maven3 requires some plugins present, even if they're not used during the build, whereas maven 2 would not. Excluding plugins and probably-unrelated-FTBFS, we're left with: maven2-settings-debian, which is probably a bug in these packages, e.g. https://sources.debian.net/src/spatial4j/0.3-1/debian/rules/#L10 dh_auto_configure mh_patchpoms -plibspatial4j-java --debian-build --keep-pom-version --maven-repo=/spatial4j-0.3/debian/maven-repo mvn install:install-file -s/etc/maven2/settings-debian.xml -Dmaven.repo.local=`pwd`/debian/maven-repo -Dfile=/usr/share/java/jts.jar -DgroupId=com.vividsolutions -DartifactId=jts -Dversion=fake -Dpackaging=jar [ERROR] Error executing Maven. [ERROR] The specified user settings file does not exist: /etc/maven2/settings-debian.xml multimodule-project-dir, which I don't get at all but causes later failures: dh_testdir mvn-debian package -Dmaven.multiModuleProjectDirectory system property is not set. Check $M2_HOME environment variable and mvn script match. touch build-stamp debian/rules binary i.e. somewhere between 7 and 20 packages, if some maven plugins are added to maven-debian-helper's Depends.
Re: OpenJDK 8 transition
I did a practice partial archive rebuild with ebourg's 1.8 default-jdk, to see what kind of problems might be left. 1144 packages[1] that dose-debuildcheck thinks depend on default-jdk, around ten unexpected/new failures. My full notes are on whiteboard[2], with a backup at time of sending[3]. Final build logs; anything marked as "failure" here failed multiple times: https://rbuild.fau.xxx/2015-08-31/ Probably broken by jdk8: default-jdk doesn't have /usr/lib/jvm/java-7-openjdk-amd64/lib/tools.jar: * biojava3-live * biojava4-live Refuses to run under jdk8: * netbeans fatal error: jni_md.h: No such file or directory * libnb-platform18-java "cannot access" errors are, I think, from -source 1.5: * xmlbeans error: cannot access TypeStoreUser * lucene4.10 error: cannot access SelfDescribing * lucene4 error: cannot access SelfDescribing * sweethome3d-furniture-editor error: cannot access Localizable * freemind error: cannot access Opcodes * gnome-split @Override And, not so sure about these: * jaxb is not abstract and does not override abstract method getAnnotationsByType(Class) in AnnotatedConstruct * jing-trang Annotations cannot be converted to AnnotationsImpl * maven-stapler-plugin javadoc: error - option -locale must be first on the command line. On top of these, there's: * Eclipse things (which we know is being worked on) * 15 packages which are probably broken due to incompatability with the builder, or flaky test sets, which should probably be re-tested. * 15 packages which need more time or memory or.. than the builder was willing to give them, and should probably be tested. * ~17 packages with RC bugs already * ~15 packages with other build problems Some of these may have already been fixed; been doing this for a few days, on and off. Chris. (I'm on-list but much prefer IRC; #debian-java @ oftc.) 1: https://paste.debian.net/310020/ 2: http://whiteboard.debian.net/a69515.txt 3: https://paste.debian.net/310023/
