(top posting) I did some preliminary tests on my local machine, invoking "mvn clean install -Pgwt-admin" command to build (only) WebAdmin from source.
Default permutation settings: all (5) browsers, one locale = 5 total permutations to compile Tested two different strategies (machine with 16G RAM / 8G swap, quad-core CPU): A, localWorkers=X (default external-JVM worker factory) -Dgwt.compiler.localWorkers=X B, localWorkers=X + maxThreads=X (thread-based worker factory) -Dgwt.compiler.localWorkers=X -Dgwt.jjs.permutationWorkerFactory=com.google.gwt.dev.ThreadedPermutationWorkerFactory -Dgwt.jjs.maxThreads=X A -> this is what we use today B -> this is possible alternative X=4 [5 perms] A, initial 2.6 GiB, peak 13.5 GiB Total time: 5:25.851s B, initial 2.6 GiB, peak 13.1 GiB Total time: 5:11.238s X=1 [5 perms] A, initial 2.6 GiB, peak 7.1 GiB Total time: 7:00.601s B, initial 2.6 GiB, peak 7.4 GiB Total time: 7:07.458s Now switched to permutation settings: all (5) browsers, 3 locales = 15 total permutations to compile (Used swapoff/swapon to force swap clear after each test run.) X=4 [15 perms] A, initial 2.6 GiB, peak physical max + swap ~4.3 GiB Total time: 11:39.086s B, initial 2.6 GiB, peak physical max + swap ~3.5 GiB Total time: 10:59.966s X=1 [15 perms] A, initial 2.6 GiB, peak 7.7 GiB Total time: 17:39.103s B, initial 2.6 GiB. peak 7.9 GiB Total time: 17:16.314s For X=1, both strategies (A & B) have very similar results. For X=4 (more than one worker), thread-based strategy seems slightly better, but still we have huge RAM consumption + max opened files quite high. I think the only real solution is to cut off (eliminate) locale permutation vector to reduce amount of GWT permutations. It's on my task list. Vojtech ----- Original Message ----- > From: "Sandro Bonazzola" <sbona...@redhat.com> > To: "Vojtech Szocs" <vsz...@redhat.com> > Cc: "Tomas Jelinek" <tjeli...@redhat.com>, "Juan Hernandez" > <jhern...@redhat.com>, "Kiril Nesenko" > <knese...@redhat.com>, "arch" <arch@ovirt.org> > Sent: Wednesday, February 26, 2014 11:58:29 AM > Subject: Re: [urgent] Re: ovirt-engine build failure > > Il 26/02/2014 11:51, Vojtech Szocs ha scritto: > > > > > > ----- Original Message ----- > >> From: "Sandro Bonazzola" <sbona...@redhat.com> > >> To: "Tomas Jelinek" <tjeli...@redhat.com> > >> Cc: "Vojtech Szocs" <vsz...@redhat.com>, "Juan Hernandez" > >> <jhern...@redhat.com>, "Kiril Nesenko" > >> <knese...@redhat.com>, "arch" <arch@ovirt.org> > >> Sent: Wednesday, February 26, 2014 10:22:22 AM > >> Subject: Re: [urgent] Re: ovirt-engine build failure > >> > >> Il 26/02/2014 09:06, Tomas Jelinek ha scritto: > >>> Hey, > >>> > >>> from this logs it seems that the root cause is: > >>> [ERROR] java.io.FileNotFoundException: > >>> /home/jenkins/workspace/manual-build-tarball/label/centos6-host/rpmbuild/BUILD/ovirt-engine-3.5.0/frontend/webadmin/modules/webadmin/target/tmp/ImageResourceGenerator4201639982561998434.png > >>> (Too many open files) > >>> > >>> e.g. the OS does not allow to open all the files the gwt compilation > >>> needs. > >>> Could you please try to increase the limit of max opened files on the > >>> host > >>> and try to rebuild? > >> > >> David is increasing max open files. We'll give it another run. > >> I really think that having to reduce localWorkers to 1 for having decent > >> build requirements on RAM and still hit too many opened files is really > >> bad. > >> Any chance of lowering build requirements in terms of RAM and opened > >> files? > > > > In general, GWT compiler generates/reads lots of intermediate files during > > its runtime. IIRC, this file amount is proportional to number of > > permutations > > being compiled, i.e. num_of_browsers x num_of_locales. This is why we don't > > hit "Too many open files" in small-scale builds (i.e. development) but we > > do > > hit it on large-scale builds (i.e. all_browsers x all_locales nightly > > build). > > > > But is increasing max open files for GWT compiler really that big of a > > deal? > > (i.e. we know that compiler needs to process lots of files per each > > permutation) > > No, it's not that big of a deal *for our packaging* but since we've people on > Gentoo building engine > on their own systems, it may impact all of them... > > > > > Alternatively, we could implement "remote GWT worker grid" -> offload > > compilation > > for each specific (browser x locale) permutation to a remote machine, this > > would > > require some extra Java implementation though. > > > > I agree on excessive RAM usage, I'm currently experimenting with some > > compiler > > options, I'll share results when I have something. > > > > Another approach we could take (something I'll discuss with my peers) is > > the > > idea of eliminating locale vector entirely -> re-implement i18n support in > > UI > > in a way that for X browsers, there will be only X total permutations (i.e. > > not X x num_of_locales). This would reduce RAM usage && compilation time && > > perhaps max files opened too. > > +1 on whatever can reduce build time and build requirements. > > > > > >> > >> > >>> > >>> ----- Original Message ----- > >>>> From: "Sandro Bonazzola" <sbona...@redhat.com> > >>>> To: "Vojtech Szocs" <vsz...@redhat.com> > >>>> Cc: "Juan Hernandez" <jhern...@redhat.com>, "Kiril Nesenko" > >>>> <knese...@redhat.com>, "arch" <arch@ovirt.org> > >>>> Sent: Wednesday, February 26, 2014 8:37:07 AM > >>>> Subject: Re: [urgent] Re: ovirt-engine build failure > >>>> > >>>> Il 25/02/2014 16:52, Vojtech Szocs ha scritto: > >>>>> > >>>>> > >>>>> ----- Original Message ----- > >>>>>> From: "Vojtech Szocs" <vsz...@redhat.com> > >>>>>> To: "Sandro Bonazzola" <sbona...@redhat.com> > >>>>>> Cc: "Juan Hernandez" <jhern...@redhat.com>, "Kiril Nesenko" > >>>>>> <knese...@redhat.com>, "arch" <arch@ovirt.org> > >>>>>> Sent: Tuesday, February 25, 2014 1:05:35 PM > >>>>>> Subject: Re: [urgent] Re: ovirt-engine build failure > >>>>>> > >>>>>> > >>>>>> > >>>>>> ----- Original Message ----- > >>>>>>> From: "Sandro Bonazzola" <sbona...@redhat.com> > >>>>>>> To: "Vojtech Szocs" <vsz...@redhat.com> > >>>>>>> Cc: "Juan Hernandez" <jhern...@redhat.com>, "arch" <arch@ovirt.org>, > >>>>>>> "Kiril > >>>>>>> Nesenko" <knese...@redhat.com> > >>>>>>> Sent: Tuesday, February 25, 2014 9:02:11 AM > >>>>>>> Subject: Re: [urgent] Re: ovirt-engine build failure > >>>>>>> > >>>>>>> Il 25/02/2014 08:47, Sandro Bonazzola ha scritto: > >>>>>>>> Il 24/02/2014 18:28, Vojtech Szocs ha scritto: > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> ----- Original Message ----- > >>>>>>>>>> From: "Sandro Bonazzola" <sbona...@redhat.com> > >>>>>>>>>> To: "Vojtech Szocs" <vsz...@redhat.com> > >>>>>>>>>> Cc: "Eyal Edri" <ee...@redhat.com>, "Juan Hernandez" > >>>>>>>>>> <jhern...@redhat.com>, "Kiril Nesenko" <knese...@redhat.com>, > >>>>>>>>>> "arch" <arch@ovirt.org> > >>>>>>>>>> Sent: Monday, February 24, 2014 9:08:09 AM > >>>>>>>>>> Subject: [urgent] Re: ovirt-engine build failure > >>>>>>>>>> > >>>>>>>>>> Top posting, > >>>>>>>>>> > >>>>>>>>>> Please give this task really urgent priority, as you can see: > >>>>>>>>>> > >>>>>>>>>> http://jenkins.ovirt.org/job/ovirt-engine_3.4_create_rpms/ > >>>>>>>>>> and > >>>>>>>>>> http://jenkins.ovirt.org/job/ovirt-engine_master_create_rpms/ > >>>>>>>>>> > >>>>>>>>>> are failing consistently so we don't have nightly builds. > >>>>>>>>> > >>>>>>>>> (CC'ing Einav) > >>>>>>>>> > >>>>>>>>> Attached TEST patch, please apply & rebuild ovirt-engine as usual. > >>>>>>>>> > >>>>>>>>> This patch should cause following output during GWT compilation: > >>>>>>>>> > >>>>>>>>> [ERROR] ######### Util.readURLAsBytes > >>>>>>>>> file:/home/vszocs/work/ovirt-engine/frontend/webadmin/modules/webadmin/target/generated-sources/gwt/org/ovirt/engine/ui/webadmin/images/search_button.png > >>>>>>>>> > >>>>>>>>> If IOException occurs, exception stacktrace will be printed right > >>>>>>>>> below > >>>>>>>>> that line. > >>>>>>>>> > >>>>>>>>> Note: please ignore the [ERROR] prefix, it's a technical issue, > >>>>>>>>> more > >>>>>>>>> details at > >>>>>>>>> https://www.mail-archive.com/engine-devel@ovirt.org/msg05735.html > >>>>>>>>> > >>>>>>>>> I just did a local upstream build with TEST patch applied, GWT > >>>>>>>>> compilation > >>>>>>>>> passed OK. > >>>>>>>> > >>>>>>>> build job running with your patch here: > >>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/235/ > >>>>>>>> waiting for a free executor right now. > >>>>>>>> Let's see how it goes. > >>>>>>> > >>>>>>> Failed on CentOS: > >>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/235/label=centos6-host/consoleFull > >>>>>> > >>>>>> Hm, in that ^^ build, following exceptions occured: > >>>>>> > >>>>>> 1. > >>>>>> [INFO] java.lang.NullPointerException > >>>>>> [INFO] at > >>>>>> com.google.gwt.dev.util.Util.computeStrongName(Util.java:172) > >>>>>> > >>>>>> 2. > >>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever computed for > >>>>>> downDisabledImage > >>>>>> [INFO] at > >>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475 > >>>>>> > >>>>>> 1. is caused by Util#readURLAsBytes but I don't see any IOException in > >>>>>> the > >>>>>> logs > >>>>>> 2. is just follow-up consequence of 1. (i.e. image metadata missing in > >>>>>> GWT > >>>>>> image cache) > >>>>>> > >>>>>> Now I noticed I missed one possible IOException inside > >>>>>> Util#readURLConnectionAsBytes ... > >>>>>> > >>>>>> Long story short, I'll send updated TEST patch soon, apologies for the > >>>>>> inconvenience. > >>>>> > >>>>> TEST-v2 patch attached. Please apply & rebuild ovirt-engine as usual. > >>>> > >>>> Done, here is the console output: > >>>> http://jenkins.ovirt.org/job/manual-build-tarball/label=centos6-host/239/consoleFull > >>>> > >>>> > >>>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Il 20/02/2014 13:34, Vojtech Szocs ha scritto: > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> ----- Original Message ----- > >>>>>>>>>>>> From: "Sandro Bonazzola" <sbona...@redhat.com> > >>>>>>>>>>>> To: "Vojtech Szocs" <vsz...@redhat.com> > >>>>>>>>>>>> Cc: "Eyal Edri" <ee...@redhat.com>, "Juan Hernandez" > >>>>>>>>>>>> <jhern...@redhat.com>, "Kiril Nesenko" <knese...@redhat.com>, > >>>>>>>>>>>> "arch" <arch@ovirt.org> > >>>>>>>>>>>> Sent: Thursday, February 20, 2014 10:02:00 AM > >>>>>>>>>>>> Subject: Re: ovirt-engine build failure > >>>>>>>>>>>> > >>>>>>>>>>>> Il 19/02/2014 16:51, Vojtech Szocs ha scritto: > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> ----- Original Message ----- > >>>>>>>>>>>>>> From: "Sandro Bonazzola" <sbona...@redhat.com> > >>>>>>>>>>>>>> To: "Vojtech Szocs" <vsz...@redhat.com> > >>>>>>>>>>>>>> Cc: "Eyal Edri" <ee...@redhat.com>, "Juan Hernandez" > >>>>>>>>>>>>>> <jhern...@redhat.com>, "Kiril Nesenko" <knese...@redhat.com>, > >>>>>>>>>>>>>> "arch" <arch@ovirt.org> > >>>>>>>>>>>>>> Sent: Wednesday, February 19, 2014 2:43:44 PM > >>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Il 19/02/2014 14:41, Vojtech Szocs ha scritto: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> ----- Original Message ----- > >>>>>>>>>>>>>>>> From: "Sandro Bonazzola" <sbona...@redhat.com> > >>>>>>>>>>>>>>>> To: "Vojtech Szocs" <vsz...@redhat.com> > >>>>>>>>>>>>>>>> Cc: "Eyal Edri" <ee...@redhat.com>, "Juan Hernandez" > >>>>>>>>>>>>>>>> <jhern...@redhat.com>, "Kiril Nesenko" > >>>>>>>>>>>>>>>> <knese...@redhat.com>, > >>>>>>>>>>>>>>>> "arch" <arch@ovirt.org> > >>>>>>>>>>>>>>>> Sent: Wednesday, February 19, 2014 2:23:16 PM > >>>>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Il 19/02/2014 14:19, Vojtech Szocs ha scritto: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> ----- Original Message ----- > >>>>>>>>>>>>>>>>>> From: "Sandro Bonazzola" <sbona...@redhat.com> > >>>>>>>>>>>>>>>>>> To: "Vojtech Szocs" <vsz...@redhat.com> > >>>>>>>>>>>>>>>>>> Cc: "Eyal Edri" <ee...@redhat.com>, "Juan Hernandez" > >>>>>>>>>>>>>>>>>> <jhern...@redhat.com>, "Kiril Nesenko" > >>>>>>>>>>>>>>>>>> <knese...@redhat.com>, > >>>>>>>>>>>>>>>>>> "arch" <arch@ovirt.org> > >>>>>>>>>>>>>>>>>> Sent: Wednesday, February 19, 2014 1:18:28 PM > >>>>>>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Il 19/02/2014 12:45, Vojtech Szocs ha scritto: > >>>>>>>>>>>>>>>>>>> Hi, I checked GWT sources: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever > >>>>>>>>>>>>>>>>>>>> computed > >>>>>>>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>>>>> removeBrickCommitRequired > >>>>>>>>>>>>>>>>>>>> [INFO] at > >>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475) > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> This happened because internal image cache for bundled > >>>>>>>>>>>>>>>>>>> (composite > >>>>>>>>>>>>>>>>>>> PNG) > >>>>>>>>>>>>>>>>>>> images was corrput. > >>>>>>>>>>>>>>>>>>> It seems to be caused by disk I/O error while reading > >>>>>>>>>>>>>>>>>>> file > >>>>>>>>>>>>>>>>>>> images/glusterAsyncTasks/remove_brick_commit_required.png > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException > >>>>>>>>>>>>>>>>>>>> [INFO] at > >>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40) > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> This happened because image data was null, again possible > >>>>>>>>>>>>>>>>>>> disk > >>>>>>>>>>>>>>>>>>> I/O > >>>>>>>>>>>>>>>>>>> error > >>>>>>>>>>>>>>>>>>> while reading image file. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Can we enable "TRACE" logging during GWT compilation? > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> -Dgwt.logLevel=TRACE > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Building master right now with your suggestions here: > >>>>>>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/ > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Looking at > >>>>>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=fedora20-host/console > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> So far I see no errors.. There should be total 5 browsers x > >>>>>>>>>>>>>>>>> 8 > >>>>>>>>>>>>>>>>> locales > >>>>>>>>>>>>>>>>> = > >>>>>>>>>>>>>>>>> 40 > >>>>>>>>>>>>>>>>> GWT permutations. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> So no changes with and without > >>>>>>>>>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera. > >>>>>>>>>>>>>>>> We've still 40 permutations. Maybe you defined > >>>>>>>>>>>>>>>> userAgent=ie8,ie9,gecko1_8,safari,opera somewhere in the POM > >>>>>>>>>>>>>>>> files? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> You are right. I checked frontend/webadmin/modules/pom.xml -> > >>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>> "gwt.userAgent" > >>>>>>>>>>>>>>> property is copied into "locale" GWT property in > >>>>>>>>>>>>>>> {WebAdmin,UserPortal}.gwt.xml > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Ok, so I'll remove > >>>>>>>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera > >>>>>>>>>>>>>> from > >>>>>>>>>>>>>> build > >>>>>>>>>>>>>> command. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Agreed. > >>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> This: > >>>>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=centos6-host/console > >>>>>>>>>>>>>>>> failed, you can look at it, it has -Dgwt.logLevel=TRACE -X. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Hm, it contains: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> [DEBUG] (f) logLevel = INFO > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Somehow the TRACE log level for GWT wasn't applied.. not sure > >>>>>>>>>>>>>>> why. > >>>>>>>>>>>>>>> Let > >>>>>>>>>>>>>>> me > >>>>>>>>>>>>>>> check > >>>>>>>>>>>>>>> this on local build. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Ok, let me know if I've to change command line. > >>>>>>>>>>>>> > >>>>>>>>>>>>> I wrote upstream patch: http://gerrit.ovirt.org/#/c/24789/ > >>>>>>>>>>>>> > >>>>>>>>>>>>> When it's merged, setting "gwt.logLevel" will work (as you did > >>>>>>>>>>>>> before) > >>>>>>>>>>>>> and we'll be able to see more details in the build log. > >>>>>>>>>>>> > >>>>>>>>>>>> Here is the new build with your patch merged: > >>>>>>>>>>>> > >>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/228/label=centos6-host/consoleFull > >>>>>>>>>>> > >>>>>>>>>>> Thanks. I did some more investigation on this. > >>>>>>>>>>> > >>>>>>>>>>> Again it's NullPointerException when rebinding > >>>>>>>>>>> ApplicationResources, > >>>>>>>>>>> but > >>>>>>>>>>> this time for different image file: > >>>>>>>>>>> ApplicationResources.wrenchImage > >>>>>>>>>>> ("images/wrench.png"). > >>>>>>>>>>> > >>>>>>>>>>> Looking at stacktrace & GWT sources, the problematic code seems > >>>>>>>>>>> to > >>>>>>>>>>> be > >>>>>>>>>>> (AbstractResourceContext:74): > >>>>>>>>>>> > >>>>>>>>>>> byte[] bytes = Util.readURLAsBytes(resource); > >>>>>>>>>>> > >>>>>>>>>>> where "resource" is (ImageResourceGenerator:311): > >>>>>>>>>>> > >>>>>>>>>>> localized.getUrl(); // LocalizedImage localized > >>>>>>>>>>> > >>>>>>>>>>> Note: in oVirt UI we don't use localized images (i.e. > >>>>>>>>>>> "wrench_fr_FR.png") > >>>>>>>>>>> so "localized" in this case points to "wrench.png". > >>>>>>>>>>> > >>>>>>>>>>> So the problem seems to be reading this file: > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> frontend/webadmin/modules/webadmin/src/main/resources/org/ovirt/engine/ui/webadmin/images/wrench.png > >>>>>>>>>>> > >>>>>>>>>>> Since we're having many GWT permutations, this file ^^ (along > >>>>>>>>>>> with > >>>>>>>>>>> similar > >>>>>>>>>>> ones) is attempted to be read multiple times *from the same > >>>>>>>>>>> (single) > >>>>>>>>>>> GWT > >>>>>>>>>>> compiler Java process, before actual permutation worker Java > >>>>>>>>>>> sub-processes > >>>>>>>>>>> are spawned*. (More precisely, above NullPointerException happens > >>>>>>>>>>> at > >>>>>>>>>>> deferred binding phase which precedes actual GWT compilation per > >>>>>>>>>>> each > >>>>>>>>>>> permutation.) > >>>>>>>>>>> > >>>>>>>>>>> I think the root cause is Util.readURLAsBytes method used > >>>>>>>>>>> internally > >>>>>>>>>>> by > >>>>>>>>>>> GWT > >>>>>>>>>>> compiler: > >>>>>>>>>>> > >>>>>>>>>>> public static byte[] readURLAsBytes(URL url) { > >>>>>>>>>>> try { > >>>>>>>>>>> URLConnection conn = url.openConnection(); > >>>>>>>>>>> conn.setUseCaches(false); > >>>>>>>>>>> return readURLConnectionAsBytes(conn); > >>>>>>>>>>> } catch (IOException e) { > >>>>>>>>>>> return null; // This causes NullPointerException, but this > >>>>>>>>>>> exception > >>>>>>>>>>> isn't logged (wtf, GWT!) > >>>>>>>>>>> } > >>>>>>>>>>> } > >>>>>>>>>>> > >>>>>>>>>>> To summarize, I'll need to write another patch which replaces > >>>>>>>>>>> Util.readURLAsBytes implementation via AOP and prints IOException > >>>>>>>>>>> details. > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> # > >>>>>>>>>>>>>>>>>> CI_MAVEN_SETTINGS=$WORKSPACE/artifactory-ovirt-org-settings.xml > >>>>>>>>>>>>>>>>>> # export MAVEN_OPTS="${MAVEN_OPTS} -Xmx2048M > >>>>>>>>>>>>>>>>>> -XX:MaxPermSize=1024m > >>>>>>>>>>>>>>>>>> -Dgwt.compiler.localWorkers=1" > >>>>>>>>>>>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -ts "${TARBALL}" > >>>>>>>>>>>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -D > >>>>>>>>>>>>>>>>>> "ovirt_build_extra_flags > >>>>>>>>>>>>>>>>>> -gs > >>>>>>>>>>>>>>>>>> ${CI_MAVEN_SETTINGS} > >>>>>>>>>>>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera > >>>>>>>>>>>>>>>>>> -Dgwt.logLevel=TRACE -X" --rebuild > >>>>>>>>>>>>>>>>>> rpmbuild/SRPMS/*.src.rpm > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Let me know if you need anything else. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Vojtech > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> ----- Original Message ----- > >>>>>>>>>>>>>>>>>>>> From: "Vojtech Szocs" <vsz...@redhat.com> > >>>>>>>>>>>>>>>>>>>> To: "Sandro Bonazzola" <sbona...@redhat.com> > >>>>>>>>>>>>>>>>>>>> Cc: "Eyal Edri" <ee...@redhat.com>, "Juan Hernandez" > >>>>>>>>>>>>>>>>>>>> <jhern...@redhat.com>, "Kiril Nesenko" > >>>>>>>>>>>>>>>>>>>> <knese...@redhat.com>, > >>>>>>>>>>>>>>>>>>>> "arch" <arch@ovirt.org> > >>>>>>>>>>>>>>>>>>>> Sent: Tuesday, February 18, 2014 6:54:20 PM > >>>>>>>>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> In your log I see following error: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> ... > >>>>>>>>>>>>>>>>>>>> [INFO] Computing all possible rebind results for > >>>>>>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' > >>>>>>>>>>>>>>>>>>>> [INFO] Rebinding > >>>>>>>>>>>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources > >>>>>>>>>>>>>>>>>>>> [INFO] Invoking generator > >>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator > >>>>>>>>>>>>>>>>>>>> [INFO] [ERROR] Generator > >>>>>>>>>>>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator' > >>>>>>>>>>>>>>>>>>>> threw > >>>>>>>>>>>>>>>>>>>> an exception while rebinding > >>>>>>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' > >>>>>>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever > >>>>>>>>>>>>>>>>>>>> computed > >>>>>>>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>>>>> removeBrickCommitRequired > >>>>>>>>>>>>>>>>>>>> [INFO] at > >>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475) > >>>>>>>>>>>>>>>>>>>> ... > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> I checked again logs at > >>>>>>>>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/212/label=fedora19-host/consoleFull > >>>>>>>>>>>>>>>>>>>> and indeed there is a similar error: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> ... > >>>>>>>>>>>>>>>>>>>> [INFO] Computing all possible rebind results for > >>>>>>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' > >>>>>>>>>>>>>>>>>>>> [INFO] Rebinding > >>>>>>>>>>>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources > >>>>>>>>>>>>>>>>>>>> [INFO] Invoking generator > >>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator > >>>>>>>>>>>>>>>>>>>> [INFO] [ERROR] Generator > >>>>>>>>>>>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator' > >>>>>>>>>>>>>>>>>>>> threw > >>>>>>>>>>>>>>>>>>>> an exception while rebinding > >>>>>>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' > >>>>>>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException > >>>>>>>>>>>>>>>>>>>> [INFO] at > >>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40) > >>>>>>>>>>>>>>>>>>>> ... > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> NullPointerException when processing > >>>>>>>>>>>>>>>>>>>> ApplicationResources > >>>>>>>>>>>>>>>>>>>> but > >>>>>>>>>>>>>>>>>>>> occuring > >>>>>>>>>>>>>>>>>>>> from > >>>>>>>>>>>>>>>>>>>> different places (ImageResourceGenerator vs. > >>>>>>>>>>>>>>>>>>>> InlineResourceContext) > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Now I understand why Maven build fails without any > >>>>>>>>>>>>>>>>>>>> details: > >>>>>>>>>>>>>>>>>>>> - GWT deferred binding fails for ApplicationResources > >>>>>>>>>>>>>>>>>>>> (i.e. > >>>>>>>>>>>>>>>>>>>> uncaught > >>>>>>>>>>>>>>>>>>>> NPE) > >>>>>>>>>>>>>>>>>>>> - GWT compilation proceeds, fails to link with > >>>>>>>>>>>>>>>>>>>> ApplicationResources > >>>>>>>>>>>>>>>>>>>> - GWT compilation task fails with non-OK status code > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> So the issue here is GWT deferred binding generator > >>>>>>>>>>>>>>>>>>>> (InlineClientBundleGenerator) failure during GWT > >>>>>>>>>>>>>>>>>>>> compilation. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> GWT compiler writes extra Java files during deferred > >>>>>>>>>>>>>>>>>>>> binding > >>>>>>>>>>>>>>>>>>>> phase, so this could point to issue with local > >>>>>>>>>>>>>>>>>>>> environment.. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> I will try to investigate this and look at GWT source > >>>>>>>>>>>>>>>>>>>> code. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>>>> Sandro Bonazzola > >>>>>>>>>>>>>>>>>> Better technology. Faster innovation. Powered by community > >>>>>>>>>>>>>>>>>> collaboration. > >>>>>>>>>>>>>>>>>> See how it works at redhat.com > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>> Sandro Bonazzola > >>>>>>>>>>>>>>>> Better technology. Faster innovation. Powered by community > >>>>>>>>>>>>>>>> collaboration. > >>>>>>>>>>>>>>>> See how it works at redhat.com > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> -- > >>>>>>>>>>>>>> Sandro Bonazzola > >>>>>>>>>>>>>> Better technology. Faster innovation. Powered by community > >>>>>>>>>>>>>> collaboration. > >>>>>>>>>>>>>> See how it works at redhat.com > >>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> -- > >>>>>>>>>>>> Sandro Bonazzola > >>>>>>>>>>>> Better technology. Faster innovation. Powered by community > >>>>>>>>>>>> collaboration. > >>>>>>>>>>>> See how it works at redhat.com > >>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> Sandro Bonazzola > >>>>>>>>>> Better technology. Faster innovation. Powered by community > >>>>>>>>>> collaboration. > >>>>>>>>>> See how it works at redhat.com > >>>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> Sandro Bonazzola > >>>>>>> Better technology. Faster innovation. Powered by community > >>>>>>> collaboration. > >>>>>>> See how it works at redhat.com > >>>>>>> > >>>>>> _______________________________________________ > >>>>>> Arch mailing list > >>>>>> Arch@ovirt.org > >>>>>> http://lists.ovirt.org/mailman/listinfo/arch > >>>>>> > >>>> > >>>> > >>>> -- > >>>> Sandro Bonazzola > >>>> Better technology. Faster innovation. Powered by community > >>>> collaboration. > >>>> See how it works at redhat.com > >>>> _______________________________________________ > >>>> Arch mailing list > >>>> Arch@ovirt.org > >>>> http://lists.ovirt.org/mailman/listinfo/arch > >>>> > >> > >> > >> -- > >> Sandro Bonazzola > >> Better technology. Faster innovation. Powered by community collaboration. > >> See how it works at redhat.com > >> > > > -- > Sandro Bonazzola > Better technology. Faster innovation. Powered by community collaboration. > See how it works at redhat.com > _______________________________________________ Arch mailing list Arch@ovirt.org http://lists.ovirt.org/mailman/listinfo/arch