----- Original Message ----- > From: "Sandro Bonazzola" <sbona...@redhat.com> > To: "Vojtech Szocs" <vsz...@redhat.com> > Cc: "Eyal Edri" <ee...@redhat.com>, "Juan Hernandez" <jhern...@redhat.com>, > "Kiril Nesenko" <knese...@redhat.com>, > "arch" <arch@ovirt.org> > Sent: Monday, February 24, 2014 9:08:09 AM > Subject: [urgent] Re: ovirt-engine build failure > > Top posting, > > Please give this task really urgent priority, as you can see: > > http://jenkins.ovirt.org/job/ovirt-engine_3.4_create_rpms/ > and > http://jenkins.ovirt.org/job/ovirt-engine_master_create_rpms/ > > are failing consistently so we don't have nightly builds.
(CC'ing Einav) Attached TEST patch, please apply & rebuild ovirt-engine as usual. This patch should cause following output during GWT compilation: [ERROR] ######### Util.readURLAsBytes file:/home/vszocs/work/ovirt-engine/frontend/webadmin/modules/webadmin/target/generated-sources/gwt/org/ovirt/engine/ui/webadmin/images/search_button.png If IOException occurs, exception stacktrace will be printed right below that line. Note: please ignore the [ERROR] prefix, it's a technical issue, more details at https://www.mail-archive.com/engine-devel@ovirt.org/msg05735.html I just did a local upstream build with TEST patch applied, GWT compilation passed OK. > > > > > Il 20/02/2014 13:34, Vojtech Szocs ha scritto: > > > > > > ----- Original Message ----- > >> From: "Sandro Bonazzola" <sbona...@redhat.com> > >> To: "Vojtech Szocs" <vsz...@redhat.com> > >> Cc: "Eyal Edri" <ee...@redhat.com>, "Juan Hernandez" > >> <jhern...@redhat.com>, "Kiril Nesenko" <knese...@redhat.com>, > >> "arch" <arch@ovirt.org> > >> Sent: Thursday, February 20, 2014 10:02:00 AM > >> Subject: Re: ovirt-engine build failure > >> > >> Il 19/02/2014 16:51, Vojtech Szocs ha scritto: > >>> > >>> > >>> ----- Original Message ----- > >>>> From: "Sandro Bonazzola" <sbona...@redhat.com> > >>>> To: "Vojtech Szocs" <vsz...@redhat.com> > >>>> Cc: "Eyal Edri" <ee...@redhat.com>, "Juan Hernandez" > >>>> <jhern...@redhat.com>, "Kiril Nesenko" <knese...@redhat.com>, > >>>> "arch" <arch@ovirt.org> > >>>> Sent: Wednesday, February 19, 2014 2:43:44 PM > >>>> Subject: Re: ovirt-engine build failure > >>>> > >>>> Il 19/02/2014 14:41, Vojtech Szocs ha scritto: > >>>>> > >>>>> > >>>>> ----- Original Message ----- > >>>>>> From: "Sandro Bonazzola" <sbona...@redhat.com> > >>>>>> To: "Vojtech Szocs" <vsz...@redhat.com> > >>>>>> Cc: "Eyal Edri" <ee...@redhat.com>, "Juan Hernandez" > >>>>>> <jhern...@redhat.com>, "Kiril Nesenko" <knese...@redhat.com>, > >>>>>> "arch" <arch@ovirt.org> > >>>>>> Sent: Wednesday, February 19, 2014 2:23:16 PM > >>>>>> Subject: Re: ovirt-engine build failure > >>>>>> > >>>>>> Il 19/02/2014 14:19, Vojtech Szocs ha scritto: > >>>>>>> > >>>>>>> > >>>>>>> ----- Original Message ----- > >>>>>>>> From: "Sandro Bonazzola" <sbona...@redhat.com> > >>>>>>>> To: "Vojtech Szocs" <vsz...@redhat.com> > >>>>>>>> Cc: "Eyal Edri" <ee...@redhat.com>, "Juan Hernandez" > >>>>>>>> <jhern...@redhat.com>, "Kiril Nesenko" <knese...@redhat.com>, > >>>>>>>> "arch" <arch@ovirt.org> > >>>>>>>> Sent: Wednesday, February 19, 2014 1:18:28 PM > >>>>>>>> Subject: Re: ovirt-engine build failure > >>>>>>>> > >>>>>>>> Il 19/02/2014 12:45, Vojtech Szocs ha scritto: > >>>>>>>>> Hi, I checked GWT sources: > >>>>>>>>> > >>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever computed > >>>>>>>>>> for > >>>>>>>>>> removeBrickCommitRequired > >>>>>>>>>> [INFO] at > >>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475) > >>>>>>>>> > >>>>>>>>> This happened because internal image cache for bundled (composite > >>>>>>>>> PNG) > >>>>>>>>> images was corrput. > >>>>>>>>> It seems to be caused by disk I/O error while reading file > >>>>>>>>> images/glusterAsyncTasks/remove_brick_commit_required.png > >>>>>>>>> > >>>>>>>>>> [INFO] java.lang.NullPointerException > >>>>>>>>>> [INFO] at > >>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40) > >>>>>>>>> > >>>>>>>>> This happened because image data was null, again possible disk I/O > >>>>>>>>> error > >>>>>>>>> while reading image file. > >>>>>>>>> > >>>>>>>>> Can we enable "TRACE" logging during GWT compilation? > >>>>>>>>> > >>>>>>>>> -Dgwt.logLevel=TRACE > >>>>>>>> > >>>>>>>> > >>>>>>>> Building master right now with your suggestions here: > >>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/ > >>>>>>> > >>>>>>> Looking at > >>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=fedora20-host/console > >>>>>>> > >>>>>>> So far I see no errors.. There should be total 5 browsers x 8 locales > >>>>>>> = > >>>>>>> 40 > >>>>>>> GWT permutations. > >>>>>> > >>>>>> So no changes with and without > >>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera. > >>>>>> We've still 40 permutations. Maybe you defined > >>>>>> userAgent=ie8,ie9,gecko1_8,safari,opera somewhere in the POM files? > >>>>> > >>>>> You are right. I checked frontend/webadmin/modules/pom.xml -> the > >>>>> "gwt.userAgent" > >>>>> property is copied into "locale" GWT property in > >>>>> {WebAdmin,UserPortal}.gwt.xml > >>>> > >>>> Ok, so I'll remove -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera from > >>>> build > >>>> command. > >>> > >>> Agreed. > >>> > >>>> > >>>> > >>>>> > >>>>>> > >>>>>> This: > >>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=centos6-host/console > >>>>>> failed, you can look at it, it has -Dgwt.logLevel=TRACE -X. > >>>>> > >>>>> Hm, it contains: > >>>>> > >>>>> [DEBUG] (f) logLevel = INFO > >>>>> > >>>>> Somehow the TRACE log level for GWT wasn't applied.. not sure why. Let > >>>>> me > >>>>> check > >>>>> this on local build. > >>>> > >>>> Ok, let me know if I've to change command line. > >>> > >>> I wrote upstream patch: http://gerrit.ovirt.org/#/c/24789/ > >>> > >>> When it's merged, setting "gwt.logLevel" will work (as you did before) > >>> and we'll be able to see more details in the build log. > >> > >> Here is the new build with your patch merged: > >> > >> http://jenkins.ovirt.org/job/manual-build-tarball/228/label=centos6-host/consoleFull > > > > Thanks. I did some more investigation on this. > > > > Again it's NullPointerException when rebinding ApplicationResources, but > > this time for different image file: ApplicationResources.wrenchImage > > ("images/wrench.png"). > > > > Looking at stacktrace & GWT sources, the problematic code seems to be > > (AbstractResourceContext:74): > > > > byte[] bytes = Util.readURLAsBytes(resource); > > > > where "resource" is (ImageResourceGenerator:311): > > > > localized.getUrl(); // LocalizedImage localized > > > > Note: in oVirt UI we don't use localized images (i.e. "wrench_fr_FR.png") > > so "localized" in this case points to "wrench.png". > > > > So the problem seems to be reading this file: > > > > > > frontend/webadmin/modules/webadmin/src/main/resources/org/ovirt/engine/ui/webadmin/images/wrench.png > > > > Since we're having many GWT permutations, this file ^^ (along with similar > > ones) is attempted to be read multiple times *from the same (single) GWT > > compiler Java process, before actual permutation worker Java sub-processes > > are spawned*. (More precisely, above NullPointerException happens at > > deferred binding phase which precedes actual GWT compilation per each > > permutation.) > > > > I think the root cause is Util.readURLAsBytes method used internally by GWT > > compiler: > > > > public static byte[] readURLAsBytes(URL url) { > > try { > > URLConnection conn = url.openConnection(); > > conn.setUseCaches(false); > > return readURLConnectionAsBytes(conn); > > } catch (IOException e) { > > return null; // This causes NullPointerException, but this exception > > isn't logged (wtf, GWT!) > > } > > } > > > > To summarize, I'll need to write another patch which replaces > > Util.readURLAsBytes implementation via AOP and prints IOException details. > > > >> > >> > >> > >> > >>> > >>>> > >>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>> > >>>>>>>> > >>>>>>>> # CI_MAVEN_SETTINGS=$WORKSPACE/artifactory-ovirt-org-settings.xml > >>>>>>>> # export MAVEN_OPTS="${MAVEN_OPTS} -Xmx2048M -XX:MaxPermSize=1024m > >>>>>>>> -Dgwt.compiler.localWorkers=1" > >>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -ts "${TARBALL}" > >>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -D "ovirt_build_extra_flags > >>>>>>>> -gs > >>>>>>>> ${CI_MAVEN_SETTINGS} -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera > >>>>>>>> -Dgwt.logLevel=TRACE -X" --rebuild rpmbuild/SRPMS/*.src.rpm > >>>>>>>> > >>>>>>>> Let me know if you need anything else. > >>>>>>>> > >>>>>>>> > >>>>>>>>> > >>>>>>>>> Vojtech > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> ----- Original Message ----- > >>>>>>>>>> From: "Vojtech Szocs" <vsz...@redhat.com> > >>>>>>>>>> To: "Sandro Bonazzola" <sbona...@redhat.com> > >>>>>>>>>> Cc: "Eyal Edri" <ee...@redhat.com>, "Juan Hernandez" > >>>>>>>>>> <jhern...@redhat.com>, "Kiril Nesenko" <knese...@redhat.com>, > >>>>>>>>>> "arch" <arch@ovirt.org> > >>>>>>>>>> Sent: Tuesday, February 18, 2014 6:54:20 PM > >>>>>>>>>> Subject: Re: ovirt-engine build failure > >>>>>>>>>> > >>>>>>>>>> In your log I see following error: > >>>>>>>>>> > >>>>>>>>>> ... > >>>>>>>>>> [INFO] Computing all possible rebind results for > >>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' > >>>>>>>>>> [INFO] Rebinding > >>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources > >>>>>>>>>> [INFO] Invoking generator > >>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator > >>>>>>>>>> [INFO] [ERROR] Generator > >>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator' > >>>>>>>>>> threw > >>>>>>>>>> an exception while rebinding > >>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' > >>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever computed > >>>>>>>>>> for > >>>>>>>>>> removeBrickCommitRequired > >>>>>>>>>> [INFO] at > >>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475) > >>>>>>>>>> ... > >>>>>>>>>> > >>>>>>>>>> I checked again logs at > >>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/212/label=fedora19-host/consoleFull > >>>>>>>>>> and indeed there is a similar error: > >>>>>>>>>> > >>>>>>>>>> ... > >>>>>>>>>> [INFO] Computing all possible rebind results for > >>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' > >>>>>>>>>> [INFO] Rebinding > >>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources > >>>>>>>>>> [INFO] Invoking generator > >>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator > >>>>>>>>>> [INFO] [ERROR] Generator > >>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator' > >>>>>>>>>> threw > >>>>>>>>>> an exception while rebinding > >>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' > >>>>>>>>>> [INFO] java.lang.NullPointerException > >>>>>>>>>> [INFO] at > >>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40) > >>>>>>>>>> ... > >>>>>>>>>> > >>>>>>>>>> NullPointerException when processing ApplicationResources but > >>>>>>>>>> occuring > >>>>>>>>>> from > >>>>>>>>>> different places (ImageResourceGenerator vs. > >>>>>>>>>> InlineResourceContext) > >>>>>>>>>> > >>>>>>>>>> Now I understand why Maven build fails without any details: > >>>>>>>>>> - GWT deferred binding fails for ApplicationResources (i.e. > >>>>>>>>>> uncaught > >>>>>>>>>> NPE) > >>>>>>>>>> - GWT compilation proceeds, fails to link with > >>>>>>>>>> ApplicationResources > >>>>>>>>>> - GWT compilation task fails with non-OK status code > >>>>>>>>>> > >>>>>>>>>> So the issue here is GWT deferred binding generator > >>>>>>>>>> (InlineClientBundleGenerator) failure during GWT compilation. > >>>>>>>>>> > >>>>>>>>>> GWT compiler writes extra Java files during deferred binding > >>>>>>>>>> phase, so this could point to issue with local environment.. > >>>>>>>>>> > >>>>>>>>>> I will try to investigate this and look at GWT source code. > >>>>>>>> > >>>>>>>> > >>>>>>>> -- > >>>>>>>> Sandro Bonazzola > >>>>>>>> Better technology. Faster innovation. Powered by community > >>>>>>>> collaboration. > >>>>>>>> See how it works at redhat.com > >>>>>>>> > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> Sandro Bonazzola > >>>>>> Better technology. Faster innovation. Powered by community > >>>>>> collaboration. > >>>>>> See how it works at redhat.com > >>>>>> > >>>> > >>>> > >>>> -- > >>>> Sandro Bonazzola > >>>> Better technology. Faster innovation. Powered by community > >>>> collaboration. > >>>> See how it works at redhat.com > >>>> > >> > >> > >> -- > >> Sandro Bonazzola > >> Better technology. Faster innovation. Powered by community collaboration. > >> See how it works at redhat.com > >> > > > -- > Sandro Bonazzola > Better technology. Faster innovation. Powered by community collaboration. > See how it works at redhat.com >
From f506406e58f2dda989fcf26c338725ea3c7733f1 Mon Sep 17 00:00:00 2001 From: Vojtech Szocs <vsz...@redhat.com> Date: Mon, 24 Feb 2014 18:11:44 +0100 Subject: [PATCH] TEST: Completely override Util.readURLAsBytes implementation Change-Id: I85643a58a71b20750949e69ae1636b1e67eaf08b Signed-off-by: Vojtech Szocs <vsz...@redhat.com> --- .../org/ovirt/engine/ui/gwtextension/DontPrune.java | 19 +++++++++++++++++++ .../gwt-extension/src/main/resources/META-INF/aop.xml | 1 + 2 files changed, 20 insertions(+) diff --git a/frontend/webadmin/modules/gwt-extension/src/main/java/org/ovirt/engine/ui/gwtextension/DontPrune.java b/frontend/webadmin/modules/gwt-extension/src/main/java/org/ovirt/engine/ui/gwtextension/DontPrune.java index 0ff767f..3d1d2e0 100644 --- a/frontend/webadmin/modules/gwt-extension/src/main/java/org/ovirt/engine/ui/gwtextension/DontPrune.java +++ b/frontend/webadmin/modules/gwt-extension/src/main/java/org/ovirt/engine/ui/gwtextension/DontPrune.java @@ -1,5 +1,8 @@ package org.ovirt.engine.ui.gwtextension; +import java.io.IOException; +import java.net.URL; +import java.net.URLConnection; import java.util.regex.Matcher; import java.util.regex.Pattern; @@ -10,7 +13,9 @@ import com.google.gwt.dev.jjs.impl.ControlFlowAnalyzer; //import com.google.gwt.dev.util.log.PrintWriterTreeLogger; //import com.google.gwt.core.ext.TreeLogger; +import org.aspectj.lang.ProceedingJoinPoint; import org.aspectj.lang.annotation.After; +import org.aspectj.lang.annotation.Around; import org.aspectj.lang.annotation.Aspect; import org.aspectj.lang.annotation.Pointcut; @@ -72,4 +77,18 @@ public class DontPrune { } } + // TODO TEST: Completely override Util.readURLAsBytes implementation + @Around("execution(public static byte[] com.google.gwt.dev.util.Util.readURLAsBytes(java.net.URL)) && args(url)") + public Object aroundReadURLAsBytes(ProceedingJoinPoint pjp, URL url) { + System.err.println("######### Util.readURLAsBytes " + url); // New code + try { + URLConnection conn = url.openConnection(); + conn.setUseCaches(false); + return com.google.gwt.dev.util.Util.readURLConnectionAsBytes(conn); + } catch (IOException e) { + e.printStackTrace(System.err); // New code + return null; + } + } + } diff --git a/frontend/webadmin/modules/gwt-extension/src/main/resources/META-INF/aop.xml b/frontend/webadmin/modules/gwt-extension/src/main/resources/META-INF/aop.xml index a94340f..8cb1b75 100644 --- a/frontend/webadmin/modules/gwt-extension/src/main/resources/META-INF/aop.xml +++ b/frontend/webadmin/modules/gwt-extension/src/main/resources/META-INF/aop.xml @@ -7,6 +7,7 @@ <weaver options="-verbose"> <include within="org.ovirt.engine.ui.gwtextension.DontPrune"/> <include within="com.google.gwt.dev.jjs.impl.ControlFlowAnalyzer"/> + <include within="com.google.gwt.dev.util.Util"/> </weaver> <aspects> -- 1.8.3.1
_______________________________________________ Arch mailing list Arch@ovirt.org http://lists.ovirt.org/mailman/listinfo/arch