Il 24/02/2014 09:08, Sandro Bonazzola ha scritto: > Top posting, > > Please give this task really urgent priority, as you can see: > > http://jenkins.ovirt.org/job/ovirt-engine_3.4_create_rpms/ > and > http://jenkins.ovirt.org/job/ovirt-engine_master_create_rpms/ > > are failing consistently so we don't have nightly builds. > >
CCing infra. > > > Il 20/02/2014 13:34, Vojtech Szocs ha scritto: >> >> >> ----- Original Message ----- >>> From: "Sandro Bonazzola" <sbona...@redhat.com> >>> To: "Vojtech Szocs" <vsz...@redhat.com> >>> Cc: "Eyal Edri" <ee...@redhat.com>, "Juan Hernandez" <jhern...@redhat.com>, >>> "Kiril Nesenko" <knese...@redhat.com>, >>> "arch" <a...@ovirt.org> >>> Sent: Thursday, February 20, 2014 10:02:00 AM >>> Subject: Re: ovirt-engine build failure >>> >>> Il 19/02/2014 16:51, Vojtech Szocs ha scritto: >>>> >>>> >>>> ----- Original Message ----- >>>>> From: "Sandro Bonazzola" <sbona...@redhat.com> >>>>> To: "Vojtech Szocs" <vsz...@redhat.com> >>>>> Cc: "Eyal Edri" <ee...@redhat.com>, "Juan Hernandez" >>>>> <jhern...@redhat.com>, "Kiril Nesenko" <knese...@redhat.com>, >>>>> "arch" <a...@ovirt.org> >>>>> Sent: Wednesday, February 19, 2014 2:43:44 PM >>>>> Subject: Re: ovirt-engine build failure >>>>> >>>>> Il 19/02/2014 14:41, Vojtech Szocs ha scritto: >>>>>> >>>>>> >>>>>> ----- Original Message ----- >>>>>>> From: "Sandro Bonazzola" <sbona...@redhat.com> >>>>>>> To: "Vojtech Szocs" <vsz...@redhat.com> >>>>>>> Cc: "Eyal Edri" <ee...@redhat.com>, "Juan Hernandez" >>>>>>> <jhern...@redhat.com>, "Kiril Nesenko" <knese...@redhat.com>, >>>>>>> "arch" <a...@ovirt.org> >>>>>>> Sent: Wednesday, February 19, 2014 2:23:16 PM >>>>>>> Subject: Re: ovirt-engine build failure >>>>>>> >>>>>>> Il 19/02/2014 14:19, Vojtech Szocs ha scritto: >>>>>>>> >>>>>>>> >>>>>>>> ----- Original Message ----- >>>>>>>>> From: "Sandro Bonazzola" <sbona...@redhat.com> >>>>>>>>> To: "Vojtech Szocs" <vsz...@redhat.com> >>>>>>>>> Cc: "Eyal Edri" <ee...@redhat.com>, "Juan Hernandez" >>>>>>>>> <jhern...@redhat.com>, "Kiril Nesenko" <knese...@redhat.com>, >>>>>>>>> "arch" <a...@ovirt.org> >>>>>>>>> Sent: Wednesday, February 19, 2014 1:18:28 PM >>>>>>>>> Subject: Re: ovirt-engine build failure >>>>>>>>> >>>>>>>>> Il 19/02/2014 12:45, Vojtech Szocs ha scritto: >>>>>>>>>> Hi, I checked GWT sources: >>>>>>>>>> >>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever computed >>>>>>>>>>> for >>>>>>>>>>> removeBrickCommitRequired >>>>>>>>>>> [INFO] at >>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475) >>>>>>>>>> >>>>>>>>>> This happened because internal image cache for bundled (composite >>>>>>>>>> PNG) >>>>>>>>>> images was corrput. >>>>>>>>>> It seems to be caused by disk I/O error while reading file >>>>>>>>>> images/glusterAsyncTasks/remove_brick_commit_required.png >>>>>>>>>> >>>>>>>>>>> [INFO] java.lang.NullPointerException >>>>>>>>>>> [INFO] at >>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40) >>>>>>>>>> >>>>>>>>>> This happened because image data was null, again possible disk I/O >>>>>>>>>> error >>>>>>>>>> while reading image file. >>>>>>>>>> >>>>>>>>>> Can we enable "TRACE" logging during GWT compilation? >>>>>>>>>> >>>>>>>>>> -Dgwt.logLevel=TRACE >>>>>>>>> >>>>>>>>> >>>>>>>>> Building master right now with your suggestions here: >>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/ >>>>>>>> >>>>>>>> Looking at >>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=fedora20-host/console >>>>>>>> >>>>>>>> So far I see no errors.. There should be total 5 browsers x 8 locales = >>>>>>>> 40 >>>>>>>> GWT permutations. >>>>>>> >>>>>>> So no changes with and without >>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera. >>>>>>> We've still 40 permutations. Maybe you defined >>>>>>> userAgent=ie8,ie9,gecko1_8,safari,opera somewhere in the POM files? >>>>>> >>>>>> You are right. I checked frontend/webadmin/modules/pom.xml -> the >>>>>> "gwt.userAgent" >>>>>> property is copied into "locale" GWT property in >>>>>> {WebAdmin,UserPortal}.gwt.xml >>>>> >>>>> Ok, so I'll remove -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera from >>>>> build >>>>> command. >>>> >>>> Agreed. >>>> >>>>> >>>>> >>>>>> >>>>>>> >>>>>>> This: >>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=centos6-host/console >>>>>>> failed, you can look at it, it has -Dgwt.logLevel=TRACE -X. >>>>>> >>>>>> Hm, it contains: >>>>>> >>>>>> [DEBUG] (f) logLevel = INFO >>>>>> >>>>>> Somehow the TRACE log level for GWT wasn't applied.. not sure why. Let me >>>>>> check >>>>>> this on local build. >>>>> >>>>> Ok, let me know if I've to change command line. >>>> >>>> I wrote upstream patch: http://gerrit.ovirt.org/#/c/24789/ >>>> >>>> When it's merged, setting "gwt.logLevel" will work (as you did before) >>>> and we'll be able to see more details in the build log. >>> >>> Here is the new build with your patch merged: >>> >>> http://jenkins.ovirt.org/job/manual-build-tarball/228/label=centos6-host/consoleFull >> >> Thanks. I did some more investigation on this. >> >> Again it's NullPointerException when rebinding ApplicationResources, but >> this time for different image file: ApplicationResources.wrenchImage >> ("images/wrench.png"). >> >> Looking at stacktrace & GWT sources, the problematic code seems to be >> (AbstractResourceContext:74): >> >> byte[] bytes = Util.readURLAsBytes(resource); >> >> where "resource" is (ImageResourceGenerator:311): >> >> localized.getUrl(); // LocalizedImage localized >> >> Note: in oVirt UI we don't use localized images (i.e. "wrench_fr_FR.png") so >> "localized" in this case points to "wrench.png". >> >> So the problem seems to be reading this file: >> >> >> frontend/webadmin/modules/webadmin/src/main/resources/org/ovirt/engine/ui/webadmin/images/wrench.png >> >> Since we're having many GWT permutations, this file ^^ (along with similar >> ones) is attempted to be read multiple times *from the same (single) GWT >> compiler Java process, before actual permutation worker Java sub-processes >> are spawned*. (More precisely, above NullPointerException happens at >> deferred binding phase which precedes actual GWT compilation per each >> permutation.) >> >> I think the root cause is Util.readURLAsBytes method used internally by GWT >> compiler: >> >> public static byte[] readURLAsBytes(URL url) { >> try { >> URLConnection conn = url.openConnection(); >> conn.setUseCaches(false); >> return readURLConnectionAsBytes(conn); >> } catch (IOException e) { >> return null; // This causes NullPointerException, but this exception >> isn't logged (wtf, GWT!) >> } >> } >> >> To summarize, I'll need to write another patch which replaces >> Util.readURLAsBytes implementation via AOP and prints IOException details. >> >>> >>> >>> >>> >>>> >>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> # CI_MAVEN_SETTINGS=$WORKSPACE/artifactory-ovirt-org-settings.xml >>>>>>>>> # export MAVEN_OPTS="${MAVEN_OPTS} -Xmx2048M -XX:MaxPermSize=1024m >>>>>>>>> -Dgwt.compiler.localWorkers=1" >>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -ts "${TARBALL}" >>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -D "ovirt_build_extra_flags >>>>>>>>> -gs >>>>>>>>> ${CI_MAVEN_SETTINGS} -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera >>>>>>>>> -Dgwt.logLevel=TRACE -X" --rebuild rpmbuild/SRPMS/*.src.rpm >>>>>>>>> >>>>>>>>> Let me know if you need anything else. >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Vojtech >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ----- Original Message ----- >>>>>>>>>>> From: "Vojtech Szocs" <vsz...@redhat.com> >>>>>>>>>>> To: "Sandro Bonazzola" <sbona...@redhat.com> >>>>>>>>>>> Cc: "Eyal Edri" <ee...@redhat.com>, "Juan Hernandez" >>>>>>>>>>> <jhern...@redhat.com>, "Kiril Nesenko" <knese...@redhat.com>, >>>>>>>>>>> "arch" <a...@ovirt.org> >>>>>>>>>>> Sent: Tuesday, February 18, 2014 6:54:20 PM >>>>>>>>>>> Subject: Re: ovirt-engine build failure >>>>>>>>>>> >>>>>>>>>>> In your log I see following error: >>>>>>>>>>> >>>>>>>>>>> ... >>>>>>>>>>> [INFO] Computing all possible rebind results for >>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' >>>>>>>>>>> [INFO] Rebinding >>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources >>>>>>>>>>> [INFO] Invoking generator >>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator >>>>>>>>>>> [INFO] [ERROR] Generator >>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator' >>>>>>>>>>> threw >>>>>>>>>>> an exception while rebinding >>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' >>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever computed >>>>>>>>>>> for >>>>>>>>>>> removeBrickCommitRequired >>>>>>>>>>> [INFO] at >>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475) >>>>>>>>>>> ... >>>>>>>>>>> >>>>>>>>>>> I checked again logs at >>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/212/label=fedora19-host/consoleFull >>>>>>>>>>> and indeed there is a similar error: >>>>>>>>>>> >>>>>>>>>>> ... >>>>>>>>>>> [INFO] Computing all possible rebind results for >>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' >>>>>>>>>>> [INFO] Rebinding >>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources >>>>>>>>>>> [INFO] Invoking generator >>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator >>>>>>>>>>> [INFO] [ERROR] Generator >>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator' >>>>>>>>>>> threw >>>>>>>>>>> an exception while rebinding >>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' >>>>>>>>>>> [INFO] java.lang.NullPointerException >>>>>>>>>>> [INFO] at >>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40) >>>>>>>>>>> ... >>>>>>>>>>> >>>>>>>>>>> NullPointerException when processing ApplicationResources but >>>>>>>>>>> occuring >>>>>>>>>>> from >>>>>>>>>>> different places (ImageResourceGenerator vs. InlineResourceContext) >>>>>>>>>>> >>>>>>>>>>> Now I understand why Maven build fails without any details: >>>>>>>>>>> - GWT deferred binding fails for ApplicationResources (i.e. uncaught >>>>>>>>>>> NPE) >>>>>>>>>>> - GWT compilation proceeds, fails to link with ApplicationResources >>>>>>>>>>> - GWT compilation task fails with non-OK status code >>>>>>>>>>> >>>>>>>>>>> So the issue here is GWT deferred binding generator >>>>>>>>>>> (InlineClientBundleGenerator) failure during GWT compilation. >>>>>>>>>>> >>>>>>>>>>> GWT compiler writes extra Java files during deferred binding >>>>>>>>>>> phase, so this could point to issue with local environment.. >>>>>>>>>>> >>>>>>>>>>> I will try to investigate this and look at GWT source code. >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Sandro Bonazzola >>>>>>>>> Better technology. Faster innovation. Powered by community >>>>>>>>> collaboration. >>>>>>>>> See how it works at redhat.com >>>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Sandro Bonazzola >>>>>>> Better technology. Faster innovation. Powered by community >>>>>>> collaboration. >>>>>>> See how it works at redhat.com >>>>>>> >>>>> >>>>> >>>>> -- >>>>> Sandro Bonazzola >>>>> Better technology. Faster innovation. Powered by community collaboration. >>>>> See how it works at redhat.com >>>>> >>> >>> >>> -- >>> Sandro Bonazzola >>> Better technology. Faster innovation. Powered by community collaboration. >>> See how it works at redhat.com >>> > > -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra