Il 25/02/2014 16:52, Vojtech Szocs ha scritto: > > > ----- Original Message ----- >> From: "Vojtech Szocs" <vsz...@redhat.com> >> To: "Sandro Bonazzola" <sbona...@redhat.com> >> Cc: "Juan Hernandez" <jhern...@redhat.com>, "Kiril Nesenko" >> <knese...@redhat.com>, "arch" <arch@ovirt.org> >> Sent: Tuesday, February 25, 2014 1:05:35 PM >> Subject: Re: [urgent] Re: ovirt-engine build failure >> >> >> >> ----- Original Message ----- >>> From: "Sandro Bonazzola" <sbona...@redhat.com> >>> To: "Vojtech Szocs" <vsz...@redhat.com> >>> Cc: "Juan Hernandez" <jhern...@redhat.com>, "arch" <arch@ovirt.org>, "Kiril >>> Nesenko" <knese...@redhat.com> >>> Sent: Tuesday, February 25, 2014 9:02:11 AM >>> Subject: Re: [urgent] Re: ovirt-engine build failure >>> >>> Il 25/02/2014 08:47, Sandro Bonazzola ha scritto: >>>> Il 24/02/2014 18:28, Vojtech Szocs ha scritto: >>>>> >>>>> >>>>> ----- Original Message ----- >>>>>> From: "Sandro Bonazzola" <sbona...@redhat.com> >>>>>> To: "Vojtech Szocs" <vsz...@redhat.com> >>>>>> Cc: "Eyal Edri" <ee...@redhat.com>, "Juan Hernandez" >>>>>> <jhern...@redhat.com>, "Kiril Nesenko" <knese...@redhat.com>, >>>>>> "arch" <arch@ovirt.org> >>>>>> Sent: Monday, February 24, 2014 9:08:09 AM >>>>>> Subject: [urgent] Re: ovirt-engine build failure >>>>>> >>>>>> Top posting, >>>>>> >>>>>> Please give this task really urgent priority, as you can see: >>>>>> >>>>>> http://jenkins.ovirt.org/job/ovirt-engine_3.4_create_rpms/ >>>>>> and >>>>>> http://jenkins.ovirt.org/job/ovirt-engine_master_create_rpms/ >>>>>> >>>>>> are failing consistently so we don't have nightly builds. >>>>> >>>>> (CC'ing Einav) >>>>> >>>>> Attached TEST patch, please apply & rebuild ovirt-engine as usual. >>>>> >>>>> This patch should cause following output during GWT compilation: >>>>> >>>>> [ERROR] ######### Util.readURLAsBytes >>>>> file:/home/vszocs/work/ovirt-engine/frontend/webadmin/modules/webadmin/target/generated-sources/gwt/org/ovirt/engine/ui/webadmin/images/search_button.png >>>>> >>>>> If IOException occurs, exception stacktrace will be printed right below >>>>> that line. >>>>> >>>>> Note: please ignore the [ERROR] prefix, it's a technical issue, more >>>>> details at >>>>> https://www.mail-archive.com/engine-devel@ovirt.org/msg05735.html >>>>> >>>>> I just did a local upstream build with TEST patch applied, GWT >>>>> compilation >>>>> passed OK. >>>> >>>> build job running with your patch here: >>>> http://jenkins.ovirt.org/job/manual-build-tarball/235/ >>>> waiting for a free executor right now. >>>> Let's see how it goes. >>> >>> Failed on CentOS: >>> http://jenkins.ovirt.org/job/manual-build-tarball/235/label=centos6-host/consoleFull >> >> Hm, in that ^^ build, following exceptions occured: >> >> 1. >> [INFO] java.lang.NullPointerException >> [INFO] at com.google.gwt.dev.util.Util.computeStrongName(Util.java:172) >> >> 2. >> [INFO] java.lang.NullPointerException: No ImageRect ever computed for >> downDisabledImage >> [INFO] at >> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475 >> >> 1. is caused by Util#readURLAsBytes but I don't see any IOException in the >> logs >> 2. is just follow-up consequence of 1. (i.e. image metadata missing in GWT >> image cache) >> >> Now I noticed I missed one possible IOException inside >> Util#readURLConnectionAsBytes ... >> >> Long story short, I'll send updated TEST patch soon, apologies for the >> inconvenience. > > TEST-v2 patch attached. Please apply & rebuild ovirt-engine as usual.
Done, here is the console output: http://jenkins.ovirt.org/job/manual-build-tarball/label=centos6-host/239/consoleFull > >> >>> >>> >>> >>> >>> >>> >>>> >>>> >>>> >>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Il 20/02/2014 13:34, Vojtech Szocs ha scritto: >>>>>>> >>>>>>> >>>>>>> ----- Original Message ----- >>>>>>>> From: "Sandro Bonazzola" <sbona...@redhat.com> >>>>>>>> To: "Vojtech Szocs" <vsz...@redhat.com> >>>>>>>> Cc: "Eyal Edri" <ee...@redhat.com>, "Juan Hernandez" >>>>>>>> <jhern...@redhat.com>, "Kiril Nesenko" <knese...@redhat.com>, >>>>>>>> "arch" <arch@ovirt.org> >>>>>>>> Sent: Thursday, February 20, 2014 10:02:00 AM >>>>>>>> Subject: Re: ovirt-engine build failure >>>>>>>> >>>>>>>> Il 19/02/2014 16:51, Vojtech Szocs ha scritto: >>>>>>>>> >>>>>>>>> >>>>>>>>> ----- Original Message ----- >>>>>>>>>> From: "Sandro Bonazzola" <sbona...@redhat.com> >>>>>>>>>> To: "Vojtech Szocs" <vsz...@redhat.com> >>>>>>>>>> Cc: "Eyal Edri" <ee...@redhat.com>, "Juan Hernandez" >>>>>>>>>> <jhern...@redhat.com>, "Kiril Nesenko" <knese...@redhat.com>, >>>>>>>>>> "arch" <arch@ovirt.org> >>>>>>>>>> Sent: Wednesday, February 19, 2014 2:43:44 PM >>>>>>>>>> Subject: Re: ovirt-engine build failure >>>>>>>>>> >>>>>>>>>> Il 19/02/2014 14:41, Vojtech Szocs ha scritto: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>> From: "Sandro Bonazzola" <sbona...@redhat.com> >>>>>>>>>>>> To: "Vojtech Szocs" <vsz...@redhat.com> >>>>>>>>>>>> Cc: "Eyal Edri" <ee...@redhat.com>, "Juan Hernandez" >>>>>>>>>>>> <jhern...@redhat.com>, "Kiril Nesenko" <knese...@redhat.com>, >>>>>>>>>>>> "arch" <arch@ovirt.org> >>>>>>>>>>>> Sent: Wednesday, February 19, 2014 2:23:16 PM >>>>>>>>>>>> Subject: Re: ovirt-engine build failure >>>>>>>>>>>> >>>>>>>>>>>> Il 19/02/2014 14:19, Vojtech Szocs ha scritto: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>> From: "Sandro Bonazzola" <sbona...@redhat.com> >>>>>>>>>>>>>> To: "Vojtech Szocs" <vsz...@redhat.com> >>>>>>>>>>>>>> Cc: "Eyal Edri" <ee...@redhat.com>, "Juan Hernandez" >>>>>>>>>>>>>> <jhern...@redhat.com>, "Kiril Nesenko" <knese...@redhat.com>, >>>>>>>>>>>>>> "arch" <arch@ovirt.org> >>>>>>>>>>>>>> Sent: Wednesday, February 19, 2014 1:18:28 PM >>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure >>>>>>>>>>>>>> >>>>>>>>>>>>>> Il 19/02/2014 12:45, Vojtech Szocs ha scritto: >>>>>>>>>>>>>>> Hi, I checked GWT sources: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever >>>>>>>>>>>>>>>> computed >>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>> removeBrickCommitRequired >>>>>>>>>>>>>>>> [INFO] at >>>>>>>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This happened because internal image cache for bundled >>>>>>>>>>>>>>> (composite >>>>>>>>>>>>>>> PNG) >>>>>>>>>>>>>>> images was corrput. >>>>>>>>>>>>>>> It seems to be caused by disk I/O error while reading file >>>>>>>>>>>>>>> images/glusterAsyncTasks/remove_brick_commit_required.png >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException >>>>>>>>>>>>>>>> [INFO] at >>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This happened because image data was null, again possible disk >>>>>>>>>>>>>>> I/O >>>>>>>>>>>>>>> error >>>>>>>>>>>>>>> while reading image file. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Can we enable "TRACE" logging during GWT compilation? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -Dgwt.logLevel=TRACE >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Building master right now with your suggestions here: >>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/ >>>>>>>>>>>>> >>>>>>>>>>>>> Looking at >>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=fedora20-host/console >>>>>>>>>>>>> >>>>>>>>>>>>> So far I see no errors.. There should be total 5 browsers x 8 >>>>>>>>>>>>> locales >>>>>>>>>>>>> = >>>>>>>>>>>>> 40 >>>>>>>>>>>>> GWT permutations. >>>>>>>>>>>> >>>>>>>>>>>> So no changes with and without >>>>>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera. >>>>>>>>>>>> We've still 40 permutations. Maybe you defined >>>>>>>>>>>> userAgent=ie8,ie9,gecko1_8,safari,opera somewhere in the POM >>>>>>>>>>>> files? >>>>>>>>>>> >>>>>>>>>>> You are right. I checked frontend/webadmin/modules/pom.xml -> the >>>>>>>>>>> "gwt.userAgent" >>>>>>>>>>> property is copied into "locale" GWT property in >>>>>>>>>>> {WebAdmin,UserPortal}.gwt.xml >>>>>>>>>> >>>>>>>>>> Ok, so I'll remove -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera >>>>>>>>>> from >>>>>>>>>> build >>>>>>>>>> command. >>>>>>>>> >>>>>>>>> Agreed. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> This: >>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=centos6-host/console >>>>>>>>>>>> failed, you can look at it, it has -Dgwt.logLevel=TRACE -X. >>>>>>>>>>> >>>>>>>>>>> Hm, it contains: >>>>>>>>>>> >>>>>>>>>>> [DEBUG] (f) logLevel = INFO >>>>>>>>>>> >>>>>>>>>>> Somehow the TRACE log level for GWT wasn't applied.. not sure why. >>>>>>>>>>> Let >>>>>>>>>>> me >>>>>>>>>>> check >>>>>>>>>>> this on local build. >>>>>>>>>> >>>>>>>>>> Ok, let me know if I've to change command line. >>>>>>>>> >>>>>>>>> I wrote upstream patch: http://gerrit.ovirt.org/#/c/24789/ >>>>>>>>> >>>>>>>>> When it's merged, setting "gwt.logLevel" will work (as you did >>>>>>>>> before) >>>>>>>>> and we'll be able to see more details in the build log. >>>>>>>> >>>>>>>> Here is the new build with your patch merged: >>>>>>>> >>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/228/label=centos6-host/consoleFull >>>>>>> >>>>>>> Thanks. I did some more investigation on this. >>>>>>> >>>>>>> Again it's NullPointerException when rebinding ApplicationResources, >>>>>>> but >>>>>>> this time for different image file: ApplicationResources.wrenchImage >>>>>>> ("images/wrench.png"). >>>>>>> >>>>>>> Looking at stacktrace & GWT sources, the problematic code seems to be >>>>>>> (AbstractResourceContext:74): >>>>>>> >>>>>>> byte[] bytes = Util.readURLAsBytes(resource); >>>>>>> >>>>>>> where "resource" is (ImageResourceGenerator:311): >>>>>>> >>>>>>> localized.getUrl(); // LocalizedImage localized >>>>>>> >>>>>>> Note: in oVirt UI we don't use localized images (i.e. >>>>>>> "wrench_fr_FR.png") >>>>>>> so "localized" in this case points to "wrench.png". >>>>>>> >>>>>>> So the problem seems to be reading this file: >>>>>>> >>>>>>> >>>>>>> frontend/webadmin/modules/webadmin/src/main/resources/org/ovirt/engine/ui/webadmin/images/wrench.png >>>>>>> >>>>>>> Since we're having many GWT permutations, this file ^^ (along with >>>>>>> similar >>>>>>> ones) is attempted to be read multiple times *from the same (single) >>>>>>> GWT >>>>>>> compiler Java process, before actual permutation worker Java >>>>>>> sub-processes >>>>>>> are spawned*. (More precisely, above NullPointerException happens at >>>>>>> deferred binding phase which precedes actual GWT compilation per each >>>>>>> permutation.) >>>>>>> >>>>>>> I think the root cause is Util.readURLAsBytes method used internally >>>>>>> by >>>>>>> GWT >>>>>>> compiler: >>>>>>> >>>>>>> public static byte[] readURLAsBytes(URL url) { >>>>>>> try { >>>>>>> URLConnection conn = url.openConnection(); >>>>>>> conn.setUseCaches(false); >>>>>>> return readURLConnectionAsBytes(conn); >>>>>>> } catch (IOException e) { >>>>>>> return null; // This causes NullPointerException, but this >>>>>>> exception >>>>>>> isn't logged (wtf, GWT!) >>>>>>> } >>>>>>> } >>>>>>> >>>>>>> To summarize, I'll need to write another patch which replaces >>>>>>> Util.readURLAsBytes implementation via AOP and prints IOException >>>>>>> details. >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> # >>>>>>>>>>>>>> CI_MAVEN_SETTINGS=$WORKSPACE/artifactory-ovirt-org-settings.xml >>>>>>>>>>>>>> # export MAVEN_OPTS="${MAVEN_OPTS} -Xmx2048M >>>>>>>>>>>>>> -XX:MaxPermSize=1024m >>>>>>>>>>>>>> -Dgwt.compiler.localWorkers=1" >>>>>>>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -ts "${TARBALL}" >>>>>>>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -D >>>>>>>>>>>>>> "ovirt_build_extra_flags >>>>>>>>>>>>>> -gs >>>>>>>>>>>>>> ${CI_MAVEN_SETTINGS} >>>>>>>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera >>>>>>>>>>>>>> -Dgwt.logLevel=TRACE -X" --rebuild rpmbuild/SRPMS/*.src.rpm >>>>>>>>>>>>>> >>>>>>>>>>>>>> Let me know if you need anything else. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Vojtech >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>>>> From: "Vojtech Szocs" <vsz...@redhat.com> >>>>>>>>>>>>>>>> To: "Sandro Bonazzola" <sbona...@redhat.com> >>>>>>>>>>>>>>>> Cc: "Eyal Edri" <ee...@redhat.com>, "Juan Hernandez" >>>>>>>>>>>>>>>> <jhern...@redhat.com>, "Kiril Nesenko" <knese...@redhat.com>, >>>>>>>>>>>>>>>> "arch" <arch@ovirt.org> >>>>>>>>>>>>>>>> Sent: Tuesday, February 18, 2014 6:54:20 PM >>>>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> In your log I see following error: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>>> [INFO] Computing all possible rebind results for >>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' >>>>>>>>>>>>>>>> [INFO] Rebinding >>>>>>>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources >>>>>>>>>>>>>>>> [INFO] Invoking generator >>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator >>>>>>>>>>>>>>>> [INFO] [ERROR] Generator >>>>>>>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator' >>>>>>>>>>>>>>>> threw >>>>>>>>>>>>>>>> an exception while rebinding >>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' >>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever >>>>>>>>>>>>>>>> computed >>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>> removeBrickCommitRequired >>>>>>>>>>>>>>>> [INFO] at >>>>>>>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475) >>>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I checked again logs at >>>>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/212/label=fedora19-host/consoleFull >>>>>>>>>>>>>>>> and indeed there is a similar error: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>>> [INFO] Computing all possible rebind results for >>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' >>>>>>>>>>>>>>>> [INFO] Rebinding >>>>>>>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources >>>>>>>>>>>>>>>> [INFO] Invoking generator >>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator >>>>>>>>>>>>>>>> [INFO] [ERROR] Generator >>>>>>>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator' >>>>>>>>>>>>>>>> threw >>>>>>>>>>>>>>>> an exception while rebinding >>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' >>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException >>>>>>>>>>>>>>>> [INFO] at >>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40) >>>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> NullPointerException when processing ApplicationResources but >>>>>>>>>>>>>>>> occuring >>>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>> different places (ImageResourceGenerator vs. >>>>>>>>>>>>>>>> InlineResourceContext) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Now I understand why Maven build fails without any details: >>>>>>>>>>>>>>>> - GWT deferred binding fails for ApplicationResources (i.e. >>>>>>>>>>>>>>>> uncaught >>>>>>>>>>>>>>>> NPE) >>>>>>>>>>>>>>>> - GWT compilation proceeds, fails to link with >>>>>>>>>>>>>>>> ApplicationResources >>>>>>>>>>>>>>>> - GWT compilation task fails with non-OK status code >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> So the issue here is GWT deferred binding generator >>>>>>>>>>>>>>>> (InlineClientBundleGenerator) failure during GWT compilation. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> GWT compiler writes extra Java files during deferred binding >>>>>>>>>>>>>>>> phase, so this could point to issue with local environment.. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I will try to investigate this and look at GWT source code. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Sandro Bonazzola >>>>>>>>>>>>>> Better technology. Faster innovation. Powered by community >>>>>>>>>>>>>> collaboration. >>>>>>>>>>>>>> See how it works at redhat.com >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Sandro Bonazzola >>>>>>>>>>>> Better technology. Faster innovation. Powered by community >>>>>>>>>>>> collaboration. >>>>>>>>>>>> See how it works at redhat.com >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Sandro Bonazzola >>>>>>>>>> Better technology. Faster innovation. Powered by community >>>>>>>>>> collaboration. >>>>>>>>>> See how it works at redhat.com >>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Sandro Bonazzola >>>>>>>> Better technology. Faster innovation. Powered by community >>>>>>>> collaboration. >>>>>>>> See how it works at redhat.com >>>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Sandro Bonazzola >>>>>> Better technology. Faster innovation. Powered by community >>>>>> collaboration. >>>>>> See how it works at redhat.com >>>>>> >>>> >>>> >>> >>> >>> -- >>> Sandro Bonazzola >>> Better technology. Faster innovation. Powered by community collaboration. >>> See how it works at redhat.com >>> >> _______________________________________________ >> Arch mailing list >> Arch@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/arch >> -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com _______________________________________________ Arch mailing list Arch@ovirt.org http://lists.ovirt.org/mailman/listinfo/arch