I added a job that cleans all FF processes after royale-asjs_jsonly. Memory usage was up to 90% with no jobs running.
From: Yishay Weiss<mailto:yishayj...@hotmail.com> Sent: Wednesday, December 29, 2021 8:59 AM To: dev@royale.apache.org<mailto:dev@royale.apache.org> Subject: RE: Build failed in Jenkins: royale-asjs_MXTests #2215 I samples CI just now and got this [1]. It doesn’t look like there a re a ton of FF processes, and CPU is more or less the same, but it looks like memory (WS column) is an issue and has increased significantly. The task manager is showing FF at the top of the memory column, and 70% usage when no jobs are running. This might explain why some jobs run out of memory. I don’t know if it explains credit running out. [1] PS C:\Users\ApacheRoyaleCI> ps FireFox Handles NPM(K) PM(K) WS(K) CPU(s) Id SI ProcessName ------- ------ ----- ----- ------ -- -- ----------- 316 29 48452 30432 12.55 1324 0 firefox 139 13 18416 1876 0.09 2436 0 firefox 42 5 1104 156 0.02 3156 0 firefox 302 25 35784 9552 2.14 4428 0 firefox 276 20 25192 4720 1.02 4552 0 firefox 332 26 35212 7240 1.91 4796 0 firefox 1015 102 331500 330536 158.45 4856 0 firefox 298 22 27792 5440 0.89 5324 0 firefox 304 25 36420 11784 6.02 5888 0 firefox 273 20 25472 4508 0.69 6052 0 firefox 308 37 66308 59084 31.05 6232 0 firefox 231 15 20760 3460 0.64 6504 0 firefox 1170 125 600252 334156 3,446.38 6588 0 firefox 294 436 2168288 294412 3,347.95 6592 0 firefox 275 20 25228 4732 1.05 6640 0 firefox 231 15 20844 3300 0.14 6660 0 firefox 276 20 25172 4724 0.86 6712 0 firefox From: Alex Harui<mailto:aha...@adobe.com.INVALID> Sent: Monday, December 20, 2021 8:52 PM To: dev@royale.apache.org<mailto:dev@royale.apache.org> Subject: Re: Build failed in Jenkins: royale-asjs_MXTests #2215 It could be that these FF jobs add up to something, but I think in the past I saw some task that was really using a significant amount of CPU. It would fire up on occasion and then run until we run out of credits. It did not start up right away after re-starting the server. So maybe set a reminder to check after a week and two weeks and three weeks and see there are a ton of FF jobs or some other task is burning up CPU. -Alex On 12/20/21, 10:46 AM, "Josh Tynjala" <joshtynj...@bowlerhat.dev> wrote: As best I can tell, the RoyaleUnit Ant task appears to always destroy the process it launches (whether it's Firefox/Chrome/Flash Player), even if something goes wrong. There's a try/finally around the code where it waits for the test results, and the process is destroyed in the finally block. So even if there's a Java exception in the Ant task while Firefox is running, that code will run. We run Firefox for the Selenium/GeckoDriver stuff too, but I would expect that code would be pretty robust too. I'm less familiar with those other tests, though. -- Josh Tynjala Bowler Hat LLC <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbowlerhat.dev%2F&data=04%7C01%7Caharui%40adobe.com%7C1c16dacf1aad429bb27808d9c3e8ecff%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637756228030747252%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Stfk0GHe9jwnKuAYZD0vOhTe80uj51a1Aocx151Bmj8%3D&reserved=0> On Mon, Dec 20, 2021 at 10:08 AM Yishay Weiss <yishayj...@hotmail.com> wrote: > Sort of related: there are lots of unclosed firefox processes that I think > are leftover from unit tests. That could be the reason we occasionally run > out of memory there. > > > PS C:\Users\ApacheRoyaleCI> ps FireFox > > Handles NPM(K) PM(K) WS(K) CPU(s) Id SI ProcessName > ------- ------ ----- ----- ------ -- -- ----------- > 318 29 54944 57176 2.36 1324 0 firefox > 139 13 18416 13232 0.06 2436 0 firefox > 42 5 1104 4152 0.02 3156 0 firefox > 304 25 35776 40028 0.75 4428 0 firefox > 276 20 25192 28772 0.25 4552 0 firefox > 332 26 35200 32672 0.67 4796 0 firefox > 996 99 287000 292076 39.92 4856 0 firefox > 298 22 27792 24408 0.33 5324 0 firefox > 304 25 35956 32004 1.38 5888 0 firefox > 273 20 25472 29336 0.13 6052 0 firefox > 308 33 96816 116296 3.91 6232 0 firefox > 231 15 20760 18620 0.14 6504 0 firefox > 1148 103 385840 305744 414.09 6588 0 firefox > 294 102 439752 450604 93.88 6592 0 firefox > 275 20 25228 29032 0.20 6640 0 firefox > 231 15 20844 19116 0.06 6660 0 firefox > 276 20 25172 29000 0.20 6712 0 firefox > > From: Alex Harui<mailto:aha...@adobe.com.INVALID> > Sent: Sunday, December 19, 2021 6:31 PM > To: dev@royale.apache.org<mailto:dev@royale.apache.org> > Subject: Re: Build failed in Jenkins: royale-asjs_MXTests #2215 > > Jenkins should be back up and running. Volunteers with access to this > server are welcome to spend some time trying to optimize/eliminate some of > the background tasks. > > When we first started running out of credits I noticed that Windows seemed > to be running some sort of indexer in the background, but didn't have time > to figure out how to keep it from running (or maybe just run in certain > folders). IMO, the CPU usage of that indexer sometimes burns our Azure > credits just fast enough that we run out with about 24 to 48 hours before > the month ends. > > And of course, volunteers are welcome to run Jenkins on their own > server/accounts that have credit cards to pay for the overage. > > -Alex > > On 12/17/21, 12:32 AM, "Alex Harui" <aha...@adobe.com.INVALID> wrote: > > I just got the email that the Jenkins server ran out of Azure credits > and will be suspended for a while. Not exactly sure when the new credits > are released. > > On 12/16/21, 11:49 PM, "Greg Dove" <greg.d...@gmail.com> wrote: > > Thanks again Edward, that was certainly much easier than last time > I set up > for this (which was quite some time ago and required downgrading > to an > older version of Firefox etc). > > However, do you know if it is normal for all tests to pass and > then see > this at the end in js?: > > [java] RESULT: > > scriptName=C:\development\asf\royale-asjs/mustella/tests/mxtests/basicTests/spark/scripts/ButtonTestScript.mxml > id=SparkButtonTest1 result=pass elapsed=1438 phase=body > started=1639727175704 extraInfo= msg= > [java] 1639727177157 Marionette INFO Stopped > listening > on port 53835 > [java] > [java] ###!!! [Child][RunMessage] Error: Channel closing: too > late to > send/recv, messages will be lost > [java] > [java] > [java] ###!!! [Child][MessageChannel::SendAndWait] Error: > Channel > error: cannot send/recv > [java] > > BUILD FAILED > C:\development\asf\royale-asjs\build.xml:1577: Java returned: 1 > > > thanks, > Greg > > On Thu, Dec 16, 2021 at 4:26 PM Greg Dove <greg.d...@gmail.com> > wrote: > > > > > Thanks Edward, I will keep that in mind for the future. > Meanwhile I made > > changes today that got it to pass. > > > > > > On Thu, Dec 16, 2021 at 4:21 PM Edward Stangler < > estang...@bradmark.com> > > wrote: > > > >> > >> If you have Windows and Firefox installed, then just download > and > >> extract geckodriver-v0.30.0-win64.zip: > >> > >> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmozilla%2Fgeckodriver%2Freleases&data=04%7C01%7Caharui%40adobe.com%7C1c16dacf1aad429bb27808d9c3e8ecff%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637756228030747252%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ZxgRnxKZI0cldKjXuGA3fpbQ1tFfYlgeiQp6y%2Bln%2FFI%3D&reserved=0 > >> > >> and do this (in your build+test environment): > >> > >> set > GECKODRIVER_HOME=c:\geckodriver-v0.30.0-win64\geckodriver.exe > >> ant mxtests-run-js > >> > >> (ant basictest-run-js passes) > >> > >> Buried in the output is: > >> > >> TypeError: panelView.contentArea is undefined > >> > >> > >> > >> On 12/15/2021 2:03 PM, Greg Dove wrote: > >> > I will have to come back to that, it is something I consider > to be not > >> an > >> > easy setup, so need to figure it out. > >> > > >> > > >> > On Thu, Dec 16, 2021 at 8:48 AM Greg Dove wrote: > >> > > >> >> Ok, so I just did > >> >> ant runmxtests > >> >> and it ran the swf tests, they passed. > >> >> So it looks like this is failing in js tests. I need to > figure out how > >> to > >> >> run those... > >> > >> > > >
