Re: The Jenkins VMs are increasingly slow / overloaded

2013-07-15 Thread Andrew Purtell
On Mon, Jul 15, 2013 at 3:30 PM, Ted Yu wrote: > Thanks for the notice, Andy. > > bq. End Time: 2013-07-17 00:00 PDT > > Should the End Time be on 07-18 (just guessing) ? > Yeah, I don't know... that's what they said.

Re: The Jenkins VMs are increasingly slow / overloaded

2013-07-15 Thread Ted Yu
Thanks for the notice, Andy. bq. End Time: 2013-07-17 00:00 PDT Should the End Time be on 07-18 (just guessing) ? On Mon, Jul 15, 2013 at 3:27 PM, Andrew Purtell wrote: > The EC2 Jenkins for HBase is still active, just running a reduced schedule. > > I have received the following notice from A

Re: The Jenkins VMs are increasingly slow / overloaded

2013-07-15 Thread Andrew Purtell
The EC2 Jenkins for HBase is still active, just running a reduced schedule. I have received the following notice from Amazon regarding the VM http://54.241.6.143/, FYI: Zone: us-west-1a Event Type: system-maintenance Description: Your instance network connections will be restarted during this win

Re: The Jenkins VMs are increasingly slow / overloaded

2013-06-10 Thread Stack
Sounds good Andrew. Its been useful having another perspective. On Mon, Jun 10, 2013 at 11:14 AM, Andrew Purtell wrote: > Since the situation with ASF Jenkins seems better (please correct me if > wrong), we are going to scale back HBase jobs on the Jenkins at > http://54.241.6.143/ to one execu

Re: The Jenkins VMs are increasingly slow / overloaded

2013-06-10 Thread Andrew Purtell
Thanks Elliott. This reminds me... to update, we've retained the SCM polling and daily builds for 0.94 and 0.94-security. All of the other builds are set to execute once per day at a random time. On Mon, Jun 10, 2013 at 12:42 PM, Elliott Clark wrote: > Thanks for putting so much into that to g

Re: The Jenkins VMs are increasingly slow / overloaded

2013-06-10 Thread Elliott Clark
Thanks for putting so much into that to get us past that rough spot. On Mon, Jun 10, 2013 at 11:14 AM, Andrew Purtell wrote: > Since the situation with ASF Jenkins seems better (please correct me if > wrong), we are going to scale back HBase jobs on the Jenkins at > http://54.241.6.143/ to one e

Re: The Jenkins VMs are increasingly slow / overloaded

2013-06-10 Thread Andrew Purtell
Since the situation with ASF Jenkins seems better (please correct me if wrong), we are going to scale back HBase jobs on the Jenkins at http://54.241.6.143/ to one execution per day. Please let me know if you have any concerns. On Tue, Mar 19, 2013 at 10:48 AM, Andrew Purtell wrote: > I have che

Re: The Jenkins VMs are increasingly slow / overloaded

2013-04-18 Thread Andrew Purtell
This is the second time you have pointed out a non-problem with EC2 Jenkins. Please do some basic sanity checking first. Also, you have not made any positive comments or contributions here, only pointed out what you believe are problems with my volunteer effort here. You may not intend it to be li

Re: The Jenkins VMs are increasingly slow / overloaded

2013-04-18 Thread Ted Yu
Looks like http://repository.jboss.org/nexus/content/groups/public-jboss is having problems. I will just wait for it to come back. On Thu, Apr 18, 2013 at 5:25 PM, Andrew Purtell wrote: > Hi Ted, > > Why are you asking me this? Do you think this is a problem with EC2 > Jenkins? Did you check r

Re: The Jenkins VMs are increasingly slow / overloaded

2013-04-18 Thread Andrew Purtell
Hi Ted, Why are you asking me this? Do you think this is a problem with EC2 Jenkins? Did you check repository.jboss.org ? Does it work for you? This is what I get at http://repository.jboss.org/nexus/content/groups/public-jboss/org/apache/apache/12/apache-12-site_en.xml : Service Temporarily U

Re: The Jenkins VMs are increasingly slow / overloaded

2013-04-18 Thread Ted Yu
Andy: Have you noticed the following from recent EC2 trunk builds ? cause : SiteToolException: The site descriptor cannot be resolved from the repository: ArtifactResolutionException: Unable to locate site descriptor: Could not transfer artifact org.apache:apache:xml:site_en:12 from/to repository.

Re: The Jenkins VMs are increasingly slow / overloaded

2013-04-18 Thread Andrew Purtell
Today I updated the EC2 plugin to see if that resolves the issue with occasional failed slave launches. It looks good so far. On Thu, Mar 21, 2013 at 9:38 PM, Stack wrote: > This machine is now running builds: http://54.241.6.143/After some > messing to fix memory allocation, it seems tests

Re: The Jenkins VMs are increasingly slow / overloaded

2013-04-05 Thread Andrew Purtell
Also, be careful to differentiate between slaves that are "offline" because they are in the process of being launched, and those that are offline because of that bug I mention. (It doesn't happen often but does happen.) If you kill an "offline" slave being launched, this will just cause churn. And

Re: The Jenkins VMs are increasingly slow / overloaded

2013-04-05 Thread Andrew Purtell
This is a bug in the EC2 module for Jenkins. There are other bugs which this one fixes so it's not a big deal relative to those. You have an account on this system. You can easily go on and delete the slaves which end up in offline state. On Fri, Apr 5, 2013 at 4:39 PM, Ted Yu wrote: > Looks li

Re: The Jenkins VMs are increasingly slow / overloaded

2013-04-05 Thread Ted Yu
Looks like 4 ECs Jenkins slaves are offline at the moment ... On Wed, Mar 27, 2013 at 1:19 PM, Ted Yu wrote: > Looks like Apache Jenkins went off several times this week. > > Is it difficult to hook up patching test with the new Jenkins ? > > Thanks > > > On Wed, Mar 27, 2013 at 7:49 AM, Andrew

Re: The Jenkins VMs are increasingly slow / overloaded

2013-03-27 Thread Ted Yu
Looks like Apache Jenkins went off several times this week. Is it difficult to hook up patching test with the new Jenkins ? Thanks On Wed, Mar 27, 2013 at 7:49 AM, Andrew Purtell wrote: > True, but unlike 0.94 the state of 0.95 and trunk is impacted by Stack's > wrangling with Maven to find a

Re: The Jenkins VMs are increasingly slow / overloaded

2013-03-27 Thread Andrew Purtell
True, but unlike 0.94 the state of 0.95 and trunk is impacted by Stack's wrangling with Maven to find a sane site and assembly, a number of build failures are due to that. Also you'll note that prior to yesterday the Linux OOM killer was nuking the bloated Maven processes on the build slaves. Let's

Re: The Jenkins VMs are increasingly slow / overloaded

2013-03-27 Thread Ted Yu
Trunk and 0.95 builds are not in good shape. 0.95 builds have been failing for 32 times. On Apache Jenkins, looks like TestAssignmentManagerOnCluster has failed quite often for 0.95 and trunk builds. On Wed, Mar 27, 2013 at 7:18 AM, Andrew Purtell wrote: > In general moving from using the m1.la

Re: The Jenkins VMs are increasingly slow / overloaded

2013-03-27 Thread Andrew Purtell
In general moving from using the m1.large (2 vcores, 7.5 GB RAM) to the m1.xlarge (4 vcores, 15 GB RAM) instance type for the slaves helped with a build/test timeout, so now I'd about claim the test environment is sane. We are now seeing that replication tests are flapping, occasionally timing out

Re: The Jenkins VMs are increasingly slow / overloaded

2013-03-26 Thread Nick Dimiduk
On Tue, Mar 26, 2013 at 1:28 PM, Andrew Purtell wrote: > The HBase 0.94 build is now testing green! > http://54.241.6.143/job/HBase-0.94/ > ^5! On Tue, Mar 26, 2013 at 1:47 AM, Andrew Purtell wrote: > > > I found that Maven was being killed on the slaves by the Linux OOM killer > > sometimes f

Re: The Jenkins VMs are increasingly slow / overloaded

2013-03-26 Thread Andrew Purtell
Ah well short lived, it's red again, but is probably a real issue. Will dig in. On Tue, Mar 26, 2013 at 10:59 PM, Stack wrote: > Good on you Andrew. > > Now to make 0.95 and trunk green. > > St.Ack > > > On Tue, Mar 26, 2013 at 1:28 PM, Andrew Purtell > wrote: > > > The HBase 0.94 build is now

Re: The Jenkins VMs are increasingly slow / overloaded

2013-03-26 Thread Stack
Good on you Andrew. Now to make 0.95 and trunk green. St.Ack On Tue, Mar 26, 2013 at 1:28 PM, Andrew Purtell wrote: > The HBase 0.94 build is now testing green! > http://54.241.6.143/job/HBase-0.94/ > > AWS approved our request to allow outbound SMTP, so it should be possible > to set up this

Re: The Jenkins VMs are increasingly slow / overloaded

2013-03-26 Thread Andrew Purtell
The HBase 0.94 build is now testing green! http://54.241.6.143/job/HBase-0.94/ AWS approved our request to allow outbound SMTP, so it should be possible to set up this Jenkins to mail dev@ now. On Tue, Mar 26, 2013 at 1:47 AM, Andrew Purtell wrote: > I found that Maven was being killed on the

Re: The Jenkins VMs are increasingly slow / overloaded

2013-03-25 Thread Andrew Purtell
I found that Maven was being killed on the slaves by the Linux OOM killer sometimes for >= 0.95. Seems the m1.large process didn't have enough memory to host the Jenkins slave, Maven with its 3G+ heap, and the forked JVMs for the medium and large tests at the same time. Switching to the m1.xlarge t

Re: The Jenkins VMs are increasingly slow / overloaded

2013-03-23 Thread Andrew Purtell
We got an alert from AWS about SMTP traffic out of that instance. Seems we have to apply for permission to send emails out directly. For now they are blocking it. I'm working on that. On Mar 23, 2013, at 9:24 PM, Stack wrote: > On Sat, Mar 23, 2013 at 3:11 AM, Andrew Purtell wrote: > >> It

Re: The Jenkins VMs are increasingly slow / overloaded

2013-03-23 Thread Stack
On Sat, Mar 23, 2013 at 3:11 AM, Andrew Purtell wrote: > It was me. If they are stomping on something you are doing please feel free > to disable or delete them. > > No worries. Was just wondering. They are all good builds to have. I tried to get it to email dev list but doesn't seem to be wor

Re: The Jenkins VMs are increasingly slow / overloaded

2013-03-23 Thread Andrew Purtell
It was me. If they are stomping on something you are doing please feel free to disable or delete them. On Saturday, March 23, 2013, Stack wrote: > Who added all the new hbase-on-hadoop2 builds and on-security builds? Was > it you Andrew? > St.Ack > > > On Thu, Mar 21, 2013 at 9:38 PM, Stack > >

Re: The Jenkins VMs are increasingly slow / overloaded

2013-03-22 Thread Stack
Who added all the new hbase-on-hadoop2 builds and on-security builds? Was it you Andrew? St.Ack On Thu, Mar 21, 2013 at 9:38 PM, Stack wrote: > This machine is now running builds: http://54.241.6.143/After some > messing to fix memory allocation, it seems tests are failing legitimately > n

Re: The Jenkins VMs are increasingly slow / overloaded

2013-03-21 Thread Stack
This machine is now running builds: http://54.241.6.143/After some messing to fix memory allocation, it seems tests are failing legitimately now for both 0.94 and trunk. They warrant fixing (thanks for setting this up Andrew). St.Ack On Tue, Mar 19, 2013 at 10:48 AM, Andrew Purtell wrote:

Re: The Jenkins VMs are increasingly slow / overloaded

2013-03-21 Thread Stack
I am looking at the teething issues getting build running here Will report back St.Ack On Tue, Mar 19, 2013 at 10:48 AM, Andrew Purtell wrote: > I have checked with the folks here and until/unless the situation with ASF > Jenkins VMs improves, we can provide a hosted Jenkins up in EC2, i

Re: The Jenkins VMs are increasingly slow / overloaded

2013-03-20 Thread Andrew Purtell
Drat, wrong recipient. Well, that won't work now anyway. :-) On Wed, Mar 20, 2013 at 10:44 PM, Andrew Purtell wrote: > Go to http://54.241.6.143 Log in as user "yuzhihong" current password > "PravAZa2". > > > On Wed, Mar 20, 2013 at 10:39 PM, Ted Yu wrote: > >> Thanks Andrew for this effort. >

Re: The Jenkins VMs are increasingly slow / overloaded

2013-03-20 Thread Andrew Purtell
Go to http://54.241.6.143 Log in as user "yuzhihong" current password "PravAZa2". On Wed, Mar 20, 2013 at 10:39 PM, Ted Yu wrote: > Thanks Andrew for this effort. > > Can I get access as well ? > > On Wed, Mar 20, 2013 at 2:37 PM, Andrew Purtell > wrote: > > > We have an EC2 hosted Jenkins ma

Re: The Jenkins VMs are increasingly slow / overloaded

2013-03-20 Thread Ted Yu
Thanks Andrew for this effort. Can I get access as well ? On Wed, Mar 20, 2013 at 2:37 PM, Andrew Purtell wrote: > We have an EC2 hosted Jenkins master and slave pool set up in us-west-1. > There > are 'HBase-TRUNK' and 'HBase-0.94' projects with bare bones configuration. > I've > mailed access

Re: The Jenkins VMs are increasingly slow / overloaded

2013-03-20 Thread Andrew Purtell
We have an EC2 hosted Jenkins master and slave pool set up in us-west-1. There are 'HBase-TRUNK' and 'HBase-0.94' projects with bare bones configuration. I've mailed access information and credentials to stack, larsh, and ndimiduk -- a list drawn up simply based on replies to this thread. Committer

Re: The Jenkins VMs are increasingly slow / overloaded

2013-03-20 Thread Andrew Purtell
g you can do will help :) > > > > > From: Andrew Purtell > To: "dev@hbase.apache.org" > Sent: Tuesday, March 19, 2013 4:30 PM > Subject: Re: The Jenkins VMs are increasingly slow / overloaded > > I'm setting it up now. The Jenkins host is

Re: The Jenkins VMs are increasingly slow / overloaded

2013-03-19 Thread lars hofhansl
ch 19, 2013 4:30 PM Subject: Re: The Jenkins VMs are increasingly slow / overloaded I'm setting it up now. The Jenkins host is ready, working on the PV AMI for the slaves. > Should we try and organize a whiparound among orgs to help pay for it once its up and running? Sure, if we pool re

Re: The Jenkins VMs are increasingly slow / overloaded

2013-03-19 Thread Andrew Purtell
I'm setting it up now. The Jenkins host is ready, working on the PV AMI for the slaves. > Should we try and organize a whiparound among orgs to help pay for it once its up and running? Sure, if we pool resources then we could pay for more parallel instances. We could also consider more interesti

Re: The Jenkins VMs are increasingly slow / overloaded

2013-03-19 Thread Nick Dimiduk
+1 Jenkins EC2 plugin works pretty well, though whatever makes a build machine get "stuck" is equally applicable on EC2. On Tue, Mar 19, 2013 at 10:48 AM, Andrew Purtell wrote: > I have checked with the folks here and until/unless the situation with ASF > Jenkins VMs improves, we can provide a ho

Re: The Jenkins VMs are increasingly slow / overloaded

2013-03-19 Thread Stack
+1 if can be done. Can point our CI link over there. What help you need Andrew? Should we try and organize a whiparound among orgs to help pay for it once its up and running? I've been on and off working w/ Roman to get our IT builds running over on bigtop but currently stuck on this strange is

Re: The Jenkins VMs are increasingly slow / overloaded

2013-03-19 Thread Andrew Purtell
I have checked with the folks here and until/unless the situation with ASF Jenkins VMs improves, we can provide a hosted Jenkins up in EC2, in us-west-1: - A m1.large instance running 24/7 hosting Jenkins - A pool of 5 m1.large instances serving as Jenkins slaves, managed by Jenkins with its

Re: The Jenkins VMs are increasingly slow / overloaded

2013-03-19 Thread Nick Dimiduk
What, like travis-ci.org ? On Tue, Mar 19, 2013 at 8:49 AM, Andrew Purtell wrote: > Would the HBase dev community be open to alternate hosted Jenkins test > infrastructure? > > -- > Best regards, > >- Andy > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > (via To

The Jenkins VMs are increasingly slow / overloaded

2013-03-19 Thread Andrew Purtell
Would the HBase dev community be open to alternate hosted Jenkins test infrastructure? -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)