H Will, I don't want to break your mojo on this other bughunting but do you see any rpm packaging errors concerning mysql-connector? I added the repo for mysql to find out the connector was already installed.
BTW this is only one of four to five intermittent errors I'm getting. Hardly any run succeeds. On Tue, May 17, 2016 at 1:55 PM, Will Stevens <wstev...@cloudops.com> wrote: > Yes. I did a clean setup in 6 CI environments. With these PRs in place all > 6 had the addHost issue. I reverted and did a clean setup again and all 6 > came up perfectly. > > I am using bubble on commit 9134cf8be930d0b982829da216471feaf214621f to > test. I create the CS and KVM instances and then check out the code and > deploy (two script in bubble). > > Once ACS is up and running it does a DeployDC and it fails at this point. > > I can test your PR and send you the logs. > On May 17, 2016 12:40 AM, "Rohit Yadav" <bhais...@apache.org> wrote: > >> Will, >> >> Alright, I've reopened a PR with the changes that were reverted to be >> reviewed/tested separately again. Can you share if you're able to get your >> CI env working again after the revert? >> Can you share in detail how may I reproduce the issues you get that you >> think are because of Nio related changes, so I can try to fix them. Thanks. >> >> Regards. >> >> On Tue, May 17, 2016 at 2:49 AM, Will Stevens <wstev...@cloudops.com> >> wrote: >> >>> I would be running these commands: >>> >>> git revert 540d9572fd491db3ce182d26636fc74ada4e171c (1543) >>> git revert f88cb880974fa56866492c437af291e40bd1a4f6 -m 1 (1538) >>> git revert 9f970f28b18534dffe33196ead60ea861f501fa9 -m 1 (1534) >>> git revert 7ce0e10fbcd949375e43535aae168421ecdaa562 -m 1 (1493) >>> >>> *Will STEVENS* >>> Lead Developer >>> >>> *CloudOps* *| *Cloud Solutions Experts >>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 >>> w cloudops.com *|* tw @CloudOps_ >>> >>> On Mon, May 16, 2016 at 5:19 PM, Will Stevens <wstev...@cloudops.com> >>> wrote: >>> >>>> I think I am going to have to revert #1493 ( >>>> https://github.com/apache/cloudstack/pull/1493) because it seems to be >>>> responsible for the `addHost` issue I am having in all my CI environments >>>> while doing DeployDC. On completely new deployments, I got the `addHost` >>>> error on all 6 of my CI environments with #1493 included. I have noticed >>>> this as an intermittent issue for the last while, but could not isolate it. >>>> >>>> I have reverted #1493 (and it's descendants) locally and I have not had >>>> the error in my first 3 tests after reverting it. I only started seeing >>>> the problem after #1493. >>>> >>>> With the freeze basically here, I need my CI environments in action >>>> cause I have a few PRs that I really want to run before I merge and >>>> freeze... >>>> >>>> *Will STEVENS* >>>> Lead Developer >>>> >>>> *CloudOps* *| *Cloud Solutions Experts >>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 >>>> w cloudops.com *|* tw @CloudOps_ >>>> >>>> On Mon, May 16, 2016 at 1:53 PM, Will Stevens <wstev...@cloudops.com> >>>> wrote: >>>> >>>>> I mean revert them locally to test (for now)... >>>>> >>>>> *Will STEVENS* >>>>> Lead Developer >>>>> >>>>> *CloudOps* *| *Cloud Solutions Experts >>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 >>>>> w cloudops.com *|* tw @CloudOps_ >>>>> >>>>> On Mon, May 16, 2016 at 1:53 PM, Will Stevens <wstev...@cloudops.com> >>>>> wrote: >>>>> >>>>>> I am getting it more than not right now. I am concerned we have a >>>>>> bigger issue here. It only started happening for me after #1493 went in, >>>>>> but I don't want to blame it on that without knowing for sure. >>>>>> >>>>>> Maybe I will revert those ones and see if it changes anything... :( >>>>>> >>>>>> *Will STEVENS* >>>>>> Lead Developer >>>>>> >>>>>> *CloudOps* *| *Cloud Solutions Experts >>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 >>>>>> w cloudops.com *|* tw @CloudOps_ >>>>>> >>>>>> On Mon, May 16, 2016 at 1:45 PM, Daan Hoogland < >>>>>> daan.hoogl...@gmail.com> wrote: >>>>>> >>>>>>> yes, I'm having the same thing. I need to restart the kvm1 vm to >>>>>>> have it be >>>>>>> able to connect to the ms. It seems intermittent. >>>>>>> >>>>>>> On Mon, May 16, 2016 at 7:38 PM, Will Stevens <wstev...@cloudops.com> >>>>>>> wrote: >>>>>>> >>>>>>> > I am currently running into an issue in DeployDC that I started >>>>>>> getting on >>>>>>> > Thursday or Friday. I am not sure if it is a bubble issue or an >>>>>>> issue with >>>>>>> > ACS. >>>>>>> > >>>>>>> > I am trying to get it sorted out now... >>>>>>> > >>>>>>> > ==== Deploy DC Started ==== >>>>>>> > >>>>>>> > Exception Occurred :['Traceback (most recent call last):\n', ' >>>>>>> File >>>>>>> > >>>>>>> "/data/git/cs1/cloudstack/tools/marvin/marvin/deployDataCenter.py", line >>>>>>> > 142, in addHosts\n ret = self.__apiClient.addHost(hostcmd)\n', >>>>>>> ' File >>>>>>> > >>>>>>> > >>>>>>> "/usr/lib/python2.7/site-packages/marvin/cloudstackAPI/cloudstackAPIClient.py", >>>>>>> > line 1530, in addHost\n response = >>>>>>> > self.connection.marvinRequest(command, response_type=response, >>>>>>> > method=method)\n', ' File >>>>>>> > "/usr/lib/python2.7/site-packages/marvin/cloudstackConnection.py", >>>>>>> line >>>>>>> > 379, in marvinRequest\n raise e\n', 'CloudstackAPIException: >>>>>>> Execute >>>>>>> > cmd: addhost failed, due to: errorCode: 530, errorText:Unable to >>>>>>> add the >>>>>>> > host\n'] >>>>>>> > >>>>>>> > Exception Occurred :['Traceback (most recent call last):\n', ' >>>>>>> File >>>>>>> > >>>>>>> "/data/git/cs1/cloudstack/tools/marvin/marvin/deployDataCenter.py", line >>>>>>> > 142, in addHosts\n ret = self.__apiClient.addHost(hostcmd)\n', >>>>>>> ' File >>>>>>> > >>>>>>> > >>>>>>> "/usr/lib/python2.7/site-packages/marvin/cloudstackAPI/cloudstackAPIClient.py", >>>>>>> > line 1530, in addHost\n response = >>>>>>> > self.connection.marvinRequest(command, response_type=response, >>>>>>> > method=method)\n', ' File >>>>>>> > "/usr/lib/python2.7/site-packages/marvin/cloudstackConnection.py", >>>>>>> line >>>>>>> > 379, in marvinRequest\n raise e\n', 'CloudstackAPIException: >>>>>>> Execute >>>>>>> > cmd: addhost failed, due to: errorCode: 530, errorText:Guid is not >>>>>>> updated >>>>>>> > for cluster with specified cluster id; need to wait for hosts in >>>>>>> this >>>>>>> > cluster to come up\n'] >>>>>>> > >>>>>>> > ===deploy dc failed, so cleaning the created entries=== >>>>>>> > >>>>>>> > *Will STEVENS* >>>>>>> > Lead Developer >>>>>>> > >>>>>>> > *CloudOps* *| *Cloud Solutions Experts >>>>>>> > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 >>>>>>> > w cloudops.com *|* tw @CloudOps_ >>>>>>> > >>>>>>> > On Mon, May 16, 2016 at 12:51 PM, Daan Hoogland < >>>>>>> daan.hoogl...@gmail.com> >>>>>>> > wrote: >>>>>>> > >>>>>>> > > I understoud the request to try it out. I have #1547 running >>>>>>> once again >>>>>>> > now >>>>>>> > > and will try Remi's new version with the next one i'll run. >>>>>>> > > >>>>>>> > > On Mon, May 16, 2016 at 6:34 PM, Will Stevens < >>>>>>> williamstev...@gmail.com> >>>>>>> > > wrote: >>>>>>> > > >>>>>>> > > > Sorry. My phone decides to write whatever the hell it wants >>>>>>> to. I need >>>>>>> > to >>>>>>> > > > proof read everything on my phone. You can probably >>>>>>> understand my >>>>>>> > > previous >>>>>>> > > > email though. >>>>>>> > > > On May 16, 2016 12:32 PM, "Will Stevens" < >>>>>>> williamstev...@gmail.com> >>>>>>> > > wrote: >>>>>>> > > > >>>>>>> > > > > Remi and I have been discussing this. I suggested we >>>>>>> implement an >>>>>>> > > > override >>>>>>> > > > > system so she could reuse lost of the code, but be able to >>>>>>> override >>>>>>> > > files >>>>>>> > > > > specific to cosmic and cloudstack without impacting each >>>>>>> other. Remi >>>>>>> > > > > implemented this concept and asked me to verify it is >>>>>>> working as >>>>>>> > > > expected, >>>>>>> > > > > but I didn't want to push my luck with the freeze so close >>>>>>> and I had >>>>>>> > > > > working bubbles. >>>>>>> > > > > >>>>>>> > > > > I think the recent changes are going to help mitigate issues >>>>>>> going >>>>>>> > > > > forward, but I have not had a chance to validate yet. >>>>>>> > > > > On May 16, 2016 12:27 PM, "Daan Hoogland" < >>>>>>> daan.hoogl...@gmail.com> >>>>>>> > > > wrote: >>>>>>> > > > > >>>>>>> > > > >> trying that one but I think we need to add that repo to the >>>>>>> cs >>>>>>> > system >>>>>>> > > in >>>>>>> > > > >> or >>>>>>> > > > >> before the build script. Also this is not the last (HEAD) >>>>>>> of the >>>>>>> > > master >>>>>>> > > > >> branch of the bubble. I saw some cosmic specific changes. >>>>>>> We have >>>>>>> > some >>>>>>> > > > >> challenges on the road ahead. >>>>>>> > > > >> >>>>>>> > > > >> On Mon, May 16, 2016 at 3:06 PM, Will Stevens < >>>>>>> > wstev...@cloudops.com> >>>>>>> > > > >> wrote: >>>>>>> > > > >> >>>>>>> > > > >> > Daan, I am using bubble commit >>>>>>> > > > 9134cf8be930d0b982829da216471feaf214621f >>>>>>> > > > >> and >>>>>>> > > > >> > things seem to be working. Let me know if that is not >>>>>>> working for >>>>>>> > > > >> you... >>>>>>> > > > >> > >>>>>>> > > > >> > *Will STEVENS* >>>>>>> > > > >> > Lead Developer >>>>>>> > > > >> > >>>>>>> > > > >> > *CloudOps* *| *Cloud Solutions Experts >>>>>>> > > > >> > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 >>>>>>> > > > >> > w cloudops.com *|* tw @CloudOps_ >>>>>>> > > > >> > >>>>>>> > > > >> > On Mon, May 16, 2016 at 8:02 AM, Daan Hoogland < >>>>>>> > > > daan.hoogl...@gmail.com >>>>>>> > > > >> > >>>>>>> > > > >> > wrote: >>>>>>> > > > >> > >>>>>>> > > > >> > > Thanks Rohit, >>>>>>> > > > >> > > >>>>>>> > > > >> > > I did try the manual install on the build host but that >>>>>>> didn't >>>>>>> > > help. >>>>>>> > > > >> I'll >>>>>>> > > > >> > > have a look at the extra yum repo to add later. >>>>>>> > > > >> > > >>>>>>> > > > >> > > On Mon, May 16, 2016 at 12:26 PM, Rohit Yadav < >>>>>>> > > > >> rohit.ya...@shapeblue.com >>>>>>> > > > >> > > >>>>>>> > > > >> > > wrote: >>>>>>> > > > >> > > >>>>>>> > > > >> > > > I checked this again today, with the recent changes >>>>>>> wrt rpm >>>>>>> > > > >> > > deps/packaging >>>>>>> > > > >> > > > the mysql-connector-python is not availble in default >>>>>>> CentOS >>>>>>> > > > >> > repositories >>>>>>> > > > >> > > > therefore we'll need to modify documentation to >>>>>>> install it >>>>>>> > from >>>>>>> > > > >> MySQL >>>>>>> > > > >> > yum >>>>>>> > > > >> > > > repos: >>>>>>> > > > >> > > > http://dev.mysql.com/downloads/repo/yum >>>>>>> > > > >> > > > >>>>>>> > > > >> > > > Regards, >>>>>>> > > > >> > > > Rohit Yadav >>>>>>> > > > >> > > > >>>>>>> > > > >> > > > >>>>>>> > > > >> > > > Regards, >>>>>>> > > > >> > > > >>>>>>> > > > >> > > > Rohit Yadav >>>>>>> > > > >> > > > >>>>>>> > > > >> > > > rohit.ya...@shapeblue.com >>>>>>> > > > >> > > > www.shapeblue.com >>>>>>> > > > >> > > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK >>>>>>> > > > >> > > > @shapeblue >>>>>>> > > > >> > > > On May 16 2016, at 8:37 am, Rohit Yadav < >>>>>>> > > > rohit.ya...@shapeblue.com> >>>>>>> > > > >> > > wrote: >>>>>>> > > > >> > > > >>>>>>> > > > >> > > > Daan, >>>>>>> > > > >> > > > >>>>>>> > > > >> > > > mysql-connector-python was recently added as a >>>>>>> pure-python >>>>>>> > > > >> dependency, >>>>>>> > > > >> > > can >>>>>>> > > > >> > > > you try to install that on your host system and retry >>>>>>> rpm >>>>>>> > > > building. >>>>>>> > > > >> > > > >>>>>>> > > > >> > > > Regards. >>>>>>> > > > >> > > > ________________________________________ >>>>>>> > > > >> > > > From: Daan Hoogland <daan.hoogl...@gmail.com> >>>>>>> > > > >> > > > Sent: 15 May 2016 14:34:49 >>>>>>> > > > >> > > > To: dev >>>>>>> > > > >> > > > Subject: old error back? >>>>>>> > > > >> > > > >>>>>>> > > > >> > > > anybody else seeing this in the bubble or during rpm >>>>>>> build? >>>>>>> > > > >> > > > >>>>>>> > > > >> > > > ``` >>>>>>> > > > >> > > > error: Failed build dependencies: >>>>>>> > > > >> > > > mysql-connector-python is needed by >>>>>>> > > > >> > > > cloudstack-4.9.0-SNAPSHOT.el7.centos.x86_64 >>>>>>> > > > >> > > > RPM Build Failed >>>>>>> > > > >> > > > ``` >>>>>>> > > > >> > > > >>>>>>> > > > >> > > > -- >>>>>>> > > > >> > > > Daan >>>>>>> > > > >> > > > >>>>>>> > > > >> > > > Regards, >>>>>>> > > > >> > > > >>>>>>> > > > >> > > > Rohit Yadav >>>>>>> > > > >> > > > >>>>>>> > > > >> > > > rohit.ya...@shapeblue.com >>>>>>> > > > >> > > > www.shapeblue.com >>>>>>> > > > >> > > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK >>>>>>> > > > >> > > > @shapeblue >>>>>>> > > > >> > > > >>>>>>> > > > >> > > >>>>>>> > > > >> > > >>>>>>> > > > >> > > >>>>>>> > > > >> > > -- >>>>>>> > > > >> > > Daan >>>>>>> > > > >> > > >>>>>>> > > > >> > >>>>>>> > > > >> >>>>>>> > > > >> >>>>>>> > > > >> >>>>>>> > > > >> -- >>>>>>> > > > >> Daan >>>>>>> > > > >> >>>>>>> > > > > >>>>>>> > > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > -- >>>>>>> > > Daan >>>>>>> > > >>>>>>> > >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Daan >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> -- Daan
