RE: Unable to add Ceph KVM node in cloudstack

[Andrija Panic]

Plese enable debug loging in acs agent,restart and retry to add host and
please post logs.
On Sep 14, 2015 8:34 AM, "Shetty, Pradeep"  wrote:

> Yes, libvirt has been compiled to support rbd and using ceph's qemu rpm
> packages.
>
>
> Regards,
> Pradeep
>
> -Original Message-
> From: Andrija Panic [mailto:andrija.pa...@gmail.com]
> Sent: 14 September 2015 11:52
> To: users@cloudstack.apache.org
> Cc: Vadim Kimlaychuk
> Subject: RE: Unable to add Ceph KVM node in cloudstack
>
> Did you compile libvirt and qemu with RBD support?
> Centos6.5 ( libvitt and qemu) does not support RBD/ceph as storage out of
> the box...
> On Sep 14, 2015 7:37 AM, "Shetty, Pradeep"  wrote:
>
> > Hi,
> >
> > Is Ceph 0.94 Hammer compatible/stable with CentOS 6.5 build and
> > cloudstack version 4.4.2
> >
> > Thanks,
> > Pradeep
> >
> > -Original Message-
> > From: Andrija Panic [mailto:andrija.pa...@gmail.com]
> > Sent: 11 September 2015 20:42
> > To: Vadim Kimlaychuk
> > Cc: users@cloudstack.apache.org
> > Subject: Re: Unable to add Ceph KVM node in cloudstack
> >
> > Dont really know that to be honest... even previously, you can patch
> > qemu source code yourslef with RBD support - nothing stops you to do
> > so - so this is just decision from RH or whatever - Im just guessing -
> > anyway it is the way it is - ubuntu oit of the box suport, CentOS not
> afaik...
> >
> > On 11 September 2015 at 13:56, Vadim Kimlaychuk 
> > wrote:
> >
> > > Andrija,
> > >
> > >   If RBD libvirt driver now belongs to RH does that mean
> > > Ubuntu/Debian will loose it some day as it is proprietary software?
> > > Haven't follow Ceph achievements since "Emperor"
> > >
> > > Vadim.
> > >
> > >
> > > On 2015-09-11 14:39, Andrija Panic wrote:
> > >
> > > Np, Im actually very disapointed by RH decision to have this
> > > approach
> > > -
> > >> since now CEPH/Inktank is part of RH etc...
> > >> I didnt want to play with Ubuntu at all (dont let me start about
> > >> all the possible issues/bugs we found) but I dont like idea of
> > >> compiling stuff on production...
> > >>
> > >> On 11 September 2015 at 13:34, Vadim Kimlaychuk
> > >> 
> > >> wrote:
> > >>
> > >> Andrija,
> > >>
> > >> I don't have such broad experience with different CentOS/RHEL
> > >> versions, but I am a bit surprised that these distributions does
> > >> not have RBD support at all. That means they have to maintain their
> > >> own "restricted"
> > >> version of these libraries apart from mainstream. Now I am even
> > >> more happy with my Debian/Ubuntu installations :)
> > >>
> > >> Thank you for information!
> > >>
> > >> Vadim.
> > >>
> > >> On 2015-09-11 14:20, Andrija Panic wrote:
> > >>
> > >> Vadim,
> > >> afaik you are right for mgmt server. As for KVM nodes - libvirt and
> > >> qemu/qemu-img in CentOS6.x does NOT have RBD support at all :) I
> > >> have not played a lot with CentOS7, but did quick check long time
> > >> ago and I thing it also doesnt support RBD - Red Hat wants you to
> > >> buy RHEV to support RBD - of course, there are workarround,
> > >> self-compile libvirt etc...similar for qemu/qemu-img.
> > >>
> > >> I know for sure that Ubuntu 14.x suports RBD out of the box - since
> > >> we are running it with CEPH.
> > >>
> > >> Best
> > >>
> > >> On 11 September 2015 at 10:47, Vadim Kimlaychuk
> > >> 
> > >> wrote:
> > >>
> > >> Andrija,
> > >>
> > >> Are you talking about CS Management server OS? I thought it does
> > >> not really matter if it includes libvirt or not. All calls to
> > >> storage should go through hypervisor and any KVM host goes with
> > >> libvirt. The version of libvirt for old OS may not be aligned with
> > >> latest Ceph release that is true. Cloud architect must check
> > >> compatibility version matrix for all products Ceph/KVM/CS anyway.
> > >>
> > >> Vadim.
> > >>
> > >> On 2015-09-11 11:24, Andrija Panic wrote:
> > >>
> > >> RBD for KVM comes out of the box for Ubuntu 14.x (perhaps also
> > >> 12.x)
> > >> - not with CentOS6, and not sure really about CentOS7 (probably not
> > >> also...)
> > >>
> > >> On 11 September 2015 at 09:53, Vadim Kimlaychuk
> > >> 
> > >> wrote:
> > >>
> > >> For KVM hosts RBD comes "out of the box" with libvirt. For
> > >> XenServer Ceph
> > >> (RBD) storage type is not yet implemented. I would say it is very
> > >> hypervisor-specific and does not depend much from Cloudstack.
> > >>
> > >> Regards,
> > >>
> > >> Vadim.
> > >>
> > >> On 2015-09-11 08:55, Shetty, Pradeep wrote:
> > >>
> > >> I have one question:
> > >> Is ceph rbd protocol supported by default on libvirt version
> > >> 0.10.2-54.el6.x86_64 OR do we need to compile libvirt to support
> > >> rbd within cloudstack.
> > >>
> > >> Regards,
> > >> Pradeep
> > >>
> > >> -Original Message-
> > >> From: Vadim Kimlaychuk [mailto:va...@kickcloud.net]
> > >> Sent: 09 September 2015 12:07
> > >> To: users@cloudstack.apache.org
> > >> Cc: Shetty, Pradeep
> > >> Subject: RE: Unable to add Ceph KVM node in cloudstack
> > >>
> > 

RPM Build from Source Fails for CentOS 7.x

[Keerthiraja SJ]

Hi All,

After I follow the doc to build rpm from source there seems to be some
basic issue.

First the source is looking for the tomcat6 and @ the same time inside the
source I could see only
/root/apache-cloudstack-4.5.2-src/packaging/centos63 directory.

Even if I try to run the ./package.sh I could see the below error

Error
===
 ./package.sh
-D_os default
Preparing to package Apache CloudStack 4.5.2
. preparing source tarball
. executing rpmbuild
error: Failed build dependencies:
tomcat6 is needed by cloudstack-4.5.2-1.el7.centos.x86_64
RPM Build Failed
[root@localhost centos63]# pwd
/root/apache-cloudstack-4.5.2-src/packaging/centos63

Is there any way I can do for CentOS 7.x

On CentOS 7.x the default tomcat package is 7.

Thanks,
Keerthi

RE: Unable to add Ceph KVM node in cloudstack

[Shetty, Pradeep]

Yes, libvirt has been compiled to support rbd and using ceph's qemu rpm 
packages.


Regards,
Pradeep

-Original Message-
From: Andrija Panic [mailto:andrija.pa...@gmail.com] 
Sent: 14 September 2015 11:52
To: users@cloudstack.apache.org
Cc: Vadim Kimlaychuk
Subject: RE: Unable to add Ceph KVM node in cloudstack

Did you compile libvirt and qemu with RBD support?
Centos6.5 ( libvitt and qemu) does not support RBD/ceph as storage out of the 
box...
On Sep 14, 2015 7:37 AM, "Shetty, Pradeep"  wrote:

> Hi,
>
> Is Ceph 0.94 Hammer compatible/stable with CentOS 6.5 build and 
> cloudstack version 4.4.2
>
> Thanks,
> Pradeep
>
> -Original Message-
> From: Andrija Panic [mailto:andrija.pa...@gmail.com]
> Sent: 11 September 2015 20:42
> To: Vadim Kimlaychuk
> Cc: users@cloudstack.apache.org
> Subject: Re: Unable to add Ceph KVM node in cloudstack
>
> Dont really know that to be honest... even previously, you can patch 
> qemu source code yourslef with RBD support - nothing stops you to do 
> so - so this is just decision from RH or whatever - Im just guessing - 
> anyway it is the way it is - ubuntu oit of the box suport, CentOS not afaik...
>
> On 11 September 2015 at 13:56, Vadim Kimlaychuk 
> wrote:
>
> > Andrija,
> >
> >   If RBD libvirt driver now belongs to RH does that mean 
> > Ubuntu/Debian will loose it some day as it is proprietary software?
> > Haven't follow Ceph achievements since "Emperor"
> >
> > Vadim.
> >
> >
> > On 2015-09-11 14:39, Andrija Panic wrote:
> >
> > Np, Im actually very disapointed by RH decision to have this 
> > approach
> > -
> >> since now CEPH/Inktank is part of RH etc...
> >> I didnt want to play with Ubuntu at all (dont let me start about 
> >> all the possible issues/bugs we found) but I dont like idea of 
> >> compiling stuff on production...
> >>
> >> On 11 September 2015 at 13:34, Vadim Kimlaychuk 
> >> 
> >> wrote:
> >>
> >> Andrija,
> >>
> >> I don't have such broad experience with different CentOS/RHEL 
> >> versions, but I am a bit surprised that these distributions does 
> >> not have RBD support at all. That means they have to maintain their 
> >> own "restricted"
> >> version of these libraries apart from mainstream. Now I am even 
> >> more happy with my Debian/Ubuntu installations :)
> >>
> >> Thank you for information!
> >>
> >> Vadim.
> >>
> >> On 2015-09-11 14:20, Andrija Panic wrote:
> >>
> >> Vadim,
> >> afaik you are right for mgmt server. As for KVM nodes - libvirt and 
> >> qemu/qemu-img in CentOS6.x does NOT have RBD support at all :) I 
> >> have not played a lot with CentOS7, but did quick check long time 
> >> ago and I thing it also doesnt support RBD - Red Hat wants you to 
> >> buy RHEV to support RBD - of course, there are workarround, 
> >> self-compile libvirt etc...similar for qemu/qemu-img.
> >>
> >> I know for sure that Ubuntu 14.x suports RBD out of the box - since 
> >> we are running it with CEPH.
> >>
> >> Best
> >>
> >> On 11 September 2015 at 10:47, Vadim Kimlaychuk 
> >> 
> >> wrote:
> >>
> >> Andrija,
> >>
> >> Are you talking about CS Management server OS? I thought it does 
> >> not really matter if it includes libvirt or not. All calls to 
> >> storage should go through hypervisor and any KVM host goes with 
> >> libvirt. The version of libvirt for old OS may not be aligned with 
> >> latest Ceph release that is true. Cloud architect must check 
> >> compatibility version matrix for all products Ceph/KVM/CS anyway.
> >>
> >> Vadim.
> >>
> >> On 2015-09-11 11:24, Andrija Panic wrote:
> >>
> >> RBD for KVM comes out of the box for Ubuntu 14.x (perhaps also 
> >> 12.x)
> >> - not with CentOS6, and not sure really about CentOS7 (probably not
> >> also...)
> >>
> >> On 11 September 2015 at 09:53, Vadim Kimlaychuk 
> >> 
> >> wrote:
> >>
> >> For KVM hosts RBD comes "out of the box" with libvirt. For 
> >> XenServer Ceph
> >> (RBD) storage type is not yet implemented. I would say it is very 
> >> hypervisor-specific and does not depend much from Cloudstack.
> >>
> >> Regards,
> >>
> >> Vadim.
> >>
> >> On 2015-09-11 08:55, Shetty, Pradeep wrote:
> >>
> >> I have one question:
> >> Is ceph rbd protocol supported by default on libvirt version
> >> 0.10.2-54.el6.x86_64 OR do we need to compile libvirt to support 
> >> rbd within cloudstack.
> >>
> >> Regards,
> >> Pradeep
> >>
> >> -Original Message-
> >> From: Vadim Kimlaychuk [mailto:va...@kickcloud.net]
> >> Sent: 09 September 2015 12:07
> >> To: users@cloudstack.apache.org
> >> Cc: Shetty, Pradeep
> >> Subject: RE: Unable to add Ceph KVM node in cloudstack
> >>
> >> Pradeep,
> >>
> >> I have installed CS 4.2 with Ceph storage successfully around year ago.
> >> There were no problem with configuration except the fact that you 
> >> can't configure RBD storage as primary storage during Zone set-up.
> >> It must be NFS.
> >> After Zone is set-up and running you may add another primary 
> >> storage and there you have list of choices where RBD type is present

RE: Unable to add Ceph KVM node in cloudstack

[Andrija Panic]

Did you compile libvirt and qemu with RBD support?
Centos6.5 ( libvitt and qemu) does not support RBD/ceph as storage out of
the box...
On Sep 14, 2015 7:37 AM, "Shetty, Pradeep"  wrote:

> Hi,
>
> Is Ceph 0.94 Hammer compatible/stable with CentOS 6.5 build and cloudstack
> version 4.4.2
>
> Thanks,
> Pradeep
>
> -Original Message-
> From: Andrija Panic [mailto:andrija.pa...@gmail.com]
> Sent: 11 September 2015 20:42
> To: Vadim Kimlaychuk
> Cc: users@cloudstack.apache.org
> Subject: Re: Unable to add Ceph KVM node in cloudstack
>
> Dont really know that to be honest... even previously, you can patch qemu
> source code yourslef with RBD support - nothing stops you to do so - so
> this is just decision from RH or whatever - Im just guessing - anyway it is
> the way it is - ubuntu oit of the box suport, CentOS not afaik...
>
> On 11 September 2015 at 13:56, Vadim Kimlaychuk 
> wrote:
>
> > Andrija,
> >
> >   If RBD libvirt driver now belongs to RH does that mean
> > Ubuntu/Debian will loose it some day as it is proprietary software?
> > Haven't follow Ceph achievements since "Emperor"
> >
> > Vadim.
> >
> >
> > On 2015-09-11 14:39, Andrija Panic wrote:
> >
> > Np, Im actually very disapointed by RH decision to have this approach
> > -
> >> since now CEPH/Inktank is part of RH etc...
> >> I didnt want to play with Ubuntu at all (dont let me start about all
> >> the possible issues/bugs we found) but I dont like idea of
> >> compiling stuff on production...
> >>
> >> On 11 September 2015 at 13:34, Vadim Kimlaychuk 
> >> wrote:
> >>
> >> Andrija,
> >>
> >> I don't have such broad experience with different CentOS/RHEL
> >> versions, but I am a bit surprised that these distributions does not
> >> have RBD support at all. That means they have to maintain their own
> >> "restricted"
> >> version of these libraries apart from mainstream. Now I am even more
> >> happy with my Debian/Ubuntu installations :)
> >>
> >> Thank you for information!
> >>
> >> Vadim.
> >>
> >> On 2015-09-11 14:20, Andrija Panic wrote:
> >>
> >> Vadim,
> >> afaik you are right for mgmt server. As for KVM nodes - libvirt and
> >> qemu/qemu-img in CentOS6.x does NOT have RBD support at all :) I have
> >> not played a lot with CentOS7, but did quick check long time ago and
> >> I thing it also doesnt support RBD - Red Hat wants you to buy RHEV to
> >> support RBD - of course, there are workarround, self-compile libvirt
> >> etc...similar for qemu/qemu-img.
> >>
> >> I know for sure that Ubuntu 14.x suports RBD out of the box - since
> >> we are running it with CEPH.
> >>
> >> Best
> >>
> >> On 11 September 2015 at 10:47, Vadim Kimlaychuk 
> >> wrote:
> >>
> >> Andrija,
> >>
> >> Are you talking about CS Management server OS? I thought it does not
> >> really matter if it includes libvirt or not. All calls to storage
> >> should go through hypervisor and any KVM host goes with libvirt. The
> >> version of libvirt for old OS may not be aligned with latest Ceph
> >> release that is true. Cloud architect must check compatibility
> >> version matrix for all products Ceph/KVM/CS anyway.
> >>
> >> Vadim.
> >>
> >> On 2015-09-11 11:24, Andrija Panic wrote:
> >>
> >> RBD for KVM comes out of the box for Ubuntu 14.x (perhaps also 12.x)
> >> - not with CentOS6, and not sure really about CentOS7 (probably not
> >> also...)
> >>
> >> On 11 September 2015 at 09:53, Vadim Kimlaychuk 
> >> wrote:
> >>
> >> For KVM hosts RBD comes "out of the box" with libvirt. For XenServer
> >> Ceph
> >> (RBD) storage type is not yet implemented. I would say it is very
> >> hypervisor-specific and does not depend much from Cloudstack.
> >>
> >> Regards,
> >>
> >> Vadim.
> >>
> >> On 2015-09-11 08:55, Shetty, Pradeep wrote:
> >>
> >> I have one question:
> >> Is ceph rbd protocol supported by default on libvirt version
> >> 0.10.2-54.el6.x86_64 OR do we need to compile libvirt to support rbd
> >> within cloudstack.
> >>
> >> Regards,
> >> Pradeep
> >>
> >> -Original Message-
> >> From: Vadim Kimlaychuk [mailto:va...@kickcloud.net]
> >> Sent: 09 September 2015 12:07
> >> To: users@cloudstack.apache.org
> >> Cc: Shetty, Pradeep
> >> Subject: RE: Unable to add Ceph KVM node in cloudstack
> >>
> >> Pradeep,
> >>
> >> I have installed CS 4.2 with Ceph storage successfully around year ago.
> >> There were no problem with configuration except the fact that you
> >> can't configure RBD storage as primary storage during Zone set-up.
> >> It must be NFS.
> >> After Zone is set-up and running you may add another primary storage
> >> and there you have list of choices where RBD type is present.
> >> After RBD is added you may assign this as default storage for
> >> particular cluster assuming that each cluster has its own primary
> storage.
> >>
> >> PS. I have also used Ceph object gateway to connect secondary
> >> storage. It works as expected. It would be nice to know which way are
> >> you going to integrate Ceph into CS?
> >>
> >> Vadim.
> >>
> >> On 2015-09-09

RE: CS 4.5.2: all hosts reboot after 3 days at production

[Vadim Kimlaychuk]

Remi,

  I have analyzed situation and found that storage may cause problem 
with host reboot as you wrote before in this thread. Reason for that -- 
we do offline backups from NFS server at that time when hosts fail. 
Basically we copy all files in primary and secondary storage offsite. 
This process starts precisely at 00:00 and somewhere around 00:10 - 
00:40 XenServer host starts to reboot.


  Reading old threads I have found that 
/opt/cloud/bin/xenheartbeat.sh may do this job. Particularly last lines 
at my xenheartbeat.sh  are:


  -
  /usr/bin/logger -t heartbeat "Problem with $hb: not reachable for 
$(($(date +%s) - $lastdate)) seconds, rebooting system!"

  echo b > /proc/sysrq-trigger
  -

  The only "unclear" moment is -- I don't have such line in my logs. 
May this command "echo b > /proc/sysrq-trigger" prevent from writing to 
syslog file? Documentation says that it does reboot immediately without 
synchronizing FS. It seems there is no other place that may do it, but 
still I am not 100% sure.


Vadim.

On 2015-09-13 18:26, Vadim Kimlaychuk wrote:


Remi,

Thank you for hint. At least one problem is identified:

[root@xcp1 ~]# xe pool-⁠list params=all | grep -⁠E 
"ha-⁠enabled|ha-⁠config"

ha-⁠enabled ( RO): false
ha-⁠configuration ( RO):

Where should I look for storage errors? Host? Management server? I have 
checked /var/log/messages and there were only regular messages, no 
"fence" or "reboot" commands.


I have dedicated NFS server that should be accessible all the time (at 
least NIC interfaces are bonded in master-slave mode). Server is used 
for both primary and secondary storage.


Thanks,

Vadim.

On 2015-⁠09-⁠13 14:38, Remi Bergsma wrote:

Hi Vadim,

Not sure what the problem is. Although I do know that when shared 
storage is used, both CloudStack and XenServer will fence (reboot) the 
box to prevent corruption in case access to the network or the storage 
is not possible. What storage do you use?


What does this return on a XenServer?:
xe pool-⁠list params=all | grep -⁠E "ha-⁠enabled|ha-⁠config"

HA should be on, or else a hypervisor crash will not recover properly.

If you search the logs for Fence or reboot, does anything come back?

The logs you mention are nothing to worry about.

Can you tell us in some more details what happens and how we can 
reproduce it?


Regards,
Remi

-⁠-⁠-⁠-⁠-⁠Original Message-⁠-⁠-⁠-⁠-⁠
From: Vadim Kimlaychuk [mailto:va...@kickcloud.net]
Sent: zondag 13 september 2015 9:32
To: users@cloudstack.apache.org
Cc: Remi Bergsma
Subject: Re: CS 4.5.2: all hosts reboot after 3 days at production

Hello Remi,

This issue has nothing to do with CS 4.5.2. We got host reboot after 
precisely 1 week with previous version of CS (4.5.1). Previous version 
has been working without restart for 106 days before. So it is not a 
software issue.


What does really make me unhappy -- accidental host reboot made entire 
cluster unusable. Cloudstack management server was up and running, 
second cluster node was up and running all the time and VM were 
transferred to the second host, but System VMs were not rebooted 
properly by CS and half of the network was down. SSVM and CPVM were in 
"disconnected" status. Client VMs were up, but couldn't connect to 
storage, because VRs were offline. Entire mess.


I have used planned maintenance mode before and cluster worked just 
perfect. We didn't have any single second downtime. But with accidental 
reboot there is no use of clusterization. :(


Vadim.

On 2015-⁠09-⁠08 09:35, Vadim Kimlaychuk wrote:

Hello Remi,

First of all I don't have /⁠var/⁠log/⁠xha.log file. I have examined 
logs

in detail and haven't found any trace that heartbeat has failed. The
only serious problem I have found in management logs before restart is
repeating many times error:

-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠
-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠
2015-⁠09-⁠06 00:47:21,591 DEBUG [c.c.a.m.AgentManagerImpl]
(RouterMonitor-⁠1:ctx-⁠2d67d422) Details from executing class
com.cloud.agent.api.NetworkUsageCommand: Exception:
java.lang.Exception
Message: vpc network usage plugin call failed
Stack: java.lang.Exception: vpc network usage plugin call failed at
com.cloud.hypervisor.xenserver.resource.XenServer56Resource.VPCNetwork
Usage(XenServer56Resource.java:172)
at
com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(Xe
nServer56Resource.java:195)
at
com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeReq
uest(XenServer56Resource.java:62)
at
com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRe
quest(XenServer610Resource.java:87)
at
com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.execut
eRequest(XenServer620SP1Resource.java:65)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAge
ntAttache.java:302)
...

-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠⁠-⁠

RE: Unable to add Ceph KVM node in cloudstack

[Shetty, Pradeep]

Hi,

Is Ceph 0.94 Hammer compatible/stable with CentOS 6.5 build and cloudstack 
version 4.4.2

Thanks,
Pradeep

-Original Message-
From: Andrija Panic [mailto:andrija.pa...@gmail.com] 
Sent: 11 September 2015 20:42
To: Vadim Kimlaychuk
Cc: users@cloudstack.apache.org
Subject: Re: Unable to add Ceph KVM node in cloudstack

Dont really know that to be honest... even previously, you can patch qemu 
source code yourslef with RBD support - nothing stops you to do so - so this is 
just decision from RH or whatever - Im just guessing - anyway it is the way it 
is - ubuntu oit of the box suport, CentOS not afaik...

On 11 September 2015 at 13:56, Vadim Kimlaychuk  wrote:

> Andrija,
>
>   If RBD libvirt driver now belongs to RH does that mean 
> Ubuntu/Debian will loose it some day as it is proprietary software? 
> Haven't follow Ceph achievements since "Emperor"
>
> Vadim.
>
>
> On 2015-09-11 14:39, Andrija Panic wrote:
>
> Np, Im actually very disapointed by RH decision to have this approach 
> -
>> since now CEPH/Inktank is part of RH etc...
>> I didnt want to play with Ubuntu at all (dont let me start about all 
>> the possible issues/bugs we found) but I dont like idea of 
>> compiling stuff on production...
>>
>> On 11 September 2015 at 13:34, Vadim Kimlaychuk 
>> wrote:
>>
>> Andrija,
>>
>> I don't have such broad experience with different CentOS/RHEL 
>> versions, but I am a bit surprised that these distributions does not 
>> have RBD support at all. That means they have to maintain their own 
>> "restricted"
>> version of these libraries apart from mainstream. Now I am even more 
>> happy with my Debian/Ubuntu installations :)
>>
>> Thank you for information!
>>
>> Vadim.
>>
>> On 2015-09-11 14:20, Andrija Panic wrote:
>>
>> Vadim,
>> afaik you are right for mgmt server. As for KVM nodes - libvirt and 
>> qemu/qemu-img in CentOS6.x does NOT have RBD support at all :) I have 
>> not played a lot with CentOS7, but did quick check long time ago and 
>> I thing it also doesnt support RBD - Red Hat wants you to buy RHEV to 
>> support RBD - of course, there are workarround, self-compile libvirt 
>> etc...similar for qemu/qemu-img.
>>
>> I know for sure that Ubuntu 14.x suports RBD out of the box - since 
>> we are running it with CEPH.
>>
>> Best
>>
>> On 11 September 2015 at 10:47, Vadim Kimlaychuk 
>> wrote:
>>
>> Andrija,
>>
>> Are you talking about CS Management server OS? I thought it does not 
>> really matter if it includes libvirt or not. All calls to storage 
>> should go through hypervisor and any KVM host goes with libvirt. The 
>> version of libvirt for old OS may not be aligned with latest Ceph 
>> release that is true. Cloud architect must check compatibility 
>> version matrix for all products Ceph/KVM/CS anyway.
>>
>> Vadim.
>>
>> On 2015-09-11 11:24, Andrija Panic wrote:
>>
>> RBD for KVM comes out of the box for Ubuntu 14.x (perhaps also 12.x) 
>> - not with CentOS6, and not sure really about CentOS7 (probably not 
>> also...)
>>
>> On 11 September 2015 at 09:53, Vadim Kimlaychuk 
>> wrote:
>>
>> For KVM hosts RBD comes "out of the box" with libvirt. For XenServer 
>> Ceph
>> (RBD) storage type is not yet implemented. I would say it is very 
>> hypervisor-specific and does not depend much from Cloudstack.
>>
>> Regards,
>>
>> Vadim.
>>
>> On 2015-09-11 08:55, Shetty, Pradeep wrote:
>>
>> I have one question:
>> Is ceph rbd protocol supported by default on libvirt version
>> 0.10.2-54.el6.x86_64 OR do we need to compile libvirt to support rbd 
>> within cloudstack.
>>
>> Regards,
>> Pradeep
>>
>> -Original Message-
>> From: Vadim Kimlaychuk [mailto:va...@kickcloud.net]
>> Sent: 09 September 2015 12:07
>> To: users@cloudstack.apache.org
>> Cc: Shetty, Pradeep
>> Subject: RE: Unable to add Ceph KVM node in cloudstack
>>
>> Pradeep,
>>
>> I have installed CS 4.2 with Ceph storage successfully around year ago.
>> There were no problem with configuration except the fact that you 
>> can't configure RBD storage as primary storage during Zone set-up.
>> It must be NFS.
>> After Zone is set-up and running you may add another primary storage 
>> and there you have list of choices where RBD type is present.
>> After RBD is added you may assign this as default storage for 
>> particular cluster assuming that each cluster has its own primary storage.
>>
>> PS. I have also used Ceph object gateway to connect secondary 
>> storage. It works as expected. It would be nice to know which way are 
>> you going to integrate Ceph into CS?
>>
>> Vadim.
>>
>> On 2015-09-09 08:09, Shetty, Pradeep wrote:
>>
>> Please share steps for adding ceph RBD as primary storage using KVM 
>> hypervisor in cloudstack.
>>
>> We are well aware of the best practices/recommendations for cloudstack.
>>
>> Regards,
>> Pradeep
>>
>> -Original Message-
>> From: Vadim Kimlaychuk [mailto:va...@kickcloud.net]
>> Sent: 08 September 2015 12:14
>> To: users@cloudstack.apache.org
>> Subject: RE: Unable to add Ce

Re: Repo CS 4.5 for CentOS 7.x Missing

[Keerthiraja SJ]

Hi Guys,

Is there a way I can get the CS 4.5.2 rpm's for CentOS 7.x

http://cloudstack.apt-get.eu/centos7/4.5/


Thanks,
Keerthi

On Sun, Sep 13, 2015 at 6:12 AM, Pierre-Luc Dion  wrote:

> There is one:  http://jenkins.buildacloud.org/view/build-release/  but the
> job is failing, I'm still not sure if it's the jenkins slave the problem.
>
>
> PL
>
>
> On Sat, Sep 12, 2015 at 5:20 PM, Wido den Hollander 
> wrote:
>
> > I don't see a Jenkins job for CentOs 7. Do we even have one?
> >
> >
> >
> > > Op 12 sep. 2015 om 19:29 heeft Keerthiraja SJ 
> het
> > volgende geschreven:
> > >
> > > Hi All,
> > >
> > > I could see there is no packages under the directory.
> > >
> > > http://cloudstack.apt-get.eu/rhel/7/4.5/
> > >
> > > During the instillation of CS 4.5.2 on CentOS 7.x I could see while
> > > pointing the repo from http://cloudstack.apt-get.eu/rhel/7/4.5/ to
> > > http://cloudstack.apt-get.eu/rhel/4.5/
> > >
> > > ---> Package xml-commons-resolver.noarch 0:1.2-15.el7 will be installed
> > > --> Finished Dependency Resolution
> > > Error: Package: cloudstack-common-4.5.2-1.el6.x86_64 (cloudstack)
> > >   Requires: python(abi) = 2.6
> > >   Installed: python-2.7.5-18.el7_1.1.x86_64 (@updates)
> > >   python(abi) = 2.7
> > >   python(abi) = 2.7
> > >   Available: python-2.7.5-16.el7.x86_64 (base)
> > >   python(abi) = 2.7
> > >   python(abi) = 2.7
> > > Error: Package: cloudstack-management-4.5.2-1.el6.x86_64 (cloudstack)
> > >   Requires: tomcat6
> > > You could try using --skip-broken to work around the problem
> > > You could try running: rpm -Va --nofiles --nodigest
> > >
> > >
> > > So kindly update the Cloudstack 4.5.2 for the 7 Version.
> > >
> > > Thanks,
> > > Keerthi
> >
>

Re: CS 4.5.2 Fails to Start.

[Remi Bergsma]

Hi,

The issue is already solved on 4.6 (which is the same as master now).

The two queries were replaced by this one 
(https://github.com/apache/cloudstack/pull/757/files):
UPDATE `cloud`.`configuration` SET value=CONCAT("*.",value) WHERE 
`name`="consoleproxy.url.domain" OR `name`="secstorage.ssl.cert.domain”;


We’re currently working on back porting that to 4.5.x as well. Not sure when 
4.5.3 will come, but if you do it manually it’s the same anyway.

Good luck with your go-live!

Regards,
Remi



On 13/09/15 18:20, "Keerthiraja SJ"  wrote:

>I update the same on
>
>https://issues.apache.org/jira/browse/CLOUDSTACK-8836
>
>Thanks,
>Keerthi
>
>On Sun, Sep 13, 2015 at 9:44 PM, Keerthiraja SJ  wrote:
>
>> Sorry pasted the wrong query.
>>
>>
>> #realhostip changes, before changing table and adding default value
>> UPDATE `cloud`.`configuration` SET value=CONCAT("*.",value) WHERE
>> `name`="consoleproxy.url.domain" OR `name`="consoleproxy.url.domain";
>> UPDATE `cloud`.`configuration` SET value=CONCAT("*.",value) WHERE
>> `name`="secstorage.ssl.cert.domain" OR `name`="secstorage.ssl.cert.domain";
>>
>> On Sun, Sep 13, 2015 at 9:41 PM, Keerthiraja SJ 
>> wrote:
>>
>>> After I change the query on
>>> /usr/share/cloudstack-management/setup/db/schema-421to430.sql I deploy the
>>> database cleanly where I could able to proceed and started successfully.
>>>
>>> #realhostip changes, before changing table and adding default value
>>> UPDATE `cloud`.`configuration` SET value=CONCAT("*.",value) WHERE
>>> `name`="consoleproxy.url.domain" OR `name`="consoleproxy.url.domain";
>>> UPDATE `cloud`.`configuration` SET value=CONCAT("*.",value) WHERE
>>> `name`="consoleproxy.url.domain" OR `name`="secstorage.ssl.cert.domain";
>>>
>>>
>>> Will this bee patched soon on the upcoming version of 4.5.3 and also on
>>> 4.6
>>>
>>> Thanks,
>>> Keerthi
>>>
>>>
>>>
>>> On Sun, Sep 13, 2015 at 9:37 PM, Keerthiraja SJ 
>>> wrote:
>>>
 I really want to know after I comment this line and do the
 database-setup will this effect the cloudstack once I build.

 I really not know what really this query use to do.

 Because I want to build a production environment right now so if I
 comment this two query and do the database initialization
 the cloudstack started without any issue.

 But will this will impact the CS after I deploy.

 Thanks,
 Keerthi

 On Sun, Sep 13, 2015 at 9:02 PM, Keerthiraja SJ 
 wrote:

> Even After I commented this line on MariaDB-server-10.0.21-1.el6.x86_64.
>
>
> #realhostip changes, before changing table and adding default value
> #UPDATE `cloud`.`configuration` SET value = CONCAT("*.",(SELECT
> `temptable`.`value` FROM (SELECT * FROM `cloud`.`configuration` WHERE
> `name`="consoleproxy.url.domain") AS `temptable` WHERE
> `temptable`.`name`="consoleproxy.url.domain")) WHERE
> `name`="consoleproxy.url.domain";
> #UPDATE `cloud`.`configuration` SET `value` = CONCAT("*.",(SELECT
> `temptable`.`value` FROM (SELECT * FROM `cloud`.`configuration` WHERE
> `name`="secstorage.ssl.cert.domain") AS `temptable` WHERE
> `temptable`.`name`="secstorage.ssl.cert.domain")) WHERE
> `name`="secstorage.ssl.cert.domain";
>
>
> Still could see error during the start up with below error on
> catalina.out
>
> ERROR
> ERROR [c.c.u.d.ScriptRunner] (main:null) Error executing: alter table
> vm_template add size bigint unsigned
> ERROR [c.c.u.d.ScriptRunner] (main:null)
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate 
> column
> name 'size'
> ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) Unable to execute
> upgrade script: 
> /usr/share/cloudstack-management/setup/db/schema-40to410.sql
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate
> column name 'size'
> at
> com.cloud.utils.db.ScriptRunner.runScript(ScriptRunner.java:185)
>
>
> ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) Unable to upgrade the
> database
> com.cloud.utils.exception.CloudRuntimeException: Unable to execute
> upgrade script: 
> /usr/share/cloudstack-management/setup/db/schema-40to410.sql
>
> Thanks,
> Keerthi
>
>
>
> On Sat, Sep 12, 2015 at 1:13 PM, Remi Bergsma <
> rberg...@schubergphilis.com> wrote:
>
>> Hi,
>>
>> The root cause was recently fixed here:
>> https://github.com/apache/cloudstack/pull/757/files
>>
>> As a workaround you can either alter the query (as mentioned below) or
>> downgrade mariadb:
>>
>> systemctl stop mariadb
>> yum -y remove mariadb-libs
>> yum -y install mariadb-1:5.5.41-2.el7_0.x86_64
>> mariadb-server-1:5.5.41-2.el7_0.x86_64
>> systemctl start mariadb
>>
>> Regards, Remi
>>
>> Sent from my iPhone
>>
>> On 12 Sep 2015, at 00:59, giraffeg forestg > 

Re: CS 4.5.2 Fails to Start.

[Keerthiraja SJ]

I update the same on

https://issues.apache.org/jira/browse/CLOUDSTACK-8836

Thanks,
Keerthi

On Sun, Sep 13, 2015 at 9:44 PM, Keerthiraja SJ  wrote:

> Sorry pasted the wrong query.
>
>
> #realhostip changes, before changing table and adding default value
> UPDATE `cloud`.`configuration` SET value=CONCAT("*.",value) WHERE
> `name`="consoleproxy.url.domain" OR `name`="consoleproxy.url.domain";
> UPDATE `cloud`.`configuration` SET value=CONCAT("*.",value) WHERE
> `name`="secstorage.ssl.cert.domain" OR `name`="secstorage.ssl.cert.domain";
>
> On Sun, Sep 13, 2015 at 9:41 PM, Keerthiraja SJ 
> wrote:
>
>> After I change the query on
>> /usr/share/cloudstack-management/setup/db/schema-421to430.sql I deploy the
>> database cleanly where I could able to proceed and started successfully.
>>
>> #realhostip changes, before changing table and adding default value
>> UPDATE `cloud`.`configuration` SET value=CONCAT("*.",value) WHERE
>> `name`="consoleproxy.url.domain" OR `name`="consoleproxy.url.domain";
>> UPDATE `cloud`.`configuration` SET value=CONCAT("*.",value) WHERE
>> `name`="consoleproxy.url.domain" OR `name`="secstorage.ssl.cert.domain";
>>
>>
>> Will this bee patched soon on the upcoming version of 4.5.3 and also on
>> 4.6
>>
>> Thanks,
>> Keerthi
>>
>>
>>
>> On Sun, Sep 13, 2015 at 9:37 PM, Keerthiraja SJ 
>> wrote:
>>
>>> I really want to know after I comment this line and do the
>>> database-setup will this effect the cloudstack once I build.
>>>
>>> I really not know what really this query use to do.
>>>
>>> Because I want to build a production environment right now so if I
>>> comment this two query and do the database initialization
>>> the cloudstack started without any issue.
>>>
>>> But will this will impact the CS after I deploy.
>>>
>>> Thanks,
>>> Keerthi
>>>
>>> On Sun, Sep 13, 2015 at 9:02 PM, Keerthiraja SJ 
>>> wrote:
>>>
 Even After I commented this line on MariaDB-server-10.0.21-1.el6.x86_64.


 #realhostip changes, before changing table and adding default value
 #UPDATE `cloud`.`configuration` SET value = CONCAT("*.",(SELECT
 `temptable`.`value` FROM (SELECT * FROM `cloud`.`configuration` WHERE
 `name`="consoleproxy.url.domain") AS `temptable` WHERE
 `temptable`.`name`="consoleproxy.url.domain")) WHERE
 `name`="consoleproxy.url.domain";
 #UPDATE `cloud`.`configuration` SET `value` = CONCAT("*.",(SELECT
 `temptable`.`value` FROM (SELECT * FROM `cloud`.`configuration` WHERE
 `name`="secstorage.ssl.cert.domain") AS `temptable` WHERE
 `temptable`.`name`="secstorage.ssl.cert.domain")) WHERE
 `name`="secstorage.ssl.cert.domain";


 Still could see error during the start up with below error on
 catalina.out

 ERROR
 ERROR [c.c.u.d.ScriptRunner] (main:null) Error executing: alter table
 vm_template add size bigint unsigned
 ERROR [c.c.u.d.ScriptRunner] (main:null)
 com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate column
 name 'size'
 ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) Unable to execute
 upgrade script: 
 /usr/share/cloudstack-management/setup/db/schema-40to410.sql
 com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate
 column name 'size'
 at
 com.cloud.utils.db.ScriptRunner.runScript(ScriptRunner.java:185)


 ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) Unable to upgrade the
 database
 com.cloud.utils.exception.CloudRuntimeException: Unable to execute
 upgrade script: 
 /usr/share/cloudstack-management/setup/db/schema-40to410.sql

 Thanks,
 Keerthi



 On Sat, Sep 12, 2015 at 1:13 PM, Remi Bergsma <
 rberg...@schubergphilis.com> wrote:

> Hi,
>
> The root cause was recently fixed here:
> https://github.com/apache/cloudstack/pull/757/files
>
> As a workaround you can either alter the query (as mentioned below) or
> downgrade mariadb:
>
> systemctl stop mariadb
> yum -y remove mariadb-libs
> yum -y install mariadb-1:5.5.41-2.el7_0.x86_64
> mariadb-server-1:5.5.41-2.el7_0.x86_64
> systemctl start mariadb
>
> Regards, Remi
>
> Sent from my iPhone
>
> On 12 Sep 2015, at 00:59, giraffeg forestg  > wrote:
>
> Hi
>
>
> CLOUDSTACK-8212 might become your help.
>
> https://issues.apache.org/jira/browse/CLOUDSTACK-8212
>
>
> 1. This is resolved by commenting out:
> ---
> [root@acs ~]# vi
> /usr/share/cloudstack-management/setup/db/schema-421to430.sql
>
> :
>
> #realhostip changes, before changing table and adding default value
> # UPDATE `cloud`.`configuration` SET value = CONCAT("*.",(SELECT
> `temptable`.`value` FROM (SELECT * FROM `cloud`.`configuration` WHERE
> `name`="consoleproxy.u
> rl.domain") AS `temptable` WHERE
> `temptable`.`name`="consoleproxy.url.dom

Re: CS 4.5.2 Fails to Start.

[Keerthiraja SJ]

Sorry pasted the wrong query.


#realhostip changes, before changing table and adding default value
UPDATE `cloud`.`configuration` SET value=CONCAT("*.",value) WHERE
`name`="consoleproxy.url.domain" OR `name`="consoleproxy.url.domain";
UPDATE `cloud`.`configuration` SET value=CONCAT("*.",value) WHERE
`name`="secstorage.ssl.cert.domain" OR `name`="secstorage.ssl.cert.domain";

On Sun, Sep 13, 2015 at 9:41 PM, Keerthiraja SJ  wrote:

> After I change the query on
> /usr/share/cloudstack-management/setup/db/schema-421to430.sql I deploy the
> database cleanly where I could able to proceed and started successfully.
>
> #realhostip changes, before changing table and adding default value
> UPDATE `cloud`.`configuration` SET value=CONCAT("*.",value) WHERE
> `name`="consoleproxy.url.domain" OR `name`="consoleproxy.url.domain";
> UPDATE `cloud`.`configuration` SET value=CONCAT("*.",value) WHERE
> `name`="consoleproxy.url.domain" OR `name`="secstorage.ssl.cert.domain";
>
>
> Will this bee patched soon on the upcoming version of 4.5.3 and also on 4.6
>
> Thanks,
> Keerthi
>
>
>
> On Sun, Sep 13, 2015 at 9:37 PM, Keerthiraja SJ 
> wrote:
>
>> I really want to know after I comment this line and do the database-setup
>> will this effect the cloudstack once I build.
>>
>> I really not know what really this query use to do.
>>
>> Because I want to build a production environment right now so if I
>> comment this two query and do the database initialization
>> the cloudstack started without any issue.
>>
>> But will this will impact the CS after I deploy.
>>
>> Thanks,
>> Keerthi
>>
>> On Sun, Sep 13, 2015 at 9:02 PM, Keerthiraja SJ 
>> wrote:
>>
>>> Even After I commented this line on MariaDB-server-10.0.21-1.el6.x86_64.
>>>
>>>
>>> #realhostip changes, before changing table and adding default value
>>> #UPDATE `cloud`.`configuration` SET value = CONCAT("*.",(SELECT
>>> `temptable`.`value` FROM (SELECT * FROM `cloud`.`configuration` WHERE
>>> `name`="consoleproxy.url.domain") AS `temptable` WHERE
>>> `temptable`.`name`="consoleproxy.url.domain")) WHERE
>>> `name`="consoleproxy.url.domain";
>>> #UPDATE `cloud`.`configuration` SET `value` = CONCAT("*.",(SELECT
>>> `temptable`.`value` FROM (SELECT * FROM `cloud`.`configuration` WHERE
>>> `name`="secstorage.ssl.cert.domain") AS `temptable` WHERE
>>> `temptable`.`name`="secstorage.ssl.cert.domain")) WHERE
>>> `name`="secstorage.ssl.cert.domain";
>>>
>>>
>>> Still could see error during the start up with below error on
>>> catalina.out
>>>
>>> ERROR
>>> ERROR [c.c.u.d.ScriptRunner] (main:null) Error executing: alter table
>>> vm_template add size bigint unsigned
>>> ERROR [c.c.u.d.ScriptRunner] (main:null)
>>> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate column
>>> name 'size'
>>> ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) Unable to execute
>>> upgrade script: /usr/share/cloudstack-management/setup/db/schema-40to410.sql
>>> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate
>>> column name 'size'
>>> at
>>> com.cloud.utils.db.ScriptRunner.runScript(ScriptRunner.java:185)
>>>
>>>
>>> ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) Unable to upgrade the
>>> database
>>> com.cloud.utils.exception.CloudRuntimeException: Unable to execute
>>> upgrade script: /usr/share/cloudstack-management/setup/db/schema-40to410.sql
>>>
>>> Thanks,
>>> Keerthi
>>>
>>>
>>>
>>> On Sat, Sep 12, 2015 at 1:13 PM, Remi Bergsma <
>>> rberg...@schubergphilis.com> wrote:
>>>
 Hi,

 The root cause was recently fixed here:
 https://github.com/apache/cloudstack/pull/757/files

 As a workaround you can either alter the query (as mentioned below) or
 downgrade mariadb:

 systemctl stop mariadb
 yum -y remove mariadb-libs
 yum -y install mariadb-1:5.5.41-2.el7_0.x86_64
 mariadb-server-1:5.5.41-2.el7_0.x86_64
 systemctl start mariadb

 Regards, Remi

 Sent from my iPhone

 On 12 Sep 2015, at 00:59, giraffeg forestg >>> > wrote:

 Hi


 CLOUDSTACK-8212 might become your help.

 https://issues.apache.org/jira/browse/CLOUDSTACK-8212


 1. This is resolved by commenting out:
 ---
 [root@acs ~]# vi
 /usr/share/cloudstack-management/setup/db/schema-421to430.sql

 :

 #realhostip changes, before changing table and adding default value
 # UPDATE `cloud`.`configuration` SET value = CONCAT("*.",(SELECT
 `temptable`.`value` FROM (SELECT * FROM `cloud`.`configuration` WHERE
 `name`="consoleproxy.u
 rl.domain") AS `temptable` WHERE
 `temptable`.`name`="consoleproxy.url.domain")) WHERE
 `name`="consoleproxy.url.domain";
 # UPDATE `cloud`.`configuration` SET `value` = CONCAT("*.",(SELECT
 `temptable`.`value` FROM (SELECT * FROM `cloud`.`configuration` WHERE
 `name`="secstorage.ssl.cert.domain") AS `temptable` WHERE
 `temptable`.`name`="secstorage.ssl.cert.domai

Re: CS 4.5.2 Fails to Start.

[Keerthiraja SJ]

After I change the query on
/usr/share/cloudstack-management/setup/db/schema-421to430.sql I deploy the
database cleanly where I could able to proceed and started successfully.

#realhostip changes, before changing table and adding default value
UPDATE `cloud`.`configuration` SET value=CONCAT("*.",value) WHERE
`name`="consoleproxy.url.domain" OR `name`="consoleproxy.url.domain";
UPDATE `cloud`.`configuration` SET value=CONCAT("*.",value) WHERE
`name`="consoleproxy.url.domain" OR `name`="secstorage.ssl.cert.domain";


Will this bee patched soon on the upcoming version of 4.5.3 and also on 4.6

Thanks,
Keerthi



On Sun, Sep 13, 2015 at 9:37 PM, Keerthiraja SJ  wrote:

> I really want to know after I comment this line and do the database-setup
> will this effect the cloudstack once I build.
>
> I really not know what really this query use to do.
>
> Because I want to build a production environment right now so if I comment
> this two query and do the database initialization
> the cloudstack started without any issue.
>
> But will this will impact the CS after I deploy.
>
> Thanks,
> Keerthi
>
> On Sun, Sep 13, 2015 at 9:02 PM, Keerthiraja SJ 
> wrote:
>
>> Even After I commented this line on MariaDB-server-10.0.21-1.el6.x86_64.
>>
>>
>> #realhostip changes, before changing table and adding default value
>> #UPDATE `cloud`.`configuration` SET value = CONCAT("*.",(SELECT
>> `temptable`.`value` FROM (SELECT * FROM `cloud`.`configuration` WHERE
>> `name`="consoleproxy.url.domain") AS `temptable` WHERE
>> `temptable`.`name`="consoleproxy.url.domain")) WHERE
>> `name`="consoleproxy.url.domain";
>> #UPDATE `cloud`.`configuration` SET `value` = CONCAT("*.",(SELECT
>> `temptable`.`value` FROM (SELECT * FROM `cloud`.`configuration` WHERE
>> `name`="secstorage.ssl.cert.domain") AS `temptable` WHERE
>> `temptable`.`name`="secstorage.ssl.cert.domain")) WHERE
>> `name`="secstorage.ssl.cert.domain";
>>
>>
>> Still could see error during the start up with below error on
>> catalina.out
>>
>> ERROR
>> ERROR [c.c.u.d.ScriptRunner] (main:null) Error executing: alter table
>> vm_template add size bigint unsigned
>> ERROR [c.c.u.d.ScriptRunner] (main:null)
>> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate column
>> name 'size'
>> ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) Unable to execute
>> upgrade script: /usr/share/cloudstack-management/setup/db/schema-40to410.sql
>> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate
>> column name 'size'
>> at
>> com.cloud.utils.db.ScriptRunner.runScript(ScriptRunner.java:185)
>>
>>
>> ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) Unable to upgrade the
>> database
>> com.cloud.utils.exception.CloudRuntimeException: Unable to execute
>> upgrade script: /usr/share/cloudstack-management/setup/db/schema-40to410.sql
>>
>> Thanks,
>> Keerthi
>>
>>
>>
>> On Sat, Sep 12, 2015 at 1:13 PM, Remi Bergsma <
>> rberg...@schubergphilis.com> wrote:
>>
>>> Hi,
>>>
>>> The root cause was recently fixed here:
>>> https://github.com/apache/cloudstack/pull/757/files
>>>
>>> As a workaround you can either alter the query (as mentioned below) or
>>> downgrade mariadb:
>>>
>>> systemctl stop mariadb
>>> yum -y remove mariadb-libs
>>> yum -y install mariadb-1:5.5.41-2.el7_0.x86_64
>>> mariadb-server-1:5.5.41-2.el7_0.x86_64
>>> systemctl start mariadb
>>>
>>> Regards, Remi
>>>
>>> Sent from my iPhone
>>>
>>> On 12 Sep 2015, at 00:59, giraffeg forestg >> > wrote:
>>>
>>> Hi
>>>
>>>
>>> CLOUDSTACK-8212 might become your help.
>>>
>>> https://issues.apache.org/jira/browse/CLOUDSTACK-8212
>>>
>>>
>>> 1. This is resolved by commenting out:
>>> ---
>>> [root@acs ~]# vi
>>> /usr/share/cloudstack-management/setup/db/schema-421to430.sql
>>>
>>> :
>>>
>>> #realhostip changes, before changing table and adding default value
>>> # UPDATE `cloud`.`configuration` SET value = CONCAT("*.",(SELECT
>>> `temptable`.`value` FROM (SELECT * FROM `cloud`.`configuration` WHERE
>>> `name`="consoleproxy.u
>>> rl.domain") AS `temptable` WHERE
>>> `temptable`.`name`="consoleproxy.url.domain")) WHERE
>>> `name`="consoleproxy.url.domain";
>>> # UPDATE `cloud`.`configuration` SET `value` = CONCAT("*.",(SELECT
>>> `temptable`.`value` FROM (SELECT * FROM `cloud`.`configuration` WHERE
>>> `name`="secstorage.ssl.cert.domain") AS `temptable` WHERE
>>> `temptable`.`name`="secstorage.ssl.cert.domain")) WHERE
>>> `name`="secstorage.ssl.cert.domain";
>>> ---
>>>
>>> and,
>>>
>>> 2. stop cloudsack-management & cloudsack-usage & mariadb
>>> 3. initialize database
>>> 4. cloudstack-setup-databases
>>> 5. cloudstack-setup-management
>>>
>>>
>>> I also the same problem has occurred.
>>>
>>> My environment:
>>> CloudStack 4.5.2 (
>>> http://packages.shapeblue.com/cloudstack/upstream/centos7/4.5)
>>> CentOS 7.1.1503
>>> mariadb-5.5.44-1
>>>
>>>
>>>
>>> ---
>>> Satoru Nakaya
>>>
>>> Japan CloudStack User Group
>>> http://cloudstack.jp/
>>>
>>>
>>> 2015-09-11 23:11 GMT+09:00 Ke

Re: CS 4.5.2 Fails to Start.

[Keerthiraja SJ]

I really want to know after I comment this line and do the database-setup
will this effect the cloudstack once I build.

I really not know what really this query use to do.

Because I want to build a production environment right now so if I comment
this two query and do the database initialization
the cloudstack started without any issue.

But will this will impact the CS after I deploy.

Thanks,
Keerthi

On Sun, Sep 13, 2015 at 9:02 PM, Keerthiraja SJ  wrote:

> Even After I commented this line on MariaDB-server-10.0.21-1.el6.x86_64.
>
>
> #realhostip changes, before changing table and adding default value
> #UPDATE `cloud`.`configuration` SET value = CONCAT("*.",(SELECT
> `temptable`.`value` FROM (SELECT * FROM `cloud`.`configuration` WHERE
> `name`="consoleproxy.url.domain") AS `temptable` WHERE
> `temptable`.`name`="consoleproxy.url.domain")) WHERE
> `name`="consoleproxy.url.domain";
> #UPDATE `cloud`.`configuration` SET `value` = CONCAT("*.",(SELECT
> `temptable`.`value` FROM (SELECT * FROM `cloud`.`configuration` WHERE
> `name`="secstorage.ssl.cert.domain") AS `temptable` WHERE
> `temptable`.`name`="secstorage.ssl.cert.domain")) WHERE
> `name`="secstorage.ssl.cert.domain";
>
>
> Still could see error during the start up with below error on catalina.out
>
> ERROR
> ERROR [c.c.u.d.ScriptRunner] (main:null) Error executing: alter table
> vm_template add size bigint unsigned
> ERROR [c.c.u.d.ScriptRunner] (main:null)
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate column
> name 'size'
> ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) Unable to execute upgrade
> script: /usr/share/cloudstack-management/setup/db/schema-40to410.sql
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate
> column name 'size'
> at com.cloud.utils.db.ScriptRunner.runScript(ScriptRunner.java:185)
>
>
> ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) Unable to upgrade the
> database
> com.cloud.utils.exception.CloudRuntimeException: Unable to execute upgrade
> script: /usr/share/cloudstack-management/setup/db/schema-40to410.sql
>
> Thanks,
> Keerthi
>
>
>
> On Sat, Sep 12, 2015 at 1:13 PM, Remi Bergsma  > wrote:
>
>> Hi,
>>
>> The root cause was recently fixed here:
>> https://github.com/apache/cloudstack/pull/757/files
>>
>> As a workaround you can either alter the query (as mentioned below) or
>> downgrade mariadb:
>>
>> systemctl stop mariadb
>> yum -y remove mariadb-libs
>> yum -y install mariadb-1:5.5.41-2.el7_0.x86_64
>> mariadb-server-1:5.5.41-2.el7_0.x86_64
>> systemctl start mariadb
>>
>> Regards, Remi
>>
>> Sent from my iPhone
>>
>> On 12 Sep 2015, at 00:59, giraffeg forestg > > wrote:
>>
>> Hi
>>
>>
>> CLOUDSTACK-8212 might become your help.
>>
>> https://issues.apache.org/jira/browse/CLOUDSTACK-8212
>>
>>
>> 1. This is resolved by commenting out:
>> ---
>> [root@acs ~]# vi
>> /usr/share/cloudstack-management/setup/db/schema-421to430.sql
>>
>> :
>>
>> #realhostip changes, before changing table and adding default value
>> # UPDATE `cloud`.`configuration` SET value = CONCAT("*.",(SELECT
>> `temptable`.`value` FROM (SELECT * FROM `cloud`.`configuration` WHERE
>> `name`="consoleproxy.u
>> rl.domain") AS `temptable` WHERE
>> `temptable`.`name`="consoleproxy.url.domain")) WHERE
>> `name`="consoleproxy.url.domain";
>> # UPDATE `cloud`.`configuration` SET `value` = CONCAT("*.",(SELECT
>> `temptable`.`value` FROM (SELECT * FROM `cloud`.`configuration` WHERE
>> `name`="secstorage.ssl.cert.domain") AS `temptable` WHERE
>> `temptable`.`name`="secstorage.ssl.cert.domain")) WHERE
>> `name`="secstorage.ssl.cert.domain";
>> ---
>>
>> and,
>>
>> 2. stop cloudsack-management & cloudsack-usage & mariadb
>> 3. initialize database
>> 4. cloudstack-setup-databases
>> 5. cloudstack-setup-management
>>
>>
>> I also the same problem has occurred.
>>
>> My environment:
>> CloudStack 4.5.2 (
>> http://packages.shapeblue.com/cloudstack/upstream/centos7/4.5)
>> CentOS 7.1.1503
>> mariadb-5.5.44-1
>>
>>
>>
>> ---
>> Satoru Nakaya
>>
>> Japan CloudStack User Group
>> http://cloudstack.jp/
>>
>>
>> 2015-09-11 23:11 GMT+09:00 Keerthiraja SJ > sjkeer...@gmail.com>>:
>>
>> I really face this issue when I install on MariaDB. If I do the same on
>> MySQL this pass it.
>>
>> On Fri, Sep 11, 2015 at 7:09 PM, Keerthiraja SJ > >
>> wrote:
>>
>> Hi All,
>>
>> Today I tried to install the CS 4.5.2 on fresh box where I could able to
>> successfully deploy
>> the cloudstack-setup-databases.
>>
>> After I tried number of times the fresh installation I could see the same
>> issue been reproducing it.
>>
>> While stating the cloudstack servers I could see below error constantly.
>>
>>
>>
>>
>> *ERROR*ERROR [c.c.u.d.ScriptRunner] (main:null) Error executing: UPDATE
>> `cloud`.`configuration` SET value = CONCAT("*.",(SELECT
>> `temptable`.`value`
>> FROM (SELECT * FROM `cloud`.`configuration` WHERE
>> `name`="consoleproxy.url.domain") AS `temptable` WHERE
>> `

Re: CS 4.5.2 Fails to Start.

[Remi Bergsma]

Hi Keerthi,

Did you clear the MySQL cloud db in between attempts?

I’m a bit confused. Are you using CentOS 6 or 7? First I thought CentOS 7, but 
now you show an CentOS 6 MariaDB package?
If you’re on CentOS 6 I’d recommend using the MySQL version that ships with it. 
Otherwise, the fix as described works, as long as you start clean.

Regards, Remi





On 13/09/15 17:32, "Keerthiraja SJ"  wrote:

>Even After I commented this line on MariaDB-server-10.0.21-1.el6.x86_64.
>
>
>#realhostip changes, before changing table and adding default value
>#UPDATE `cloud`.`configuration` SET value = CONCAT("*.",(SELECT
>`temptable`.`value` FROM (SELECT * FROM `cloud`.`configuration` WHERE
>`name`="consoleproxy.url.domain") AS `temptable` WHERE
>`temptable`.`name`="consoleproxy.url.domain")) WHERE
>`name`="consoleproxy.url.domain";
>#UPDATE `cloud`.`configuration` SET `value` = CONCAT("*.",(SELECT
>`temptable`.`value` FROM (SELECT * FROM `cloud`.`configuration` WHERE
>`name`="secstorage.ssl.cert.domain") AS `temptable` WHERE
>`temptable`.`name`="secstorage.ssl.cert.domain")) WHERE
>`name`="secstorage.ssl.cert.domain";
>
>
>Still could see error during the start up with below error on catalina.out
>
>ERROR
>ERROR [c.c.u.d.ScriptRunner] (main:null) Error executing: alter table
>vm_template add size bigint unsigned
>ERROR [c.c.u.d.ScriptRunner] (main:null)
>com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate column
>name 'size'
>ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) Unable to execute upgrade
>script: /usr/share/cloudstack-management/setup/db/schema-40to410.sql
>com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate column
>name 'size'
>at com.cloud.utils.db.ScriptRunner.runScript(ScriptRunner.java:185)
>
>
>ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) Unable to upgrade the
>database
>com.cloud.utils.exception.CloudRuntimeException: Unable to execute upgrade
>script: /usr/share/cloudstack-management/setup/db/schema-40to410.sql
>
>Thanks,
>Keerthi
>
>
>
>On Sat, Sep 12, 2015 at 1:13 PM, Remi Bergsma 
>wrote:
>
>> Hi,
>>
>> The root cause was recently fixed here:
>> https://github.com/apache/cloudstack/pull/757/files
>>
>> As a workaround you can either alter the query (as mentioned below) or
>> downgrade mariadb:
>>
>> systemctl stop mariadb
>> yum -y remove mariadb-libs
>> yum -y install mariadb-1:5.5.41-2.el7_0.x86_64
>> mariadb-server-1:5.5.41-2.el7_0.x86_64
>> systemctl start mariadb
>>
>> Regards, Remi
>>
>> Sent from my iPhone
>>
>> On 12 Sep 2015, at 00:59, giraffeg forestg > > wrote:
>>
>> Hi
>>
>>
>> CLOUDSTACK-8212 might become your help.
>>
>> https://issues.apache.org/jira/browse/CLOUDSTACK-8212
>>
>>
>> 1. This is resolved by commenting out:
>> ---
>> [root@acs ~]# vi
>> /usr/share/cloudstack-management/setup/db/schema-421to430.sql
>>
>> :
>>
>> #realhostip changes, before changing table and adding default value
>> # UPDATE `cloud`.`configuration` SET value = CONCAT("*.",(SELECT
>> `temptable`.`value` FROM (SELECT * FROM `cloud`.`configuration` WHERE
>> `name`="consoleproxy.u
>> rl.domain") AS `temptable` WHERE
>> `temptable`.`name`="consoleproxy.url.domain")) WHERE
>> `name`="consoleproxy.url.domain";
>> # UPDATE `cloud`.`configuration` SET `value` = CONCAT("*.",(SELECT
>> `temptable`.`value` FROM (SELECT * FROM `cloud`.`configuration` WHERE
>> `name`="secstorage.ssl.cert.domain") AS `temptable` WHERE
>> `temptable`.`name`="secstorage.ssl.cert.domain")) WHERE
>> `name`="secstorage.ssl.cert.domain";
>> ---
>>
>> and,
>>
>> 2. stop cloudsack-management & cloudsack-usage & mariadb
>> 3. initialize database
>> 4. cloudstack-setup-databases
>> 5. cloudstack-setup-management
>>
>>
>> I also the same problem has occurred.
>>
>> My environment:
>> CloudStack 4.5.2 (
>> http://packages.shapeblue.com/cloudstack/upstream/centos7/4.5)
>> CentOS 7.1.1503
>> mariadb-5.5.44-1
>>
>>
>>
>> ---
>> Satoru Nakaya
>>
>> Japan CloudStack User Group
>> http://cloudstack.jp/
>>
>>
>> 2015-09-11 23:11 GMT+09:00 Keerthiraja SJ > sjkeer...@gmail.com>>:
>>
>> I really face this issue when I install on MariaDB. If I do the same on
>> MySQL this pass it.
>>
>> On Fri, Sep 11, 2015 at 7:09 PM, Keerthiraja SJ > >
>> wrote:
>>
>> Hi All,
>>
>> Today I tried to install the CS 4.5.2 on fresh box where I could able to
>> successfully deploy
>> the cloudstack-setup-databases.
>>
>> After I tried number of times the fresh installation I could see the same
>> issue been reproducing it.
>>
>> While stating the cloudstack servers I could see below error constantly.
>>
>>
>>
>>
>> *ERROR*ERROR [c.c.u.d.ScriptRunner] (main:null) Error executing: UPDATE
>> `cloud`.`configuration` SET value = CONCAT("*.",(SELECT
>> `temptable`.`value`
>> FROM (SELECT * FROM `cloud`.`configuration` WHERE
>> `name`="consoleproxy.url.domain") AS `temptable` WHERE
>> `temptable`.`name`="consoleproxy.url.domain")) WHERE
>> `name`="consoleproxy.url.domain"
>> ERRO

Re: CS 4.5.2 Fails to Start.

[Keerthiraja SJ]

Even After I commented this line on MariaDB-server-10.0.21-1.el6.x86_64.


#realhostip changes, before changing table and adding default value
#UPDATE `cloud`.`configuration` SET value = CONCAT("*.",(SELECT
`temptable`.`value` FROM (SELECT * FROM `cloud`.`configuration` WHERE
`name`="consoleproxy.url.domain") AS `temptable` WHERE
`temptable`.`name`="consoleproxy.url.domain")) WHERE
`name`="consoleproxy.url.domain";
#UPDATE `cloud`.`configuration` SET `value` = CONCAT("*.",(SELECT
`temptable`.`value` FROM (SELECT * FROM `cloud`.`configuration` WHERE
`name`="secstorage.ssl.cert.domain") AS `temptable` WHERE
`temptable`.`name`="secstorage.ssl.cert.domain")) WHERE
`name`="secstorage.ssl.cert.domain";


Still could see error during the start up with below error on catalina.out

ERROR
ERROR [c.c.u.d.ScriptRunner] (main:null) Error executing: alter table
vm_template add size bigint unsigned
ERROR [c.c.u.d.ScriptRunner] (main:null)
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate column
name 'size'
ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) Unable to execute upgrade
script: /usr/share/cloudstack-management/setup/db/schema-40to410.sql
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate column
name 'size'
at com.cloud.utils.db.ScriptRunner.runScript(ScriptRunner.java:185)


ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) Unable to upgrade the
database
com.cloud.utils.exception.CloudRuntimeException: Unable to execute upgrade
script: /usr/share/cloudstack-management/setup/db/schema-40to410.sql

Thanks,
Keerthi



On Sat, Sep 12, 2015 at 1:13 PM, Remi Bergsma 
wrote:

> Hi,
>
> The root cause was recently fixed here:
> https://github.com/apache/cloudstack/pull/757/files
>
> As a workaround you can either alter the query (as mentioned below) or
> downgrade mariadb:
>
> systemctl stop mariadb
> yum -y remove mariadb-libs
> yum -y install mariadb-1:5.5.41-2.el7_0.x86_64
> mariadb-server-1:5.5.41-2.el7_0.x86_64
> systemctl start mariadb
>
> Regards, Remi
>
> Sent from my iPhone
>
> On 12 Sep 2015, at 00:59, giraffeg forestg  > wrote:
>
> Hi
>
>
> CLOUDSTACK-8212 might become your help.
>
> https://issues.apache.org/jira/browse/CLOUDSTACK-8212
>
>
> 1. This is resolved by commenting out:
> ---
> [root@acs ~]# vi
> /usr/share/cloudstack-management/setup/db/schema-421to430.sql
>
> :
>
> #realhostip changes, before changing table and adding default value
> # UPDATE `cloud`.`configuration` SET value = CONCAT("*.",(SELECT
> `temptable`.`value` FROM (SELECT * FROM `cloud`.`configuration` WHERE
> `name`="consoleproxy.u
> rl.domain") AS `temptable` WHERE
> `temptable`.`name`="consoleproxy.url.domain")) WHERE
> `name`="consoleproxy.url.domain";
> # UPDATE `cloud`.`configuration` SET `value` = CONCAT("*.",(SELECT
> `temptable`.`value` FROM (SELECT * FROM `cloud`.`configuration` WHERE
> `name`="secstorage.ssl.cert.domain") AS `temptable` WHERE
> `temptable`.`name`="secstorage.ssl.cert.domain")) WHERE
> `name`="secstorage.ssl.cert.domain";
> ---
>
> and,
>
> 2. stop cloudsack-management & cloudsack-usage & mariadb
> 3. initialize database
> 4. cloudstack-setup-databases
> 5. cloudstack-setup-management
>
>
> I also the same problem has occurred.
>
> My environment:
> CloudStack 4.5.2 (
> http://packages.shapeblue.com/cloudstack/upstream/centos7/4.5)
> CentOS 7.1.1503
> mariadb-5.5.44-1
>
>
>
> ---
> Satoru Nakaya
>
> Japan CloudStack User Group
> http://cloudstack.jp/
>
>
> 2015-09-11 23:11 GMT+09:00 Keerthiraja SJ  sjkeer...@gmail.com>>:
>
> I really face this issue when I install on MariaDB. If I do the same on
> MySQL this pass it.
>
> On Fri, Sep 11, 2015 at 7:09 PM, Keerthiraja SJ  >
> wrote:
>
> Hi All,
>
> Today I tried to install the CS 4.5.2 on fresh box where I could able to
> successfully deploy
> the cloudstack-setup-databases.
>
> After I tried number of times the fresh installation I could see the same
> issue been reproducing it.
>
> While stating the cloudstack servers I could see below error constantly.
>
>
>
>
> *ERROR*ERROR [c.c.u.d.ScriptRunner] (main:null) Error executing: UPDATE
> `cloud`.`configuration` SET value = CONCAT("*.",(SELECT
> `temptable`.`value`
> FROM (SELECT * FROM `cloud`.`configuration` WHERE
> `name`="consoleproxy.url.domain") AS `temptable` WHERE
> `temptable`.`name`="consoleproxy.url.domain")) WHERE
> `name`="consoleproxy.url.domain"
> ERROR [c.c.u.d.ScriptRunner] (main:null)
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
> link failure
>
> The last packet successfully received from the server was 25 milliseconds
> ago.  The last packet sent successfully to the server was 25 milliseconds
> ago.
> ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) Unable to execute
> upgrade
> script: /usr/share/cloudstack-management/setup/db/schema-421to430.sql
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
> link failure
>
>
> Caused by: java.net.S

RE: CS 4.5.2: all hosts reboot after 3 days at production

[Vadim Kimlaychuk]

Remi,

   Thank you for hint. At least one problem is identified:

[root@xcp1 ~]# xe pool-list params=all | grep -E "ha-enabled|ha-config"
ha-enabled ( RO): false
ha-configuration ( RO):

   Where should I look for storage errors? Host? Management server? 
I have checked /var/log/messages and there were only regular messages, 
no "fence" or "reboot" commands.


   I have dedicated NFS server that should be accessible all the 
time (at least NIC interfaces are bonded in master-slave mode). Server 
is used for both primary and secondary storage.


Thanks,

Vadim.



On 2015-09-13 14:38, Remi Bergsma wrote:


Hi Vadim,

Not sure what the problem is. Although I do know that when shared 
storage is used, both CloudStack and XenServer will fence (reboot) the 
box to prevent corruption in case access to the network or the storage 
is not possible. What storage do you use?


What does this return on a XenServer?:
xe pool-list params=all | grep -E "ha-enabled|ha-config"

HA should be on, or else a hypervisor crash will not recover properly.

If you search the logs for Fence or reboot, does anything come back?

The logs you mention are nothing to worry about.

Can you tell us in some more details what happens and how we can 
reproduce it?


Regards,
Remi

-Original Message-
From: Vadim Kimlaychuk [mailto:va...@kickcloud.net]
Sent: zondag 13 september 2015 9:32
To: users@cloudstack.apache.org
Cc: Remi Bergsma
Subject: Re: CS 4.5.2: all hosts reboot after 3 days at production

Hello Remi,

This issue has nothing to do with CS 4.5.2. We got host reboot after 
precisely 1 week with previous version of CS (4.5.1). Previous version 
has been working without restart for 106 days before. So it is not a 
software issue.


What does really make me unhappy -- accidental host reboot made entire 
cluster unusable. Cloudstack management server was up and running, 
second cluster node was up and running all the time and VM were 
transferred to the second host, but System VMs were not rebooted 
properly by CS and half of the network was down. SSVM and CPVM were in 
"disconnected" status. Client VMs were up, but couldn't connect to 
storage, because VRs were offline. Entire mess.


I have used planned maintenance mode before and cluster worked just 
perfect. We didn't have any single second downtime. But with accidental 
reboot there is no use of clusterization. :(


Vadim.

On 2015-09-08 09:35, Vadim Kimlaychuk wrote:


Hello Remi,

First of all I don't have /var/log/xha.log file. I have examined logs
in detail and haven't found any trace that heartbeat has failed. The
only serious problem I have found in management logs before restart is
repeating many times error:

-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠
-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠
2015-09-06 00:47:21,591 DEBUG [c.c.a.m.AgentManagerImpl]
(RouterMonitor-1:ctx-2d67d422) Details from executing class
com.cloud.agent.api.NetworkUsageCommand: Exception:
java.lang.Exception
Message: vpc network usage plugin call failed
Stack: java.lang.Exception: vpc network usage plugin call failed at
com.cloud.hypervisor.xenserver.resource.XenServer56Resource.VPCNetwork
Usage(XenServer56Resource.java:172)
at
com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(Xe
nServer56Resource.java:195)
at
com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeReq
uest(XenServer56Resource.java:62)
at
com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRe
quest(XenServer610Resource.java:87)
at
com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.execut
eRequest(XenServer620SP1Resource.java:65)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAge
ntAttache.java:302)
...

-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠
-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠

Just couple of seconds before XCP2 host restart:

-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠
-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠
2015-09-06 00:48:27,884 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-83:ctx-ff822baf) Ping from 2(xcp1)
2015-09-06 00:48:27,884 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgentCronJob-83:ctx-ff822baf) Process host VM state report from
ping process. host: 2
2015-09-06 00:48:27,904 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgentCronJob-83:ctx-ff822baf) Process VM state report. host: 2,
number of records in report: 6
2015-09-06 00:48:27,904 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgentCronJob-83:ctx-ff822baf) VM state report. host: 2, vm id:
85, power state: PowerOn
2015-09-06 00:48:27,907 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgentCronJob-83:ctx-ff822baf) VM power state does not change,
skip DB writing. vm id: 85
2015-09-06 00:48:27,907 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgentCronJob-83:ctx-ff822baf) VM state report. host: 2, vm id:
1, power state: PowerOn
2015-09-06 

RE: CS 4.5.2: all hosts reboot after 3 days at production

[Remi Bergsma]

Hi Vadim,

Not sure what the problem is. Although I do know that when shared storage is 
used, both CloudStack and XenServer will fence (reboot) the box to prevent 
corruption in case access to the network or the storage is not possible. What 
storage do you use?

What does this return on a XenServer?:
xe pool-list params=all | grep -E "ha-enabled|ha-config"

HA should be on, or else a hypervisor crash will not recover properly.

If you search the logs for Fence or reboot, does anything come back?

The logs you mention are nothing to worry about.

Can you tell us in some more details what happens and how we can reproduce it?

Regards,
Remi


-Original Message-
From: Vadim Kimlaychuk [mailto:va...@kickcloud.net] 
Sent: zondag 13 september 2015 9:32
To: users@cloudstack.apache.org
Cc: Remi Bergsma
Subject: Re: CS 4.5.2: all hosts reboot after 3 days at production

Hello Remi,

 This issue has nothing to do with CS 4.5.2. We got host reboot after 
precisely 1 week with previous version of CS (4.5.1). Previous version has been 
working without restart for 106 days before. So it is not a software issue.

 What does really make me unhappy -- accidental host reboot made entire 
cluster unusable. Cloudstack management server was up and running, second 
cluster node was up and running all the time and VM were transferred to the 
second host, but System VMs were not rebooted properly by CS and half of the 
network was down. SSVM and CPVM were in "disconnected" status. Client VMs were 
up, but couldn't connect to storage, because VRs were offline. Entire mess.

 I have used planned maintenance mode before and cluster worked just 
perfect. We didn't have any single second downtime. But with accidental reboot 
there is no use of clusterization. :(

Vadim.

On 2015-09-08 09:35, Vadim Kimlaychuk wrote:

> Hello Remi,
> 
> First of all I don't have /var/log/xha.log file. I have examined logs 
> in detail and haven't found any trace that heartbeat has failed. The 
> only serious problem I have found in management logs before restart is 
> repeating many times error:
> 
> -⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠
> -⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠
> 2015-09-06 00:47:21,591 DEBUG [c.c.a.m.AgentManagerImpl]
> (RouterMonitor-1:ctx-2d67d422) Details from executing class
> com.cloud.agent.api.NetworkUsageCommand: Exception: 
> java.lang.Exception
> Message: vpc network usage plugin call failed
> Stack: java.lang.Exception: vpc network usage plugin call failed at
> com.cloud.hypervisor.xenserver.resource.XenServer56Resource.VPCNetwork
> Usage(XenServer56Resource.java:172)
> at
> com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(Xe
> nServer56Resource.java:195)
> at
> com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeReq
> uest(XenServer56Resource.java:62)
> at
> com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRe
> quest(XenServer610Resource.java:87)
> at
> com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.execut
> eRequest(XenServer620SP1Resource.java:65)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAge
> ntAttache.java:302)
> ...
> 
> -⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠
> -⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠
> 
> Just couple of seconds before XCP2 host restart:
> 
> -⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠
> -⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠
> 2015-09-06 00:48:27,884 DEBUG [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-83:ctx-ff822baf) Ping from 2(xcp1)
> 2015-09-06 00:48:27,884 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
> (DirectAgentCronJob-83:ctx-ff822baf) Process host VM state report from 
> ping process. host: 2
> 2015-09-06 00:48:27,904 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
> (DirectAgentCronJob-83:ctx-ff822baf) Process VM state report. host: 2, 
> number of records in report: 6
> 2015-09-06 00:48:27,904 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
> (DirectAgentCronJob-83:ctx-ff822baf) VM state report. host: 2, vm id: 
> 85, power state: PowerOn
> 2015-09-06 00:48:27,907 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
> (DirectAgentCronJob-83:ctx-ff822baf) VM power state does not change, 
> skip DB writing. vm id: 85
> 2015-09-06 00:48:27,907 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
> (DirectAgentCronJob-83:ctx-ff822baf) VM state report. host: 2, vm id: 
> 1, power state: PowerOn
> 2015-09-06 00:48:27,910 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
> (DirectAgentCronJob-83:ctx-ff822baf) VM power state does not change, 
> skip DB writing. vm id: 1
> 2015-09-06 00:48:27,910 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
> (DirectAgentCronJob-83:ctx-ff822baf) VM state report. host: 2, vm id: 
> 2, power state: PowerOn
> 2015-09-06 00:48:27,913 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
> (DirectAgentCronJob-83:ctx-ff822baf) VM power state does not change, 
> skip DB writing. vm id: 2
> 2015-

Re: CS 4.5.2: all hosts reboot after 3 days at production

[Vadim Kimlaychuk]

Hello Remi,

This issue has nothing to do with CS 4.5.2. We got host reboot 
after precisely 1 week with previous version of CS (4.5.1). Previous 
version has been working without restart for 106 days before. So it is 
not a software issue.


What does really make me unhappy -- accidental host reboot made 
entire cluster unusable. Cloudstack management server was up and 
running, second cluster node was up and running all the time and VM were 
transferred to the second host, but System VMs were not rebooted 
properly by CS and half of the network was down. SSVM and CPVM were in 
"disconnected" status. Client VMs were up, but couldn't connect to 
storage, because VRs were offline. Entire mess.


I have used planned maintenance mode before and cluster worked 
just perfect. We didn't have any single second downtime. But with 
accidental reboot there is no use of clusterization. :(


Vadim.

On 2015-09-08 09:35, Vadim Kimlaychuk wrote:


Hello Remi,

First of all I don't have /var/log/xha.log file. I have examined logs 
in detail and haven't found any trace that heartbeat has failed. The 
only serious problem I have found in management logs before restart is 
repeating many times error:


-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠
2015-09-06 00:47:21,591 DEBUG [c.c.a.m.AgentManagerImpl] 
(RouterMonitor-1:ctx-2d67d422) Details from executing class 
com.cloud.agent.api.NetworkUsageCommand: Exception: java.lang.Exception

Message: vpc network usage plugin call failed
Stack: java.lang.Exception: vpc network usage plugin call failed
at 
com.cloud.hypervisor.xenserver.resource.XenServer56Resource.VPCNetworkUsage(XenServer56Resource.java:172)
at 
com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:195)
at 
com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62)
at 
com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)
at 
com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
at 
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302)

...

-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠

Just couple of seconds before XCP2 host restart:

-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠
2015-09-06 00:48:27,884 DEBUG [c.c.a.m.DirectAgentAttache] 
(DirectAgentCronJob-83:ctx-ff822baf) Ping from 2(xcp1)
2015-09-06 00:48:27,884 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] 
(DirectAgentCronJob-83:ctx-ff822baf) Process host VM state report from 
ping process. host: 2
2015-09-06 00:48:27,904 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] 
(DirectAgentCronJob-83:ctx-ff822baf) Process VM state report. host: 2, 
number of records in report: 6
2015-09-06 00:48:27,904 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] 
(DirectAgentCronJob-83:ctx-ff822baf) VM state report. host: 2, vm id: 
85, power state: PowerOn
2015-09-06 00:48:27,907 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] 
(DirectAgentCronJob-83:ctx-ff822baf) VM power state does not change, 
skip DB writing. vm id: 85
2015-09-06 00:48:27,907 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] 
(DirectAgentCronJob-83:ctx-ff822baf) VM state report. host: 2, vm id: 
1, power state: PowerOn
2015-09-06 00:48:27,910 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] 
(DirectAgentCronJob-83:ctx-ff822baf) VM power state does not change, 
skip DB writing. vm id: 1
2015-09-06 00:48:27,910 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] 
(DirectAgentCronJob-83:ctx-ff822baf) VM state report. host: 2, vm id: 
2, power state: PowerOn
2015-09-06 00:48:27,913 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] 
(DirectAgentCronJob-83:ctx-ff822baf) VM power state does not change, 
skip DB writing. vm id: 2
2015-09-06 00:48:27,913 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] 
(DirectAgentCronJob-83:ctx-ff822baf) VM state report. host: 2, vm id: 
82, power state: PowerOn
2015-09-06 00:48:27,916 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] 
(DirectAgentCronJob-83:ctx-ff822baf) VM power state does not change, 
skip DB writing. vm id: 82
2015-09-06 00:48:27,916 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] 
(DirectAgentCronJob-83:ctx-ff822baf) VM state report. host: 2, vm id: 
94, power state: PowerOn
2015-09-06 00:48:27,919 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] 
(DirectAgentCronJob-83:ctx-ff822baf) VM power state does not change, 
skip DB writing. vm id: 94
2015-09-06 00:48:27,919 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] 
(DirectAgentCronJob-83:ctx-ff822baf) VM state report. host: 2, vm id: 
90, power state: PowerOn
2015-09-06 00:48:27,922 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] 
(DirectAgentCronJob-83:ctx-ff822baf) VM power state does not change, 
skip DB writing. vm id: 90