Re: Upgrading from 4.9.3 to 4.11

[ilya musayev]

You’d have to put a separate Advanced Zone to use L2 Networking. 

What would be delegating DHCP and managing DNS? 

If you can keep the DHCP+Metadata with cloudstack Router VM (it will only do 
DHCP and Metadata and no routing) - you can use everything else from upstream 
provider. This is how most Advanced Zones with VLAN isolation deployed and we 
have many of these.

You can probably manage to abstract DHCP as well but integrating with a 
provider/plugin.. GloboDNS guys wrote some sort of integration like it - 
sometime ago.

Regards,
ilya

> On Oct 31, 2018, at 10:03 AM, McClune, James  
> wrote:
> 
> Hello CloudStack Community,
> 
> I work for a school district in Ohio and we've been running ACS 4.9.3 for
> over a year now. Great piece of software! We're using Ceph (Luminous) for
> our storage backend and KVM/libvirt for virtualization.
> 
> We're starting to expand our private cloud. We'd like to rely on external
> systems for IP services (e.g. DHCP, DNS, routing, etc.). Right now, we're
> running a basic network zone (with ACS managing IP services). I'm building
> a roadmap for upgrading to ACS 4.11. I'm looking to implement the L2
> functionality in 4.11, as described here:
> 
> https://www.shapeblue.com/layer-2-networks-in-cloudstack/
> 
> I was wondering if anyone has advice on upgrading to 4.11 from 4.9.3 (i.e.
> important things to watch for, problems encountered, etc.). I'm referencing
> the documentation here:
> 
> http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.11.0.0/upgrade/upgrade-4.9.html
> 
> I appreciate all input! :)
> 
> Thanks,
> 
> --
> 
> James McClune
> 
> Technical Support Specialist
> 
> Norwalk City Schools
> 
> Phone: 419-660-6590
> 
> mcclu...@norwalktruckers.net

Re: ISOs not readying?

[ilya musayev]

I came across something similar and was troubleshooting the problem as
well. See if restart of management server and secondary store system vm
allows you to download it.

If it does - we have to troubleshoot further, I suspect we have a thread
leak. Please check on number of open file handles on management server as
well before restart.

On Fri, Oct 26, 2018 at 11:47 AM Andrija Panic 
wrote:

> Just tested URL above, with normal (not direct) download, and it works
> fine. (ACS 4.11.2 RC3)
>
> Did you check the health of the Secondary Storage Virtual Machine - I would
> suggest to do so, since this seems as possible issue with SSVM and it might
> means other issues for your deployment also.
>
>
>1.
>
>ssh -i  /root/.ssh/id.rsa_cloud  -p 3922 root@SSVM>
>2. *SSVM health check* - Run the following script inside ssvm:
>/usr/local/cloud/systemvm/ssvm-check.sh
>It checks for 1)connectivity with  DNS server 2) resolving of  domain
>names 3)status of secondary storage 4)ability to write to secondary
> storage
>5)connectivity with management server at port 8250 and 6) status of java
>process.
>
>
> Please try to add template:
> http://dl.openvm.eu/cloudstack/ubuntu/x86_64/ubuntu-16.04-kvm.qcow2.bz2 -
> if this works...
>
> Cheers,
> Andrija
>
> On Fri, 26 Oct 2018 at 19:41, Alexandre Bruyere <
> bruyere.alexan...@gmail.com>
> wrote:
>
> > Direct download that bypasses secondary storage seems to make it ready
> > instantly (with the correct filesize), so I'm going to assume the link is
> > good.
> >
> > On Fri, Oct 26, 2018 at 1:28 PM Alexandre Bruyere <
> > bruyere.alexan...@gmail.com> wrote:
> >
> > > No progress in UI, no status.
> > >
> > > How would I make sure there are no redirections? At a glance there
> > doesn't
> > > seem to be any.
> > >
> > > On Fri, Oct 26, 2018 at 12:33 PM Andrija Panic <
> andrija.pa...@gmail.com>
> > > wrote:
> > >
> > >> Make sure no http redirections, no ssl support (not sure if https URLs
> > are
> > >> supported now in 4.11 or not). Only plain http adn status code 200 :)
> > >>
> > >> Do you see percentage /download progress in UI ?
> > >>
> > >> On Fri, Oct 26, 2018, 18:20 Alexandre Bruyere <
> > >> bruyere.alexan...@gmail.com>
> > >> wrote:
> > >>
> > >> > Hello.
> > >> >
> > >> > I've set up a Ubuntu 16.04.5 Cloudstack setup using this tutorial:
> > >> >
> > >> > https://rohityadav.cloud/blog/cloudstack-kvm/
> > >> >
> > >> > The only place I've strayed is that I've configured four network
> > >> interfaces
> > >> > - enp3s0 set as manual config, enp3s0.10 set as manual config (I've
> > made
> > >> > sure my system was VLAN-enabled), cloudbr0 as a
> statically-configured
> > >> > bridge to enp3s0 and cloudbr1 as a statically-configured bridge to
> > >> > enp3s0.10 on VLAN 10.
> > >> >
> > >> > My problem is, I'm currently trying to import an iso (Ubuntu Server)
> > >> from
> > >> > URL (
> http://releases.ubuntu.com/16.04/ubuntu-16.04.5-server-amd64.iso
> > )
> > >> in
> > >> > two zones (one on cloudbr0, one on cloudbr1).
> > >> >
> > >> > Adding the ISO seems to work fine, but they never become "ready".
> Any
> > >> tips
> > >> > as to why?
> > >> >
> > >> > (I will be testing direct download shortly)
> > >> >
> > >>
> > >
> >
>
>
> --
>
> Andrija Panić
>

Re: Host HA vs transient NFS problems on KVM

[ilya musayev]

Would you please file the JIRA bugs describing in exact details

1) your setup
2) what was done or happened
3) expected result

I imagine this will be fixed in the next point release if issues are indeed
correct. We’ve yet to try this framework and if it does not work as
anticipated we will have lots of issues.



On Tue, Oct 23, 2018 at 8:30 AM Andrei Mikhailovsky
 wrote:

> Hi Jean,
>
> I have previously done some HA testing and have pretty much came to
> similar conclusions as you have. My testing showed that using HA is very
> unreliable at best and data loosing at worst cases. I have had the
> following outcome from various testing scenarios:
>
> 1. Works as expected (very rarely)
> 2. Starts 2 vms on different hosts (data loss / corruption)
> 3. Reboots ALL KVM hosts (even those hosts that do not have a single vm
> with nfs volumes)
>
> Now, I can not justify having HA with even a slim chances of having 2 or 3
> above. Honestly, I do not know a single business that is happy to accept
> those scenarios. Frankly speaking, for me the cloudstack HA options create
> more problems than solve and thus I've not enabled them. I have decided
> that ACS with KVM is not HA friendly, full stop. Having said this, I've not
> tested the latest couple of releases, so I will give it a benefit of the
> doubt and wait for user's reports to prove my conclusion otherwise. I've
> wasted enough of my own time on KVM HA.
>
> My HA approach to ACS is more of a manual nature, which is far more
> reliable and is less prone to issues in my experience. I have a monitoring
> system sending me alerts when VMs, host servers and storage become
> unreachable. It is not as convenient as a fully working automatic HA, I
> agree, but it is far better to be woken up at 3am to deal with restarting a
> handful of vms and perhaps a KVM host force reboot than dealing with mass
> KVM hosts reboots and/or trying to find duplicate vms lurking somewhere on
> the host servers. Been there, done that - NO THANKS!
>
> Cheers
>
> Andrei
>
> - Original Message -
> > From: "Jean-Francois Nadeau" 
> > To: "users" 
> > Sent: Monday, 22 October, 2018 22:13:35
> > Subject: Host HA vs transient NFS problems on KVM
>
> > Dear community,
> >
> > I want to share my concern upgrading from 4.9 to 4.11 in regards to how
> the
> > host HA framework works and the handling of various failure conditions.
> >
> > Since we have been running CS on 4.9.3 with NFS on KVM,  VM HA have been
> > working as expected when hypervisor crashed and I agree we might have
> > been lucky knowing the limitations of the KVM investigator and the
> > possibility to fire the same VM on 2 KVM hosts is real when you know the
> > recipe for it.
> >
> > Still, on 4.9.3 we were tolerant to transient primary NFS storage access
> > issues, typical of a network problem (and we've seen it lately for a 22
> > minutes disconnection).  Although these events are quite rare,  when they
> > do happen their blast radius can be a huge impact on the business.
> >
> > So when we initially tested CS on 4.9.3 we purposely blocked access to
> NFS
> > and we observe the results.   Changing the kvmhearbeat.sh script so it
> > doesn't reboot the node after 5 minutes has been essential to defuse the
> > potential of a massive KVM hosts reboot.In the end,  it's far less
> > damage to let NFS recover than having all those VMs rebooted.   On 4.9.3
> > the cloudtack-agent will remain "Up"  and not fire any VM twice if the
> NFS
> > storage becomes available again within 30 minutes.
> >
> > Now, testing the upgrade from 4.9 to 4.11 in our lab and the same
> failure
> > conditions we rapidly saw a different behavior although not perfectly
> > consistent.  On 4.11.2 without host HA enabled,  we will see the agent
> > "try" to disconnect after 5 minutes tho sometimes the KVM host goes into
> > Disconnect state and sometimes it goes straight to Down state.  In that
> > case we'll see a duplicate VM created in no time and once the NFS issue
> is
> > resolved,  we have 2 copies of that VM and cloudstack only knowns about
> > that last copy.   This is obviously a disaster forcing us to look at how
> > host HA can help.
> >
> > Now with host HA enabled and simulating the same NFS hiccup,  we won't
> get
> > duplicate VMs but we will get a KVM host reset.  The problem here is
> that,
> > yes the host HA does ensure we don't have dup VMs but at scale this would
> > also provoke a lot of KVM host resets (if not all of them).   If we are
> at
> > risk with host HA to have massive KVM host resets,  then I might prefer
> to
> > disable host/VM HA entirely and just handle KVM host failures manually.
> > This is supper annoying for the ops team,  but far less risky for the
> > business.
> >
> > Im trying to find if there's a middle ground here between the 4.9
> behavior
> > with NFS hiccups and the reliability of the new host HA framework.
> >
> > best,
> >
> > Jean-Francois
>

Re: Buggy dashboard statistics

[ilya musayev]

I dont see these issues in my env and both views are ok.

We’ve worked with ShapeBlue on testing ACS rather extensively from from 4.5 -> 
4.11.. no issues of this type have been observed.

I can only assume you tried chrome “incognito” mode and its not caching issue.. 

You should be able to see where the errors are if you view the console.. 

> On Aug 22, 2018, at 9:59 PM, Ivan Kudryavtsev  
> wrote:
> 
> That interface doesn't work for me completely))
> https://box.bw-sw.com/f/959fb9b92a74424eaf0b/
> 
> capacity.check.period is 30 (300 seconds)
> 
> Actually, I don't care about those metrics at all because use Zabbix, but
> you know, enterprise users will not be happy with such a trashy look.
> 
> 2018-08-23 11:54 GMT+07:00 ilya musayev :
> 
>> Oh - i never pay attention to these in general - but thats rather
>> alarming.
>> 
>> Have you tried using “metrics” tab under infrastructure - does that look
>> red when you drill down from zone -> cluster -> host?
>> 
>> Lastly - what is you interval for checking capacity set to in global
>> settings?
>> 
>>> On Aug 22, 2018, at 9:47 PM, Ivan Kudryavtsev 
>> wrote:
>>> 
>>> Ok, let's go.
>>> Share it here: https://box.bw-sw.com/f/f264de747a5b4d38ac07/
>>> As you can see, all gauges are red, despite should be almost fine.
>>> 
>>> 2018-08-23 10:52 GMT+07:00 ilya musayev :
>>> 
>>>> Can you share screenshot somewhere please..
>>>> 
>>>> 
>>>> 
>>>>> On Aug 22, 2018, at 5:22 AM, Ivan Kudryavtsev <
>> kudryavtsev...@bw-sw.com>
>>>> wrote:
>>>>> 
>>>>> Hi, it's all red in my case. Buggy, indeed.
>>>>> 
>>>>> ср, 22 авг. 2018 г., 18:46 Benjamin Naber <
>> benjamin.na...@coders-area.de
>>>>> :
>>>>> 
>>>>>> Hi @all,
>>>>>> 
>>>>>> 
>>>>>> i successfully upgraded our environment from 4.10 to 4.11.1.
>>>>>> 
>>>>>> The Management gui looks at Dashboard statistics now abit buggy.
>> Someone
>>>>>> other have a similar problem with 4.11.1 ?
>>>>>> 
>>>>>> Browser cache clearing has no effect. Are there any Server Cache that
>>>> can
>>>>>> be cleared.
>>>>>> 
>>>>>> King regards
>>>>>> 
>>>>>> Ben
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> --
>>> With best regards, Ivan Kudryavtsev
>>> Bitworks LLC
>>> Cell: +7-923-414-1515
>>> WWW: http://bitworks.software/ <http://bw-sw.com/>
>> 
>> 
> 
> 
> -- 
> With best regards, Ivan Kudryavtsev
> Bitworks LLC
> Cell: +7-923-414-1515
> WWW: http://bitworks.software/ <http://bw-sw.com/>

Re: Buggy dashboard statistics

[ilya musayev]

Oh - i never pay attention to these in general - but thats rather alarming. 

Have you tried using “metrics” tab under infrastructure - does that look red 
when you drill down from zone -> cluster -> host?

Lastly - what is you interval for checking capacity set to in global settings?

> On Aug 22, 2018, at 9:47 PM, Ivan Kudryavtsev  
> wrote:
> 
> Ok, let's go.
> Share it here: https://box.bw-sw.com/f/f264de747a5b4d38ac07/
> As you can see, all gauges are red, despite should be almost fine.
> 
> 2018-08-23 10:52 GMT+07:00 ilya musayev :
> 
>> Can you share screenshot somewhere please..
>> 
>> 
>> 
>>> On Aug 22, 2018, at 5:22 AM, Ivan Kudryavtsev 
>> wrote:
>>> 
>>> Hi, it's all red in my case. Buggy, indeed.
>>> 
>>> ср, 22 авг. 2018 г., 18:46 Benjamin Naber >> :
>>> 
>>>> Hi @all,
>>>> 
>>>> 
>>>> i successfully upgraded our environment from 4.10 to 4.11.1.
>>>> 
>>>> The Management gui looks at Dashboard statistics now abit buggy. Someone
>>>> other have a similar problem with 4.11.1 ?
>>>> 
>>>> Browser cache clearing has no effect. Are there any Server Cache that
>> can
>>>> be cleared.
>>>> 
>>>> King regards
>>>> 
>>>> Ben
>>>> 
>>>> 
>> 
>> 
> 
> 
> -- 
> With best regards, Ivan Kudryavtsev
> Bitworks LLC
> Cell: +7-923-414-1515
> WWW: http://bitworks.software/ <http://bw-sw.com/>

Re: Buggy dashboard statistics

[ilya musayev]

Can you share screenshot somewhere please..



> On Aug 22, 2018, at 5:22 AM, Ivan Kudryavtsev  
> wrote:
> 
> Hi, it's all red in my case. Buggy, indeed.
> 
> ср, 22 авг. 2018 г., 18:46 Benjamin Naber :
> 
>> Hi @all,
>> 
>> 
>> i successfully upgraded our environment from 4.10 to 4.11.1.
>> 
>> The Management gui looks at Dashboard statistics now abit buggy. Someone
>> other have a similar problem with 4.11.1 ?
>> 
>> Browser cache clearing has no effect. Are there any Server Cache that can
>> be cleared.
>> 
>> King regards
>> 
>> Ben
>> 
>> 

Re: 4.9 to 4.11 upgrade broken

[ilya musayev]

yes - please try the proper 4.11 systemvm templates.

> On Aug 21, 2018, at 1:54 PM, Asai  wrote:
> 
> Can I manually download the systemvm template from here? 
> http://download.cloudstack.org/systemvm/4.11/ 
> <http://download.cloudstack.org/systemvm/4.11/>
> 
> Then manually overwrite it in the filesystem and update it accordingly in the 
> database?
> 
> Asai
> 
> 
>> On Aug 21, 2018, at 1:40 PM, Asai  wrote:
>> 
>> 4.11.0
>> 
>> As outlined in this 
>> http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.11.0.0/upgrade/upgrade-4.9.html
>>  
>> <http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.11.0.0/upgrade/upgrade-4.9.html>
>>> On Aug 21, 2018, at 1:37 PM, ilya musayev  
>>> wrote:
>>> 
>>> which template did you use? 
>>> 
>>>> On Aug 21, 2018, at 1:36 PM, Asai  wrote:
>>>> 
>>>> Greetings,
>>>> 
>>>> I just tried to upgrade from 4.9 to 4.11, but it looks like the system VM 
>>>> template I downloaded according to the upgrade guide is the wrong 
>>>> template.  It’s 4.11, but I upgraded to 4.11.1 and I get this error 
>>>> message:
>>>> 
>>>> Caused by: com.cloud.utils.exception.CloudRuntimeException: 4.11.1.0KVM 
>>>> SystemVm template not found. Cannot upgrade system Vms
>>>>at 
>>>> com.cloud.upgrade.dao.Upgrade41100to41110.updateSystemVmTemplates(Upgrade41100to41110.java:281)
>>>>at 
>>>> com.cloud.upgrade.dao.Upgrade41100to41110.performDataMigration(Upgrade41100to41110.java:68)
>>>>at 
>>>> com.cloud.upgrade.DatabaseUpgradeChecker.upgrade(DatabaseUpgradeChecker.java:578)
>>>>... 53 more
>>>> 2018-08-21 13:28:25,257 INFO  [o.e.j.s.h.ContextHandler] (main:null) 
>>>> (logid:) Started o.e.j.s.h.MovedContextHandler@15bfd87{/,null,AVAILABLE}
>>>> 2018-08-21 13:28:25,317 INFO  [o.e.j.s.AbstractConnector] (main:null) 
>>>> (logid:) Started ServerConnector@4b1c1ea0{HTTP/1.1,[http/1.1]}{:::8080}
>>>> 2018-08-21 13:28:25,318 INFO  [o.e.j.s.Server] (main:null) (logid:) 
>>>> Started @20725ms
>>>> 
>>>> Can anyone assist with getting this corrected?
>>>> 
>>>> Thank you,
>>>> Asai
>>>> 
>>>> 
>>> 
>> 
> 

Re: 4.9 to 4.11 upgrade broken

[ilya musayev]

which template did you use? 

> On Aug 21, 2018, at 1:36 PM, Asai  wrote:
> 
> Greetings,
> 
> I just tried to upgrade from 4.9 to 4.11, but it looks like the system VM 
> template I downloaded according to the upgrade guide is the wrong template.  
> It’s 4.11, but I upgraded to 4.11.1 and I get this error message:
> 
> Caused by: com.cloud.utils.exception.CloudRuntimeException: 4.11.1.0KVM 
> SystemVm template not found. Cannot upgrade system Vms
>   at 
> com.cloud.upgrade.dao.Upgrade41100to41110.updateSystemVmTemplates(Upgrade41100to41110.java:281)
>   at 
> com.cloud.upgrade.dao.Upgrade41100to41110.performDataMigration(Upgrade41100to41110.java:68)
>   at 
> com.cloud.upgrade.DatabaseUpgradeChecker.upgrade(DatabaseUpgradeChecker.java:578)
>   ... 53 more
> 2018-08-21 13:28:25,257 INFO  [o.e.j.s.h.ContextHandler] (main:null) (logid:) 
> Started o.e.j.s.h.MovedContextHandler@15bfd87{/,null,AVAILABLE}
> 2018-08-21 13:28:25,317 INFO  [o.e.j.s.AbstractConnector] (main:null) 
> (logid:) Started ServerConnector@4b1c1ea0{HTTP/1.1,[http/1.1]}{:::8080}
> 2018-08-21 13:28:25,318 INFO  [o.e.j.s.Server] (main:null) (logid:) Started 
> @20725ms
> 
> Can anyone assist with getting this corrected?
> 
> Thank you,
> Asai
> 
> 

Re: Upgrading from 4.9 to 4.10

[ilya musayev]

+1 on 4.11 - it’s LTS release and got much more attention

On Wed, Aug 15, 2018 at 9:13 AM Dag Sonstebo 
wrote:

> Asai,
>
> First of all I strongly advise you to upgrade to 4.11.1 instead of 4.10 –
> this will cause you a lot less pain.
>
> With regards to the template upload in 4.9 – do template uploads normally
> work? I’d suggest you check through the management-server.log and cloud.log
> on the SSVM to troubleshoot further. Also maybe destroy the SSVM and let
> this recreate, just in case it’s not healthy.
>
> Regards,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
>
> On 15/08/2018, 17:09, "Asai"  wrote:
>
> Greetings,
>
> We’re attempting an upgrade from 4.9 to 4.10, but we cannot seem to
> get past the SystemVM 4.10 download stage.  When registering a new template
> according to the documentation, the newly created systemvm-4.10 never
> enters the ready state.  I have tried downloading from the repository as
> well as uploading the systemvm from my local computer but it never seems to
> complete, and we cannot move forward.
>
> Can anyone share any insights into this problem?
> Asai
>
>
> dag.sonst...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>

Re: LDAP in 4.11.1

[ilya musayev]

You are most likely getting incorrect query to ldap.

If you can - use ldap connectivity without ssl and do tcpdump to see if you
can capture the ldapquery.

You can then identify the issue. Alternately, you can look into enabling
trace logging for ldap module and perhaps it prints the query.

Daan can shed more details on this issue as he was the one working on it.

On Thu, Aug 2, 2018 at 12:55 PM  wrote:

> Yes, full restart of management server to be safe. I have tried putting in
> the LDAP name, the Pre-Windows name, etc..
>
> -Original Message-
> From: Rafael Weingärtner 
> Sent: Thursday, August 2, 2018 1:22 PM
> To: users 
> Subject: Re: LDAP in 4.11.1
>
> Did you restart ACS after configuring it?
>
> On Thu, Aug 2, 2018 at 2:03 PM,  wrote:
>
> > Hello guys,
> >
> >
> >
> > I am attempting to bind an ACS domain to an LDAP domain and I get the
> > error "can not link a domain unless a basedn is configured for it" but
> > this is set in the settings. Anyone seen this?
> >
> >
> >
> >
> >
> >
>
>
> --
> Rafael Weingärtner
>
>

Re: [PROPOSE] Combining Apache CloudStack Documentation

[ilya musayev]

I like it but wonder if Upgrade section needs to be added? ..

On Tue, Jul 24, 2018 at 2:25 AM Paul Angus  wrote:

> Hi All,
>
> We currently have four sources of documentation [1]. Which make managing
> the documentation convoluted, and worse, make navigating and searching the
> documentation really difficult.
>
> I have taken the current documentation and combined them into one repo,
> then created 7 sections:
>
> CloudStack Concepts and Terminology
> Quick Installation Guide
> Installation Guide
> Usage Guide
> Developers Guide
> Plugins Guide
> Release Notes
>
> I haven't changed any of the content, but I've moved some of it around to
> make more sense (to me).  You can see the result on RTD [2]
>
> I'd like to PROPOSE to move this demo version of the documentation over to
> the Apache repos and make it THE documentation source, update the website,
> and mark the current repos/sites as archive data.
>
> [1]
> https://github.com/apache/cloudstack-docs.git <
> https://github.com/apache/cloudstack-docs.git> is a bit of a dodge-podge
> of resources
> https://github.com/apache/cloudstack-docs-install.git is the install guide
> https://github.com/apache/cloudstack-docs-admin.git is the current admin
> manual.
> https://github.com/apache/cloudstack-docs-rn.git is the release notes for
> individual releases
>
> [2]  https://beta-cloudstack-docs.readthedocs.io/en/latest/
>
> paul.an...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>

Re: 4.11.0 -> 4.11.1 problem: Guest VMs losing connection after few minutes

[ilya musayev]

Have you tried destroying router vm and let CloudStack create new one ?

On Fri, Jul 20, 2018 at 1:33 AM Jevgeni Zolotarjov 
wrote:

> - an ip-address conflict.
>   JZ: unlikely, but not impossible. I tried to restart router VM in
> Network-Guest networks -> defaultGuestNetwork -> VirtualAppliances
> While rebooting ping to this router VM disappeared. Hence, no other device
> is using the same IP.
> But!!! when this virtual router started, then network connection to all
> guest VMs disappeared. So, it must be something with this virtual router.
>
> - flakey hardware being one of
> -+ if card in the host
> JZ: higly unlikely
>
> -+ a router with bad firmware
> JZ: also unlikely
>
> - of course a strange cofiguration of the software router in you host might
> be the issue as well
> JZ: I didnt do any special configuration. Just used default.
>
> by all I know this happening after upgrade sounds like an unhappy incident
> but can't be sure.
> The iptables restart, was this on the VirtualRouter or on the host, or
> maybe on the guest? and the restart network?
>
> JZ: iptables restart on host machine. (or network restart on host)
>
>
>
> On Fri, Jul 20, 2018 at 11:14 AM Daan Hoogland 
> wrote:
>
> > that behaviour sound familiar from a couple of cases:
> > - an ip-address conflict.
> > - flakey hardware being one of
> > -+ if card in the host
> > -+ a router with bad firmware
> > - of course a strange cofiguration of the software router in you host
> might
> > be the issue as well
> >
> > by all I know this happening after upgrade sounds like an unhappy
> incident
> > but can't be sure.
> > The iptables restart, was this on the VirtualRouter or on the host, or
> > maybe on the guest? and the restart network?
> >
> >
> > On Fri, Jul 20, 2018 at 7:43 AM, Jevgeni Zolotarjov <
> > j.zolotar...@gmail.com>
> > wrote:
> >
> > > I updated cloudstack 4.11.0 -> 4.11.1
> > >
> > > Everything went OK during update, but after host reboot guest VMs lost
> > > connection after few minutes of normal work.
> > > I tried restarting network - systemctl restart network.service
> > > then connection was restored again for few minutes
> > >
> > > Finally I could restore connection by restarting iptables - systemctl
> > > restart iptables.service
> > >
> > > But then again guest VMs lost connection after few minutes of normal
> > > operation.
> > > The time of normal operation can be 5 minutes, but sometimes up to 40
> > > minutes.
> > >
> > > Please help me to track the root cause and fix it
> > >
> > > Host OS - Centos 7.5
> > > virtualisation - KVM
> > >
> >
> >
> >
> > --
> > Daan
> >
>

Re: Storage traffic clarification.

[ilya musayev]

Jon

with Basic Network - it implies you have all in one network for everything.

If you have a storage network that is L3 routable and you don’t want to use
guest network - then when you create a zone - use storage label and define
what bridge will be used to get there.

If it’s not guest bridge you wan to use - then use the management Bridge.

 Regards
Ilya

On Wed, Jun 20, 2018 at 12:25 AM Jon Marshall  wrote:

> I am probably missing something obvious but according to this article (
> https://www.shapeblue.com/understanding-cloudstacks-physical-networking-architecture/)
> by default primary and secondary storage traffic travels across the
> management network.
>
> As an example assume basic networking with 2 NICS, one for management with
> an IP subnet,  the other NIC for guest traffic using a different subnet. A
> physical host should only have one default gateway and this would have to
> be from the guest VM subnet.
>
> I setup two tests  -
>
> 1) the NFS server had an IP address from the management subnet
>
> 2) the NFS server was on a completely different IP subnet ie. not the
> management or the guest IP subnets.
>
> Both worked but in test 2 I can't see how the storage traffic could be
> using the management NIC because there is no default gateway on the compute
> nodes for the management subnet and the NFS server is on a remote network.
>
> So is storage traffic in test 2 actually running across the guest NIC ?
>
> And as the recommendation is to have separate storage from guest traffic
> does this mean the NFS server has to be in the management subnet ?
>
> Thanks
>

Convert KVM Instance to CloudStack

[ilya musayev]

Hi Users and Dev

I apologize for cross posting.. 

I have bunch of VMs that were deployed by CloudStack - however - the management 
server along with a DB is no longer available.

This is a POC environment - but i would love not to loose and recreate the VMs 
if possible,

Hence i’m thinking of writing re-injestion process of existing running KVM 
instances back into new cloudstack - without doing template imports and such.

Has anyone create a tooling for this endevour by any chance? If not - i might 
have to create one :(


Thanks
ilya

Re: Dynamic roles question

[ilya musayev]

Ivan

This is already done in 4.11, I’m not next to comp to check but ShapeBlue
has a feature created that would allow for movement between different roles.

Regards
ilya

On Thu, May 17, 2018 at 6:03 AM Ivan Kudryavtsev <kudryavtsev...@bw-sw.com>
wrote:

> Hello, community.
>
> I'm thinking about implementing the feature for accounts which permits to
> change account role. Basically, the rationale is trials or demonstration
> modes which restricts users from doing extra stuff, like VM creation,
> service offering changes, etc. Basically, after trial the account should be
> switched to a normal mode or removed. By permitting such role switching we
> can support the feature, otherwise we have to create unique role for every
> user and manage it separately. Please, let me know your thoughts about
> that. Have a good day.
>
> --
> With best regards, Ivan Kudryavtsev
> Bitworks Software, Ltd.
> Cell: +7-923-414-1515
> WWW: http://bitworks.software/ <http://bw-sw.com/>
>

Re: related to CLOUDSTACK-10310 Fix KVM reboot on storage issue need workaround

[ilya musayev]

We maybe missing a bit of context here - are you using NFS as shared
storage in the cluster?

If so - are you certain you aren’t loosing connectivity to NFS primary
storage?

Please upload the agent.log to pastebin or similar site and share the link.



On Tue, May 15, 2018 at 3:13 AM hanumant borwandkar <
hanumant.borwand...@gmail.com> wrote:

> Hi,
>
> We are using cloudstack inhouse with the version
>
> cloudstack-common-4.9.2.0-1.el7.centos.x86_64
> cloudstack-agent-4.9.2.0-1.el7.centos.x86_64
>
> But unfortunately sometime or after every few day compute host getting
> rebooted by cloudstack-agent and all VM running on that compute get
> affected.
>
> It seems that I m facing issue related to* CLOUDSTACK-10310 Fix KVM reboot
> on storage issue.*
>
> I tried to modify
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
> as per github follows
>
> *  /usr/bin/logger -t heartbeat "kvmheartbeat.sh stopped cloudstack-agent
> because it was unable to write the heartbeat to the storage."*
> *  sync &*
> *  sleep 5*
> *  #echo b > /proc/sysrq-trigger*
> * service cloudstack-agent stop*
>
> But no improvement still compute getting rebooted .
>
> Can someone able to provide workaround or fix for this issue
>
> Regards,
> Hanumant Borwandkar
>

Re: Cloudtack manager UI returns Error 404

[ilya musayev]

You must be running java 1.7 - I assume Suresh was alluding to this.
We’ve seen this error with 1.8...

On Wed, May 16, 2018 at 10:36 PM Suresh Kumar Anaparti <
sureshkumar.anapa...@gmail.com> wrote:

> Hi Natalia,
>
> Any changes in the java version or system configuration?
>
> -Suresh
>
> On Thu, May 17, 2018 at 4:14 AM, Dag Sonstebo 
> wrote:
>
> > Hi Natalia,
> >
> > My guess is you have some sort of corruption on your original host. Since
> > the management service itself is stateless it’s generally much quicker
> just
> > building a new one than trying to recover the old one.
> >
> > So my suggestions would be:
> > - Recover your DB to a dedicated DB host if you haven’t done so already.
> > - Build a new management server and just point this to the original DB as
> > described in http://docs.cloudstack.apache.org/projects/cloudstack-
> > installation/en/4.9/management-server/index.html#
> > additional-management-servers
> > NOTE: do not use the “–deploy-as” option as this will write a new blank
> > database. If you used encryption keys during the original setup you also
> > need to specify these.
> >
> > Once you have these two speaking you should in theory be back up and
> > running again.
> >
> > Regards,
> > Dag Sonstebo
> > Cloud Architect
> > ShapeBlue
> >
> >
> > dag.sonst...@shapeblue.com
> > www.shapeblue.com
> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > @shapeblue
> >
> >
> >
> > On 16/05/2018, 16:44, "Natalia Costas Lago"  wrote:
> >
> >
> > Dear all,
> >
> > For some reason our cloudstack manager died and now we are not able
> to
> > make it work. Our cloudstack version is 4.9.0.
> >
> > We tried to do a clean install of the manager and recover the
> database
> > from the last backup, but we were not able to recover the service.
> >
> > At present we can see in the logs that it almost boot, but at the end
> > we
> > get this error:
> >
> > May 16 17:33:51 cldmanager.srv.cesga.es server: INFO
> > [o.a.c.e.o.NetworkOrchestrator] (localhost-startStop-1:null) (logid:)
> > Network Manager will run the NetworkGarbageCollector every '600'
> > seconds.
> > May 16 17:33:51 cldmanager.srv.cesga.es server: INFO
> > [c.c.a.ApiServer]
> > (Thread-12:null) (logid:) ApiServer listening on port 8096
> > May 16 17:33:51 cldmanager.srv.cesga.es server: INFO
> > [o.a.c.s.SecondaryStorageManagerImpl] (localhost-startStop-1:null)
> > (logid:) Start secondary storage vm manager
> > May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN No
> > appenders
> > could be found for logger (com.cloud.utils.db.ConnectionConcierge).
> > May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN Please
> > initialize the log4j system properly.
> > May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN See
> > http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
> > May 16 17:34:20 cldmanager.srv.cesga.es server: Exception in thread
> > "CapacityChecker" java.lang.NoClassDefFoundError:
> > org/apache/cloudstack/managed/context/ManagedContextTimerTask$1
> > May 16 17:34:20 cldmanager.srv.cesga.es server: at
> > org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(
> > ManagedContextTimerTask.java:27)
> > May 16 17:34:20 cldmanager.srv.cesga.es server: at
> > java.util.TimerThread.mainLoop(Timer.java:555)
> > May 16 17:34:20 cldmanager.srv.cesga.es server: at
> > java.util.TimerThread.run(Timer.java:505)
> > May 16 17:34:20 cldmanager.srv.cesga.es server: Caused by:
> > java.lang.ClassNotFoundException:
> > org.apache.cloudstack.managed.context.ManagedContextTimerTask$1
> > May 16 17:34:20 cldmanager.srv.cesga.es server: at
> > org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
> > WebappClassLoaderBase.java:1892)
> > May 16 17:34:20 cldmanager.srv.cesga.es server: at
> > org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
> > WebappClassLoaderBase.java:1735)
> > May 16 17:34:20 cldmanager.srv.cesga.es server: ... 3 more
> > May 16 17:34:24 cldmanager.srv.cesga.es server: Exception in thread
> > "Timer-1" java.lang.NoClassDefFoundError:
> > org/apache/cloudstack/managed/context/ManagedContextTimerTask$1
> > May 16 17:34:24 cldmanager.srv.cesga.es server: at
> > org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(
> > ManagedContextTimerTask.java:27)
> > May 16 17:34:24 cldmanager.srv.cesga.es server: at
> > java.util.TimerThread.mainLoop(Timer.java:555)
> > May 16 17:34:24 cldmanager.srv.cesga.es server: at
> > java.util.TimerThread.run(Timer.java:505)
> >
> > Any help is appreciated..
> >
> > Kind regards,
> >
> > NOTA: Para cualquier incidencia/consulta/petición que requiera
> > seguimiento por favor enviar a helpdesk_comunicacio...@cesga.es
> >
> > --
> > 

Re: ssvm NFS public ip

[ilya musayev]

There is a global setting you have to set to use internal non public IP.

Try setting sec.storage.allow.internal.site to an internal network cidr.

You may need to destroy ssvm for settings to take effect. In my case there
is some sort of minor bug where it takes upward of 5 minutes for ssvm to
program the internal routes, but since this is onetime operation - I just
live with it.

On Wed, Apr 11, 2018 at 10:50 AM Nicolas Bouige  wrote:

> a small update about my problem.
>
> I 've recreated the zone from scratch this morning and  one of my
> "cloudbr" used for the secondary storage was misconfigured.
>
> So Now, I can ping the secondary storage from KVM host, CS-MGMT,  SSVM and
> mount the nfs on them...but...the agent still not going up and the
> ssvm_check.sh give me an ip public for the nfs instead of the private.
>
>
> i got only this error in /var/log:cloud.log :
>
> 17:30:22,600 ERROR AgentShell:477 - Unable to start agent: Resource class
> not found: com.cloud.storage.resource.PremiumSecondaryStorageResource due
> to: java.lang.ClassNotFoundException:
> com.cloud.storage.resource.PremiumSecondaryStorageReso
>   urce
> Unable to start agent: Resource class not found:
> com.cloud.storage.resource.Prem
> iumSecondaryStorageResource due to: java.lang.ClassNotFoundException:
> com.cloud.
> storage.resource.PremiumSecondaryStorageResource
>
> i've tried to update a differente systemvm template with no success...
> I 've compared the configuration of the ssvm with one of our deployment
> and i juste noticed the "resource"
> (org.apache.cloudstack.storage.resource.NfsSecondaryResource) was not the
> same.
> i changed it but still the same error...
>
> There is a way to find java binaries/script and  upload them on the ssvm ?
>
> If one of you have an idea, it would be appreciate.
>
> Thanks !
>
> Best regards,
> N.B
>
>

Re: [DISCUSS] CloudMonkey 6.0.0-alpha (about six years after initial version in 2012)

[ilya musayev]

This is great news and cloud monkey is used more than you think :)

I will share the news with my team.

On Tue, Apr 10, 2018 at 5:07 AM Will Stevens  wrote:

> +1. It has been a great tool for years.  Looking forward to the golang
> version.
>
> On Apr 10, 2018 7:59 AM, "Rohit Yadav"  wrote:
>
> All,
>
>
> Few months ago, I started porting the current code to be compatible with
> both Python2 and Python3 to make it run with both Python2 (for older
> systems such as CentOS6 etc) and Python3 (for newer platforms). The work
> was not a success, another problem was that cloudmonkey was not easy to
> install and required several dependencies that would certainly fail on
> older systems with Python 2.6.x.
>
>
> Considering all things, I started working on an experimental golang port
> [2] and happy to announce that the initial alpha version shows a lot of
> promise and is 5-20x faster than the python based cli [1]. The compiled
> binary runs on several targets, including windows [1].
>
>
> I cannot commit to a timeline/release date yet but the aim of this thread
> is to discuss and propose the simplification of the CLI which may require
> removal of some features and some breaking changes may be introduced:
>
>
> - Make json the default output format
>
> - Remove coloured output
>
> - Remove unpopular, least user output formats? xml, default (line-separate
> key=value), table?
>
> - Remove `set` options: color, expires, (custom) prompt
>
> - Remove `paramcompletion` option, this will be true/enabled by default
>
> - Remove signature version and expires (I'm not sure why this is needed or
> used)
>
> - Remove history_file, cache_file, log_file options, use the default paths
> in folder at (user's  home directory)/.cloudmonkey.
>
> - Remove shell based execution from interactive interpreter mode (using !
> or shell keywords)
>
> - Remove support for CloudStack older than 4.5, i.e. it won't be tested
> against older cloudstacks.
>
> - Remove a default API cache with the client, for a fresh env without any
> ~/.cloudmonkey/cache; users can run `sync` command against a management
> server.
>
> - Interactive API parameter completion in CLI mode: the current API
> parameter completion requires the user to manually copy/paste the uuids, or
> autocomplete by typing parts of the uuids/option.
>
> - Improve how maps are passed.
>
> - Good to have: bash/zsh completion.
>
>
> Please share your thoughts, and objections (especially if you're using the
> proposed features to be removed in version 6.x).
>
>
> [1] https://twitter.com/rhtyd/status/983448788059770882
>
> [2] https://github.com/rhtyd/cmk
>
>
> - Rohit
>
> 
>
>
>
> rohit.ya...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>

Re: 4.11.0.0 problem adding new shared network NIC to VM "A NIC with this MAC address exits for network:"

[ilya musayev]

Stephan

Please kindly open a jira issue as a blocker for 4.11

Thank you


On Tue, Apr 3, 2018 at 2:12 AM Stephan Seitz 
wrote:

> Hi there!
>
> After upgrading a working 4.9.2 setup to 4.11.0.0 we've noticed that we're
> unable to add a NIC (shared network) to a running VM via GUI.
> The Error is "A NIC with this MAC address exits for network: (uuid of the
> current (other NIC) network)"
> Using cloudmonkey and giving a self-computed MAC-address does not rise the
> error and works as expected.
>
> After digging into the code, I think I've found the problem, but am too
> bad in java to provide a sustainable fix.
>
> I've attached a patch which obviously (with own manual tests) fixes that
> issue by simply commenting out the respective codeblock.
>
> As far as I dug into the code, the third parameter of NicProfile is
> handled by
>
> NetUtils.long2Mac(NetUtils.createSequenceBasedMacAddress(ipVO.getMacAddress(),
> NetworkModel.MACIdentifier.value())
>
> inside
>
> NicProfileHelperImpl.java
>
> I assume createSequenceBasedMacAddress can handle null as MAC, but doesn't
> get called at all if
>
>
> _nicDao.findByNetworkIdAndMacAddress(networkId, macAddress) results !=
> null if macAddress is null.
>
>
>
> Sorry, if I'm wrong, didn't had decent IDE at hand :)
>
>
>
> Cheers,
>
> Stephan Seitz
>
> --
>
> Heinlein Support GmbH
> Schwedter Str. 8/9b, 10119 Berlin
>
> http://www.heinlein-support.de
>
> Tel: 030 / 405051-44
> Fax: 030 / 405051-19
>
> Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht
> Berlin-Charlottenburg,
> Geschäftsführer: Peer Heinlein -- Sitz: Berlin
>
>
>

Re: CS 4.11 : Erreor : status Index: 0, Size: 0 When Migrate instance !

[ilya musayev]

I was going to mention same thing.

Try MariaDB 5.5 - which is equivalent to MySQL 5.6

I know we did work to support MySQL 5.7 - but I personally haven’t tested
it and have no experience. I’m not certain what MariaDB equivalent would be.

Also check the output of Catalina.out for SQL related errors.

On Wed, Mar 28, 2018 at 12:10 PM Dag Sonstebo 
wrote:

> Hi Olivier,
>
>
>
> I’ve just spun the same environment as you have up in the lab, using
> CentOS7 management, XS6.5SP1 hypervisors, advanced zone and installing from
> http://cloudstack.apt-get.eu/centos/7/4.11/ , and I can’t reproduce the
> issues you are seeing.
>
> The only difference in my environment is the MariaDB version – and I
> suspect this may be your problem.
>
>
>
> [root@ref-trl-418-x-cs411-dsonstebo-mgmt1 ~]# cat
> /etc/yum.repos.d/cloudstack.repo | grep http
>
> baseurl=http://cloudstack.apt-get.eu/centos/7/4.11/
>
>
>
> [root@ref-trl-418-x-cs411-dsonstebo-mgmt1 ~]# cat /etc/redhat-release
>
> CentOS Linux release 7.4.1708 (Core)
>
>
>
> [root@ref-trl-418-x-cs411-dsonstebo-mgmt1 ~]# mysql --version
>
> *mysql  Ver 15.1 Distrib 5.5.56-MariaDB, for Linux (x86_64) using readline
> 5.1   My lab version*
>
> *mysql  Ver 15.1 Distrib 10.0.34-MariaDB, for Linux (x86_64) using
> readline 5.1 << Your version *
>
>
>
> > SELECT * FROM cloud.version
>
>
>
> + --- +  +  + - +
>
> | id  | version  | updated  | step  |
>
> + --- +  +  + - +
>
> | 1   | 4.0.0| 2018-03-28 18:44:26 | Complete  |
>
> | 2   | 4.1.0| 2018-03-28 18:44:49 | Complete  |
>
> …..
>
> | 21  | 4.9.3.0  | 2018-03-28 18:45:02 | Complete  |
>
> | 22  | 4.10.0.0 | 2018-03-28 18:45:03 | Complete  |
>
> | 23  | 4.11.0.0 | 2018-03-28 18:45:03 | Complete  |
>
>
>
> > SELECT * FROM cloud.host_view
>
>
>
>  1. row *
>
>   id: 1
>
> uuid: df4cbefc-3dc1-4f41-ac7e-723920808114
>
> name: ref-trl-418-x-cs411-dsonstebo-xs1
>
>   status: Up
>
> disconnected:
>
> type: Routing
>
>   private_ip_address: removed
>
>  version: 4.11.0.0
>
>  hypervisor_type: XenServer
>
>   hypervisor_version: 6.5.0
>
> capabilities: xen-3.0-x86_64 , xen-3.0-x86_32p ,
> hvm-3.0-x86_32 , hvm-3.0-x86_32p , hvm-3.0-x86_64
>
>last_ping: 1486584835
>
>  created: 2018-03-28 18:52:06
>
>  removed:
>
>   resource_state: Enabled
>
>   mgmt_server_id: 7393350059856
>
>  cpu_sockets: 3
>
> cpus: 3
>
>speed: 1994
>
>  ram: 7427139712
>
>   cluster_id: 1
>
> cluster_uuid: 30cb6537-8d6e-4a42-965c-b50c30262a55
>
> cluster_name: p1-c1
>
> cluster_type: CloudManaged
>
>   data_center_id: 1
>
> data_center_uuid: 407346a2-4ee3-4fa2-b17f-37f1fa1150df
>
> data_center_name: ref-trl-418-x-cs411-dsonstebo
>
> data_center_type: Advanced
>
>   pod_id: 1
>
> pod_uuid: 68481e39-9427-486f-a983-71b36737fae4
>
> pod_name: Pod1
>
>  tag: GPU
>
> guest_os_category_id:
>
>   guest_os_category_uuid:
>
>   guest_os_category_name:
>
> memory_used_capacity: 536870912
>
> memory_reserved_capacity: 0
>
>cpu_used_capacity: 500
>
>cpu_reserved_capacity: 0
>
>   job_id:
>
> job_uuid:
>
>   job_status:
>
>   job_account_id:
>
> oobm_enabled:
>
> oobm_power_state:
>
>   ha_enabled:
>
> ha_state:
>
>  ha_provider:
>
>   annotation:
>
>   last_annotated:
>
> username:
>
>  2. row *
>
>
>
>
>
> Regards,
>
> Dag Sonstebo
>
> Cloud Architect
>
> ShapeBlue
>
>
>
> dag.sonst...@shapeblue.com
> www.shapeblue.com
> @shapeblue
>
>
>
>
> *From: *Olivier GUIN 
>
>
> *Reply-To: *"users@cloudstack.apache.org" , "
> olivier.g...@ariasnet.com" 
>
> *Date: *Wednesday, 28 March 2018 at 15:35
>
>
> *To: *"users@cloudstack.apache.org" 
> *Subject: *Re: CS 4.11 : Erreor : status Index: 0, Size: 0 When Migrate
> instance !
>
>
>
> Hi,
>
> It's new install, CS 4.11 on CentOS7, I've mariadb cluster
>
> I use my script https://cloudbox.wayscom.com/index.php/s/FHCwWt35fGZk0gD
>
> [root@crtl-1 opt]# mysql --version
> mysql  Ver 15.1 Distrib 10.0.34-MariaDB, for Linux (x86_64) using readline
> 5.1
> [root@crtl-1 opt]# cat /etc/redhat-release
> CentOS Linux release 7.4.1708 (Core)
>
> My cloudstack.repo :
> [cloudstack]
> name=cloudstack
> 

Re: Welcoming Mike as the new Apache CloudStack VP

[ilya musayev]

Welcome Mike, thank you Wido!

On Mon, Mar 26, 2018 at 8:59 AM Simon Weller 
wrote:

> Thanks for all of your hard work Wido, we really appreciate it.
>
>
> Congratulations Mike!
>
>
> - Si
>
> 
> From: Wido den Hollander 
> Sent: Monday, March 26, 2018 9:11 AM
> To: d...@cloudstack.apache.org; users@cloudstack.apache.org
> Subject: Welcoming Mike as the new Apache CloudStack VP
>
> Hi all,
>
> It's been a great pleasure working with the CloudStack project as the
> ACS VP over the past year.
>
> A big thank you from my side for everybody involved with the project in
> the last year.
>
> Hereby I would like to announce that Mike Tutkowski has been elected to
> replace me as the Apache Cloudstack VP in our annual VP rotation.
>
> Mike has a long history with the project and I am are happy welcome him
> as the new VP for CloudStack.
>
> Welcome Mike!
>
> Thanks,
>
> Wido
>

Re: Intel meltdown/spectre kvm upgrade results

[ilya musayev]

Thanks Ivan

On Fri, Jan 12, 2018 at 9:00 PM Ivan Kudryavtsev 
wrote:

> Hi, colleagues,
>
> just would like to share that yesterday successfuly upgraded my ubuntu
> 14.04 kvm cloud to custom built linux 4.14.11 keenel with ubuntu 2018/01/08
> intel cpu microcode update. Compute CPUs - Xeon E5-2670, Xeon X5650,
> everything works nice, no claims from customers, no sensitive load change.
> Live migration between new and old kernels goes well, back migration too.
> It seems that kvm, libvirt and qemu patches are not here yet for Ubuntu.
> Waiting for additional updates. Btw, It is CS4.3.
>
> Have a nice migration.
>

Re: Performance considerations related to Intel Meltdown on KVM CPU types

[ilya musayev]

Thanks for sharing

On Mon, Jan 8, 2018 at 7:11 AM Nux!  wrote:

> Hello,
>
> Just stumbled upon this
> https://twitter.com/berrange/status/950209752486817792
>
> "ensure KVM guest CPU model you choose has the "pcid" feature, otherwise
> guests will suffer terrible performance from the Meltdown fixes. This means
> using a named Haswell, Broadwell or Skylake based model or host passthrough"
>
>
> This means whoever is running with the KVM default CPU (like I do) as
> opposed to specific ones or host passthrough needs to change this in order
> to avoid bad performance once the new mitigating kernel is installed.
>
>
> Bad news is older Xeons do not support this, check if "invpcid" flag shows
> up in /proc/cpuinfo (you might see "pcid", that one is not enough).
>
>
>
>
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
>

Re: IPMI out of management

[ilya]

Probably not the answer you'd expect to hear - but if you really need to
have - try backporting and building from source.

On 8/15/17 4:01 AM, victor wrote:
> Hello Rohit,
> 
> As mentioned in the following url, IPMI method will help to achieve HA
> with KVM hypervisor.
> 
> 
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+HA+with+IPMI+Fencing
> 
> =
> 
> I have successfully implemented IPMI out of management with my 2 KVM
> hyper-visors successfully.  But I couldn't find a way how we can achieve
> HA with the host.  What will happen to the VM's in a host which is in
> halted state. Can you explain how the out of management help you in this
> case. I couldn't trace out much from the logs.
> 
> Regards
> Victor
> 
> 
> On 08/07/2017 12:33 AM, Rohit Yadav wrote:
>> Victor,
>>
>>
>> Which proposed features do you want to know, if they are covered with
>> ipmi oobm? As per current master, 4.10/4.9 releases, IPMI based
>> out-of-band management works with CloudStack.
>>
>>
>> - Rohit
>>
>> 
>> From: victor 
>> Sent: Thursday, August 3, 2017 6:44:26 AM
>> To: users@cloudstack.apache.org; Rohit Yadav
>> Subject: Re: IPMI out of management
>>
>> Hello Guys,
>>
>> I am also able to successfully configure ipmi fencing with cloudstack.
>> Also can you guys let me know whether all the proposed features are
>> fully covered with the  ipmi out of management.
>>
>> Regards
>> Victor
>>
>> On 08/03/2017 03:35 AM, Rohit Yadav wrote:
>>> Thanks Gabriel, Rodrigo -- good to know this is in use.
>>>
>>>
>>> Victor - yes it works, there is a ipmisim [1] tool you can test the
>>> implementation against, as well as real h/w.
>>>
>>>
>>> [1] https://pypi.python.org/pypi/ipmisim
>>>
>>>
>>> - Rohit
>>>
>>> 
>>> From: Rodrigo Baldasso 
>>> Sent: Tuesday, August 1, 2017 2:38:28 PM
>>> To: users@cloudstack.apache.org
>>> Subject: Re: IPMI out of management
>>>
>>> I'm using here and works fine with my Supermicro servers.
>>>
>>> - - - - - - - - - - - - - - - - - - -
>>>
>>> Rodrigo Baldasso - LHOST
>>>
>>> (51) 9 8419-9861
>>> - - - - - - - - - - - - - - - - - - -
>>> On 01/08/2017 09:01:20, victor  wrote:
>>> Hello Guys,
>>>
>>> Have anybody able to configure and test ipmi "out of management" with
>>> cloudstack successfully.
>>>
>>> Regards
>>>
>>> Victor
>>>
>>>
>>>
>>> rohit.ya...@shapeblue.com
>>> www.shapeblue.com
>>> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>>> @shapeblue
>>>
>>>
>>>
>>
>> rohit.ya...@shapeblue.com
>> www.shapeblue.com
>> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> @shapeblue
>>     
> 

Re: Failed to find db.properties

[ilya]

Rafael has a point there - please try with tomcat7

Also - please look into catalina.out/log for further debugging info.

On 8/23/17 9:23 AM, Rafael Weingärtner wrote:
> did you run setup-management --tomcat7?
> 
> On Wed, Aug 23, 2017 at 12:20 PM, Jevgeni Zolotarjov > wrote:
> 
>> Yes, I did, as it is suggested in the guide:
>> cloudstack-setup-database then setup-management
>>
>>
>> On Aug 23, 2017 7:11 PM, "Rafael Weingärtner" >>
>> wrote:
>>
>> did you run cloudstack-setup-database and ...setup-management?
>>
>>
>> On Wed, Aug 23, 2017 at 11:16 AM, Jevgeni Zolotarjov <
>> j.zolotar...@gmail.com
>>> wrote:
>>
>>> Here you go
>>>
>>> management-server.log is not available, which probably means, the
>>> application did not reach that far and exited due to missing DB access
>>>
>>> On Wed, Aug 23, 2017 at 6:03 PM, Gabriel Beims Bräscher <
>>> gabrasc...@gmail.com> wrote:
>>>
 Hi Jevgeni,

 Can you share with us some log files (catalina.log,
>> management-server.log
 and others that might be useful)?


 2017-08-23 11:53 GMT-03:00 Jevgeni Zolotarjov :

> Hi
>
> I am installing Cloudstack 4.10 on Debian Jessie using guides from
> http://cloudstack-installation.readthedocs.io/en/latest/
 management-server/
> index.html
>
> Apparently the installation process was smooth. But I cannot get
>> client
> running on tomcat7
> From catalina.out log file I can read
>
> Failed to find db.properties
>
> But this file is present in
> ./etc/cloudstack/management/db.properties
> ./etc/cloudstack/usage/db.properties
>
> Please help.
>

>>>
>>>
>>
>>
>> --
>> Rafael Weingärtner
>>
> 
> 
> 

Re: Instance with a larger disk size then Template

[ilya]

Just a thought - as i do this very frequently.

If you are using LVM on your ROOT partition - you dont need to power it
on via Live CD.

It can all be done online while the system running.




On 8/3/17 6:40 AM, Imran Ahmed wrote:
> Hi Erik,
> 
> Thanks for suggestion, I tried this too and was successful till lvextending 
> the logical volume. However at the stage of running resize2fs  it produced 
> errors like : Bad super block..."  so I ended up installing from an ISO and 
> partitioning without LVM this time so that I could use this template to 
> resize in future.
> 
> Cheers,
> 
> Imran 
> 
> -Original Message-
> From: Erik Weber [mailto:terbol...@gmail.com] 
> Sent: Thursday, August 03, 2017 3:56 PM
> To: users@cloudstack.apache.org
> Subject: Re: Instance with a larger disk size then Template
> 
> A faster approach than those mentioned is to create a new partition on
> the unused disk space, and add it to the volume group, then use
> lvextend and resizing the fs.
> 
> On Thu, Aug 3, 2017 at 12:00 PM, Imran Ahmed  wrote:
>> Hi All,
>>
>> I am creating an instance with a 300GB disk from a CentOS 7 template that
>> has 5GB disk (LVM Based).
>> The issue is that the root LVM partition inside the new VM instance  still
>> shows 5GB .
>>
>> The device size  (/dev/vda) however shows 300GB.  The question is what is
>> the best strategy to resize the root LVM partition so that I could use all
>> 300G.
>>
>> Kind regards,
>>
>> Imran
>>
> 

Re: AW: Instance with a larger disk size then Template

[ilya]

Great feedback - did not know cloud-init supported this.



On 8/3/17 3:17 AM, S. Brüseke - proIO GmbH wrote:
> Hi Imran,
> 
> you are talking about 3 different levels here to reach your goal of resizing 
> a volume. First level is the volume itself. This is what you can do within 
> CS. After that you need to extend the partition and then you need to expand 
> the filesystem. The last to levels you need to do within the os of the server.
> 
> What we do is using cloud-init within our template to automate this. But our 
> templates do not use LVM. Our templates are checking at boot if the root 
> volume has been extended and expanding the partition and the filesystem.
> 
> If you want to know more about it, I can give you more details.
> 
> Mit freundlichen Grüßen / With kind regards,
> 
> Swen Brüseke
> 
> -Ursprüngliche Nachricht-
> Von: Imran Ahmed [mailto:im...@eaxiom.net] 
> Gesendet: Donnerstag, 3. August 2017 12:00
> An: users@cloudstack.apache.org
> Betreff: Instance with a larger disk size then Template
> 
> Hi All,
> 
> I am creating an instance with a 300GB disk from a CentOS 7 template that has 
> 5GB disk (LVM Based).
> The issue is that the root LVM partition inside the new VM instance  still 
> shows 5GB .  
> 
> The device size  (/dev/vda) however shows 300GB.  The question is what is the 
> best strategy to resize the root LVM partition so that I could use all 300G.
> 
> Kind regards,
> 
> Imran 
> 
> 
> 
> - proIO GmbH -
> Geschäftsführer: Swen Brüseke
> Sitz der Gesellschaft: Frankfurt am Main
> 
> USt-IdNr. DE 267 075 918
> Registergericht: Frankfurt am Main - HRB 86239
> 
> Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte 
> Informationen. 
> Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich 
> erhalten haben, 
> informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. 
> Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nicht 
> gestattet. 
> 
> This e-mail may contain confidential and/or privileged information. 
> If you are not the intended recipient (or have received this e-mail in error) 
> please notify 
> the sender immediately and destroy this e-mail.  
> Any unauthorized copying, disclosure or distribution of the material in this 
> e-mail is strictly forbidden. 
> 
> 

Re: Some things I found out installing on Centos 7

[ilya]

Eric

Its a good feedback for us as community to focus on cleaning up
documentation. We are adding many features - and we need to make sure
they are properly reflected.


Also - way back when - some of the documentation was written by
technical writers sponsored by Citrix. I can only assume technical
writters did not understand what they were writting - because myself
being a cloudstack user for 5 years - i could not understand the meaning
of some sentences.

However what i failed to do - is raise an issue and help rewrite it.

Thanks for taking time to write this.

Regards,
ilya

PS:

I have mixed feelings with VMware implementation. I've been using it
since 2005. Perhaps for small to mid range setups - it will do well.

My experience with going above several hundred hypervisors was not
great. I wont go into details as to what happened - other than to say -
it has great number of challenges of its own (just like any other
solution). But i learned that KVM had less feature - but was also far
less complex and stable - and cloudstack helped bridge the gap of not
have vCenter.



On 8/2/17 1:12 AM, Eric Green wrote:
> First, about me -- I've been administering Linux systems since 1995. No, 
> that's not a typo -- that's 22 years. I've also worked for a firewall 
> manufacturer in the past, I designed the layer 2 VLAN support for a firewall 
> vendor, so I know VLAN's and such. I run a fairly complex production network 
> with multiple VLAN's, multiple networks, etc. already, and speak fluent Cisco 
> CLI. In short, I'm not an amateur at this networking stuff, but figuring out 
> how Cloudstack wanted my CentOS 7 networking to be configured, and doing all 
> the gymnastics to make it happen, consumed nearly a week because the 
> documentation simply isn't up to date, thorough, or accurate, at least for 
> Centos 7. 
> 
> So anyhow, my configuration:
> 
> Cloudstack 4.9.2.0 from the RPM repository at cloudstack.apt-get.eu
> 
> Centos 7 servers with:
> 
> 2 10gbit Ethernet ports -> bond0 
> 
> A handful of VLANS:
> 
> 100 -- from my top of rack switch is sent to my core backbone switch layer 3 
> routed to my local network as 10.100.x.x and thru the NAT border firewall and 
> router to the Internet. Management.
> 101 -- same but for 10.101.x.x  -- public.
> 102 -- same but for 10.102.x.x  -- guest public (see below).
> 192 -- A video surveillance camera network that is not routed to anywhere, 
> via a drop from the core video surveillance POE switch to an access mode port 
> on my top of rack switch. Not routed.
> 200 -- 10 gig drop over to my production racks to my storage network there 
> for accessing legacy storage. Not routed. (Legacy storage is not used for 
> Cloudstack instance or secondary storage but can be accessed by virtual 
> machines being migrated to this rack).
> 1000-2000 -- VLAN's that exist in my top of rack switch on the Cloudstack 
> rack and assigned to my trunk ports to the cloud servers but routed nowhere 
> else, for VPC's and such. 
> 
> Stuck with VLAN's rather than one of the SDN modules like VXNET because a) 
> it's the oldest and most likely to be stable, b) compatible with my 
> already-existing network hardware and networks (wouldn't have to somehow map 
> a VLAN to a SDN virtual network to reach 192 or 200 or create a public 102), 
> and c) least complex to set up and configure given my existing top-of-rack 
> switch that does VLANs just fine.
> 
> Okay, here's how I had to configure Centos 7 to make it work: 
> 
> enp4s[01] -> bond0 -> bond0.100 -> br100  -- had to create two interface 
> files, add them to bond0 bridge, then create a bond0.100 vlan interface, then 
> a br100 bridge,  for my management network. In
> /etc/sysconfig-network-scripts: 
> 
> # ls ifcfg-*
> ifcfg-bond0 ifcfg-bond0.100 ifcfg-br100 ifcfg-enp4s0 ifcfg-enp4s1
> 
> (where 4s0 and 4s1 are my 10 gigabit Ethernets).
> 
> Don't create anything else. You'll just confuse Cloudstack. Any other 
> configuration of the network simply fails to work. In particular, creating 
> br101 etc. fails because CloudStack wants to create its own VLANs and  
> bridges and if you traffic label it as br101 it'll try making vlan br101.101 
> (doesn't work, duh). Yes, I know this contradicts every single piece of 
> advice I've seen on this list. All I know is that this is what works, while 
> every other piece of advice I've seen for labeling the public and private 
> guest networking fails. 
> 
> When creating the networks in the GUI under Advanced networking, set bond0 as 
> your physical network and br100 as the KVM traffic label for the Management 
> network and Storage network and give them addresses with VLAN 100 (assuming 
> you're using the same network for both management and storage networks, which 
>

Re: CloudStack-UI 1.0.6 released on July, 25, 2017

[ilya]

this is awesome - will give this a try!

On 7/25/17 1:59 AM, Ivan Kudryavtsev wrote:
> Release 1.0.6 Overview
> 
> If you don't see a properly marked document and would like to see the same
> press release with images, follow the link:
> https://github.com/bwsw/cloudstack-ui/wiki/106-ReleaseNotes-En
> 
> On July 25, 2017 we released Cloudstack-UI 1.0.6. During the past sprint,
> project team concentrated on extensive code refactoring, which led to
> several bug-fixes and also allowed us to include new functions in the
> release.
> 
> Most important introductions of this release are new VM filtering and
> grouping interface, improved representation of resource usage bar, tabs for
> VM and Template tags and the time format choice.
> 
> Another far-reaching point of the release is our decision to migrate from
> MDL to Material2 with which we expect to deliver even more of the nice
> features.
> 
> An extended summary of the released features and improvements is provided
> below.
> 
> Refactoring
> 
> Developers have spent more than a week on refactoring the old code of VM
> creation which was written without proper design on early project phase. It
> has been reimplemented from scratch.
> New
> interface for VM filtering and grouping
> 
> A new component was created, which allows presenting a new interface for VM
> filtering and grouping in this release. In addition, this component will
> help to create a new way of VMs displaying - it will be possible to switch
> between views and a new form will be added - a table format similar to the
> one offered by the standard ACS interface.
> 
> Renewed
> VM tabs
> 
> Reorganized VM details tabs. Previously tabs were "keyword" labeled, in
> this release they were changed to icons. It helps to implement more tabs
> with compact and visually self-descriptive labels.
> 
> NIC and SG details and configuration are moved to a separate tab "Network"
> which will help to implement additional network operations for advanced CS
> zones.
> Improved
> Resource Usage Bar
> 
> Resource usage bar allows switching between "used" and "free" presentations
> to help users understanding capabilities in a better way.
> 
> Virtual
> Machines and Templates Tags
> 
> The release introduces VM tags and Template tags tabs. Unlike the native
> ACS interface which doesn't use tags, CloudStack-UI uses tags very
> extensively to provide additional UX capabilities. In future versions tags
> will interact with UI plugins to provide additional capabilities like
> clientless SSH access.
> 
> Time
> format choice
> 
> Users have the possibility to choose a time format (AM / PM <-> 24h), which
> is not represented in the native ACS interface. This functionality allows
> users to use the system in a convenient time mode in different locations.
> Moving
> from MDL to Material2
> 
> Since Angular Material 2 is mature now, includes better and more beautiful
> components we decided switching from MDL to Material2 which will be
> completed
> 
> in
> several iterations.
> Deployment
> Instructions
> 
> The release can be found at GitHub releases (
> https://github.com/bwsw/cloudstack-ui/releases/tag/1.0.6)
> 
> Prepared Docker image is available at Dockerhub (
> https://hub.docker.com/r/bwsw/cloudstack-ui/).
> 
> You can pull it with:
> 
> # docker pull bwsw/cloudstack-ui:1.0.6
> 
> The project changelog is here:
> https://github.com/bwsw/cloudstack-ui/wiki/Changelog
> 
> Deployment guide and project info can be found at GitHub pages:
> https://bwsw.github.io/cloudstack-ui/
> Release
> 1.0.7 Expectations
> 
> Version 1.0.7 is planned to be released on August, 7th, 2017. The release
> is expected to include new functionality as follows:
> 
>- Templates to VM tags copy support (client-side);
>- Web-SSH plugin, which enables SSH connection facility to VMs without
>desktop client installed;
>- Pulse plugin, which enables displaying of CPU/RAM/IO/NET metrics per
>VM.
> 
> Community
> Message
> 
> Dear community member, 

Re: CloudStack with advance networking

[ilya]

Luis

Please layout your configuration.

number of hosts
nics per host - any bonding?
type of hypervisor
clustered storage or local storage
network layout
do you want a simplified (firewall and ACL rules arent handled by
cloudstack) or do you want cloudstack to support it?


Depending on what it is you targeting - we have many community members
who specialize in specific setups.

On 7/25/17 5:29 AM, Luis wrote:
> Thank you for your replay
> I have a zone running but the Vlan's were not communicating I don't know if 
> they need an special configuration or just create the VLAN, then I created a 
> new installation with no vlans but secondary storage is not mounted it looks 
> like is not communicating with gateway, I am trying to do an advance zone 
> configuration but I am using internal IPs for public traffic, this is for a 
> laboratory, do you know how I should create the VLNAN's? I am using a Cisco 
> 3560
> Public IP's: 209.229.131.0/24Internal IP's:10.0.0.0/24
> but the gateway for the public traffic is not working.
> 
>   From: ilya <ilya.mailing.li...@gmail.com>
>  To: users@cloudstack.apache.org 
>  Sent: Monday, July 24, 2017 7:42 PM
>  Subject: Re: CloudStack with advance networking
>
> I looked over the guide - for current version - this guide is overly
> complex - it was also written 4 years ago.
> 
> As of 4.3+, all of the things you need can be handled in the UI.
> 
> It should take about 10 minutes to get the zone up - assuming you have
> your environment properly configured.
> 
> I will make a refreshed simplified version of this tutorial when time
> allows.
> 
> On 7/19/17 1:16 PM, Taylor wrote:
>> I am using it in a proof of concept to test out how quickly we could adopt 
>> it in production.
>>
>>
>> I dont know the answer to your question at this point, I am too new to the 
>> stack. But I am currently reading articles like this one: 
>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+Advanced+Network+Tutorial+-+Step+by+Step
>>
>> Good luck!
>>
>> Can anyone else offer advice?
>>
>> 
>> From: Luis <lmartinez...@yahoo.com>
>> Sent: Wednesday, July 19, 2017 3:57 PM
>> To: tschnei...@live.com
>> Subject: Re: CloudStack with advance networking
>>
>> Thank you for your replay, do you use it for production or personnal use?
>>
>> I have a diagram but i dont know if the vlans need some special co 
>> figuration like routing.
>>
>> Sent from Yahoo Mail on 
>> Android<https://overview.mail.yahoo.com/mobile/?.src=Android>
>>
>> On Wed, Jul 19, 2017 at 1:02 PM, Taylor
>> <tschnei...@live.com> wrote:
>>
>> I have not tried the advanced install yet but I am preparing to.
>>
>> Give this article a read: 
>> http://www.shapeblue.com/understanding-cloudstacks-physical-networking-architecture/
>>
>> You bascially need to determine which vlans you want to host which traffic. 
>> Then determine which NICs are connected to which VLANS for all your servers 
>> (CS, NFS, Hypervisor). You will need to name the vSwitches in XenServer.
>>
>> Sorry I cannot be more helpful.
>>
>> Taylor
>>
>> 
>>
>> From: Luis 
>> <lmartinez...@yahoo.com.INVALID<mailto:lmartinez...@yahoo.com.INVALID>>
>> Sent: Wednesday, July 19, 2017 12:28 PM
>> To: Users
>> Subject: CloudStack with advance networking
>>
>> Hi
>> I am trying to set up CS with advance networking. This is what i have
>> 1 server with CS Manager1 XenServer as node1 FreeNas for the primary and 
>> secondary1 Cisco 3560
>> My question is, how do i have to create the vlans, is there any special 
>> configuration? Can some one give me an example?
>> Thank you.
>>
> 
> 
>
> 

Re: CloudStack with advance networking

[ilya]

I looked over the guide - for current version - this guide is overly
complex - it was also written 4 years ago.

As of 4.3+, all of the things you need can be handled in the UI.

It should take about 10 minutes to get the zone up - assuming you have
your environment properly configured.

I will make a refreshed simplified version of this tutorial when time
allows.

On 7/19/17 1:16 PM, Taylor wrote:
> I am using it in a proof of concept to test out how quickly we could adopt it 
> in production.
> 
> 
> I dont know the answer to your question at this point, I am too new to the 
> stack. But I am currently reading articles like this one: 
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+Advanced+Network+Tutorial+-+Step+by+Step
> 
> Good luck!
> 
> Can anyone else offer advice?
> 
> 
> From: Luis 
> Sent: Wednesday, July 19, 2017 3:57 PM
> To: tschnei...@live.com
> Subject: Re: CloudStack with advance networking
> 
> Thank you for your replay, do you use it for production or personnal use?
> 
> I have a diagram but i dont know if the vlans need some special co figuration 
> like routing.
> 
> Sent from Yahoo Mail on 
> Android
> 
> On Wed, Jul 19, 2017 at 1:02 PM, Taylor
>  wrote:
> 
> I have not tried the advanced install yet but I am preparing to.
> 
> Give this article a read: 
> http://www.shapeblue.com/understanding-cloudstacks-physical-networking-architecture/
> 
> You bascially need to determine which vlans you want to host which traffic. 
> Then determine which NICs are connected to which VLANS for all your servers 
> (CS, NFS, Hypervisor). You will need to name the vSwitches in XenServer.
> 
> Sorry I cannot be more helpful.
> 
> Taylor
> 
> 
> 
> From: Luis 
> >
> Sent: Wednesday, July 19, 2017 12:28 PM
> To: Users
> Subject: CloudStack with advance networking
> 
> Hi
> I am trying to set up CS with advance networking. This is what i have
> 1 server with CS Manager1 XenServer as node1 FreeNas for the primary and 
> secondary1 Cisco 3560
> My question is, how do i have to create the vlans, is there any special 
> configuration? Can some one give me an example?
> Thank you.
> 

[README][Quarterly Call] - CloudStack Development, Blockers and Community Efforts

[ilya]

Hi Devs and Users

Hope this message finds you well,

As mentioned earlier, we would like to start with quarterly calls to
discuss the direction of cloudstack project.

I propose to split the 90 minute call into 3 topics:

1) Development efforts - 60 minutes
Upcoming Features you are working on developing (to avoid
collision andmaintain the roadmap).
  Depending on number of topics we need to discuss - time for
each topic will be set accordingly.
  If you would like to particiapate - please respond to this
thread and adhere to sample format below:  











   

2) Release Blockers - 20 minutes
  If you would like to participate - please respond to this
thread and adhere to sample format below:  






3) Community Efforts - 10+ minutes





The proposed date and time  - Thursday August 17th 9AM PT.

Minutes will be taken and posted on dev list. Due to number of things we
need to discuss - we have to keep the call very structured, each topic -
timed and very high level.
If there are issues and or suggestions, we will note it down in few
sentences, identify interested parties and have them do a "post"
discussion on the mailing list.

Looking forward to your comments,

Regards,
ilya

Re: KVM+HA

[ilya musayev]

Apology for fragmented messages, in existing framework cloudstack does not
know for certain if your VMs are dead, or KVM hypervisor crashed, or its
just a network blip, or perhaps you stopped kvm agent (or agent died). It
takes a conservative approach and does not re-start the VMs on other
hypervisors to avoid split brain scenario.

The only time it will restart KVM hypervisor and move VMs over - is when
you loose a primary storage access to one of the hypervisors in the cluster
- using NFS heartbeat method i mentioned earlier.

New framework addresses the limitations above by
1) checking if there is any disk activity on VMs that are in uncertain
state - if no activity for ALL VMs for "x" number of seconds
2) cloudstack will issue IPMI fence command to power down/reboot a host
(via ILO or DRAC or something else similar)
3) the VMs will be restarted elsewhere

Regards
ilya

On Tue, Jul 18, 2017 at 6:10 AM, ilya musayev <ilya.mailing.li...@gmail.com>
wrote:

> What share primary storage backend do you have for your VMs?
>
> If it is NFS - cloudstack agent writes heartbeat. When issue occurs - the
> neighbor hosts will check if the hypervisor that failed - still writes to
> heartbeat file. There are bunch of corner case where cloudstack HA does not
> kick in - due to uncertainty.
>
> The new framework should address those uncertainties.
>
> KVM HA with IPMI Fencing - Apache Cloudstack - Apache Software ...
> <https://www.google.com/url?sa=t=j==s=web=1=rja=8=0ahUKEwi59uv58pLVAhXHslQKHSU_B5YQFgg2MAA=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FCLOUDSTACK%2FKVM%2BHA%2Bwith%2BIPMI%2BFencing=AFQjCNG_-JHCYhcZm0lM9M4gKM4vKQ3hew>
> [CLOUDSTACK-8943] KVM HA is broken, let's fix it - ASF JIRA
> <https://www.google.com/url?sa=t=j==s=web=2=rja=8=0ahUKEwi59uv58pLVAhXHslQKHSU_B5YQFgg9MAE=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCLOUDSTACK-8943=AFQjCNGkOyC0hR4otCJ1LZF4j-2HSayMyQ>
>
> Regards
> ilya
>
> On Tue, Jul 18, 2017 at 6:06 AM, ilya musayev <
> ilya.mailing.li...@gmail.com> wrote:
>
>> Hi Victor
>>
>> We recently rewrote KVM HA framework. Its being merged into latest build.
>>
>>
>> On Tue, Jul 18, 2017 at 5:39 AM, victor <vic...@ihnetworks.com> wrote:
>>
>>> Hello Guys,
>>>
>>> I am facing the same issue that mentioned in the following url .
>>>
>>> -
>>>
>>> https://issues.apache.org/jira/browse/CLOUDSTACK-3535
>>>
>>> -
>>>
>>> When the host is put in maintenance mode , then ha enabled VM's are
>>> automatically migrated to available host. But when the kvm host is down, no
>>> HA is done. The vm's are still down until I put the host node back up.
>>>
>>>
>>> I have tried everything like the following.
>>>
>>> =
>>>
>>> 1, system VM's  and client vm's are created in shared storage
>>>
>>> 3, Added ha.tag host tags
>>>
>>> 2, Created host by adding ha tag
>>>
>>> 3, Created VE's  in Ha enabled host with ha enabled service offering
>>>
>>> 
>>>
>>> Do you guys have successfully tested Ha. I am really stuck at this part.
>>>
>>> Regards
>>>
>>>
>>>
>>>
>>
>

Re: KVM+HA

[ilya musayev]

What share primary storage backend do you have for your VMs?

If it is NFS - cloudstack agent writes heartbeat. When issue occurs - the
neighbor hosts will check if the hypervisor that failed - still writes to
heartbeat file. There are bunch of corner case where cloudstack HA does not
kick in - due to uncertainty.

The new framework should address those uncertainties.

KVM HA with IPMI Fencing - Apache Cloudstack - Apache Software ...
<https://www.google.com/url?sa=t=j==s=web=1=rja=8=0ahUKEwi59uv58pLVAhXHslQKHSU_B5YQFgg2MAA=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FCLOUDSTACK%2FKVM%2BHA%2Bwith%2BIPMI%2BFencing=AFQjCNG_-JHCYhcZm0lM9M4gKM4vKQ3hew>
[CLOUDSTACK-8943] KVM HA is broken, let's fix it - ASF JIRA
<https://www.google.com/url?sa=t=j==s=web=2=rja=8=0ahUKEwi59uv58pLVAhXHslQKHSU_B5YQFgg9MAE=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCLOUDSTACK-8943=AFQjCNGkOyC0hR4otCJ1LZF4j-2HSayMyQ>

Regards
ilya

On Tue, Jul 18, 2017 at 6:06 AM, ilya musayev <ilya.mailing.li...@gmail.com>
wrote:

> Hi Victor
>
> We recently rewrote KVM HA framework. Its being merged into latest build.
>
>
> On Tue, Jul 18, 2017 at 5:39 AM, victor <vic...@ihnetworks.com> wrote:
>
>> Hello Guys,
>>
>> I am facing the same issue that mentioned in the following url .
>>
>> -
>>
>> https://issues.apache.org/jira/browse/CLOUDSTACK-3535
>>
>> -
>>
>> When the host is put in maintenance mode , then ha enabled VM's are
>> automatically migrated to available host. But when the kvm host is down, no
>> HA is done. The vm's are still down until I put the host node back up.
>>
>>
>> I have tried everything like the following.
>>
>> =
>>
>> 1, system VM's  and client vm's are created in shared storage
>>
>> 3, Added ha.tag host tags
>>
>> 2, Created host by adding ha tag
>>
>> 3, Created VE's  in Ha enabled host with ha enabled service offering
>>
>> 
>>
>> Do you guys have successfully tested Ha. I am really stuck at this part.
>>
>> Regards
>>
>>
>>
>>
>

Re: KVM+HA

[ilya musayev]

Hi Victor

We recently rewrote KVM HA framework. Its being merged into latest build.


On Tue, Jul 18, 2017 at 5:39 AM, victor  wrote:

> Hello Guys,
>
> I am facing the same issue that mentioned in the following url .
>
> -
>
> https://issues.apache.org/jira/browse/CLOUDSTACK-3535
>
> -
>
> When the host is put in maintenance mode , then ha enabled VM's are
> automatically migrated to available host. But when the kvm host is down, no
> HA is done. The vm's are still down until I put the host node back up.
>
>
> I have tried everything like the following.
>
> =
>
> 1, system VM's  and client vm's are created in shared storage
>
> 3, Added ha.tag host tags
>
> 2, Created host by adding ha tag
>
> 3, Created VE's  in Ha enabled host with ha enabled service offering
>
> 
>
> Do you guys have successfully tested Ha. I am really stuck at this part.
>
> Regards
>
>
>
>

[NOTICE] Meeting with Accelerite Leadership

[ilya musayev]

Dear CloudStackers,

Last week, Johh Kinsella and myself were suppose to meet with Accelerite
leadership team. Unfortunately John could not make it - so i was alone.

We discussed ways we can improve community collaboration and leverage
Accelerite"s resources to align and drive larger community agenda including
extendes roadmap.

Many topics have been mentioned, below is the summary of our discussion. I
will list things in the order i see being important.


---
1) Proposal was made to have a quarterly call (or more often as needed)
with all interested parties to discuss:
Upcoming Features you are working on developing (to avoid collision
and
maintain the roadmap)
Blockers that are impacting release and adoption
Other topics

The length of the call would be 90 minutes. Each party will get a
fair
amount of time. The agenda will be collated and presented prior to the
call with a link to FS on Confluence and time allotted for each topic.

Minutes will be taken and posted on dev list. If there are issues and or
suggestions, we will note it down in few sentences, identify interested
parties and have them do a "post" discussion on the mailing list.

The proposed date and time  - Thursday August 17th 9AM PT

--
2) Accelerite is considering funding a position for a person who will be
working within community - as community manager. Help organize and
facilitate discussions, make sure Confluence and JIRA are up to date,
help new users with answering basic questions or finding right
individual to assist with solution. While funded by Accelerite - it must
be clear that the person is working with/for Apache CloudStack project.

3) Marketing was mentioned, i suggested we do more press releases - and
possibly make use of interns

4) OpenStack VS CloudStack (unbiased technology comparison), there is a
common question - we need to come up something that can help justify
Apache CloudStack to clients leadership

5) Cinder integration with Cloudstack was mentioned - but no solid plans
yet.

6) Creating Appliances of CloudStack - that are ready to be consumed and
user can spin nested VMs to try CloudStack effortlessly

7) CoudStack Template Repository (plugin)- there is a code written for it by
Citrix and resides on ASF git - but for some reason it was dropped or
never completed. If we can give user a rich marketplace of appliances to
consume - we will certainly get a good edge. This can improve the adoption.

8) MeetUps - we need to re-kickstart this initiative within SF Bay Area
and stream it to other locations/meetups.

9) Demo environment of CloudStack.  David mentioned Citrix donated gear
is in one of ASF locations - but sitting idle. I proposed we make use of
it and let new CloudStack explorers try it out - without the hassle of
deploying it.

10) If we can get CloudStack into EPEL fedora and ubuntu upstream
repositories - it will help with adoption as well.

Please let me know if you would be interested in item #1, which is
quarterly meeting. The proposed time is 9am PST, August 17th.

I will help setting up the first few initial calls and be a moderator.

Looking forward to your comments

Regards
ilya

[JOB OPPORTUNITY] LeaseWeb is looking for CloudStack Developer

[ilya]

Hi Folks,

Promised to help LeaseWeb recruiter with posting Job to "dev" and "user"
list - apology for cross posting.

Please reach out to recruiter directly, job description can be seen here:

https://drive.google.com/open?id=0B06G3DVBuP9zQXRwMzVKZTJOVlU

Recruiter for this position can be reached here:
https://www.linkedin.com/in/darwinbpoveda/

Regards
ilya

Re: some issues after upgrade cloudstack to 4.9.2

[ilya]

Please see comments in-line

On 5/25/17 2:14 AM, Marc Poll Garcia wrote:
> Hi all,
> 
> we have just upgraded our cloud environment based on Cloudstack 4.5.2 to
> 4.9.2 and we're expriencing some issues after this.
> 
> Our setup is the following one:
> 
> - 1 x cloudstack managment server
> - 1 x bbdd server cloudstack database on it
> - 2 x Vmware Hipervisors (hosts)
> 
> I'm performing a list of tests:
> 
> *- Sometimes, and randomly console from instances does not load.*
> *- Not possible to upload template from local.*
> 
> We see the following on log:
> 
> 2017-05-25 09:00:31,665 ERROR [c.c.s.ImageStoreUploadMonitorImpl]
> (Upload-Monitor-1:ctx-0b3bf6e9) (logid:e9c82a0f) *Template
> b87459ac-8fbe-4b34-ae25-21235c3fcd1d failed to upload due to operation
> timed out*
We need more info on this, try

grep ctx-0b3bf6e9 -A5 -B5 cloudstack-management.log

> 2017-05-25 09:02:18,265 ERROR [c.c.c.ClusterServiceServletContainer]
> (Thread-11:null) (logid:) *Unexpected exception *
Is there more for this?


> 2017-05-25 09:03:35,940 ERROR [c.c.c.ClusterManagerImpl] (main:null)
> (logid:) *Unable to ping management server at 192.168.100.2:9090
> <http://192.168.100.2:9090> due to ConnectException*

Safe to ignore.


Here is a suggestion, stop your MGMT server, then empty the log file
(make a backup of it prior if needed)
cat /dev/null > cloudstack-management.log
start cloudstack management, give it few minutes to load - run
operations that fail - post the management logs to pastebin or google
drive..


Also - what version of tomcat are you running?

CentOS 6 and 7 come with an older version of tomcat that has an issue
with socket timeouts, you can read it here:
https://issues.apache.org/jira/browse/CLOUDSTACK-7907


as to console not working - make sure console proxy VM and cloudstack MS
can access the ESXi and vCenter or proper ports, like 443, 901 and
higher VNC ports.

To find out why it does not work sometimes, login to console proxy VM,
and via netstart check for any connections in TIME_WAIT state. That will
tell you if there are network issues.

Regards
ilya



> 
> Why is it happening?
> It does not happen on our old 4.5.2 version.
> 
> Is there any way to fix it? changing any global parameter or permissions
> issue?
> 
> We need a clue with that because if affecting to our production environment.
> 
> Thanks in advance.
> 
> Kind regards.
> 

Re: SSVM NIO SSL Handshake error

[ilya]

Just wanted to make sure you are

1) running java 1.7 on management server
2) 192.168.12.1 is actual cloudstack management server and not gateway

Check the permission on your keystore and make sure cloud user can
access it.


Regards
ilya

On 5/23/17 5:11 AM, Jason Kinsella wrote:
> 2017-05-23 11:58:22,461 INFO  [utils.nio.NioClient] (main:null) Connecting to 
> 192.168.12.1:8250

Re: OVS Plugin

[ilya]

Hi Simon

Would you mind expanding a little more on your setup?

Specifically what is being used underneath.

thanks
ilya

On 4/14/17 9:11 AM, Simon Weller wrote:
> I'd strongly suggest you consider using the native VXLAN support for KVM. It 
> works extremely well and we run it in production.
> 
> 
> - Si
> 
> 
> 
> 
> 
> From: Dag Sonstebo <dag.sonst...@shapeblue.com>
> Sent: Friday, April 14, 2017 10:57 AM
> To: users@cloudstack.apache.org
> Subject: Re: OVS Plugin
> 
> Hi Imran,
> 
> OVS is the same as GRE tunnelling, which you will have as an isolation method 
> for guest networking – see 
> http://docs.cloudstack.apache.org/en/latest/networking/ovs-plugin.html.
> 
> Please let us know how you get on – especially how your hypervisor nodes cope 
> with CPU load once the GRE tunnels start growing in numbers (historically 
> this has not scaled well).
> 
> Regards,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
> 
> On 13/04/2017, 21:37, "Imran Ahmed" <im...@eaxiom.net> wrote:
> 
> Dear Team,
> 
> I have setup cloudstack 4.9 with KVM hypervisor and advanced networking on
> CentOS7. Also we installed and setup openvswitch on the hypervisors (KVM)
> hosts.
> 
> Below are traffic labels for kvm
> 
> Cloudbr0  for management
> Cloudbr1 for guest
> Cloudbr2 for public
> 
> After configuring the zone, pod , cluster, host, primary and secondary
> storages we wanted to enable the OVS plugin under service providers for 
> the
> guest network.
> 
> However OVS is not shown in the list.
> 
> Please suggest what could be wrong here.
> 
> Kind regards,
> 
> Imran
> 
> 
> 
> 
> dag.sonst...@shapeblue.com
> www.shapeblue.com<http://www.shapeblue.com>
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
> 
> 
> 
> 

Re: issue downloading a copy template to different zone

[ilya]

Marco,

I've seen this b4 but never really bothered as to why it does not work..

Thanks for figuring out the solution

Would you please file a JIRA issue so we can fix this in the future.

https://issues.apache.org/jira/browse/CLOUDSTACK

Regards
ilya

On 4/12/17 10:15 AM, Marco Giovannini wrote:
> Hi guys,
> 
> 
> I'm using the cloustack 4.6.2 and I'm hitting the following issue.
> 
> I created a new  secondary storage:
> 
> add imagestore zoneid=--xxx-xxx name=_Secondary_SATA1
> provider=NFS url=nfs://ix/vol/x_Secondary_SATA1/ESX
> 
> 
> When I create a new template which end up on this SS and I try to download
> I get an error /userdata/x   access denied.
> 
> 
> After investigating I found out the issue.
> 
> 
> During the SS creation the field parent in the image_store table is not
> initialized leaving the default value to NULL.
> 
> 
> This value is used by the SSVM  in the path of the  link to download the
> template.
> 
> 
> Broken link:
> 
> lrwxrwxrwx 1 root root  84 Apr  5 09:06
> e55f5761-2dab-4095-b426-fe72c296150e.ova ->
> /mnt/SecStorage/null/template/tmpl/875/1603/8ce602e5-e920-335a-bb24-b9595722a49f.ova
> 
> 
> Working link:
> 
> lrwxrwxrwx 1 root root 116 Jan 23 11:01
> e64820a2-dc45-48f6-b17e-bdeb6d47e650.ova ->
> /mnt/SecStorage/5597b498-9507-3df2-ad46-ca9da35c7f7d/template/tmpl/173/1516/7f07df49-a26f-3833-85da-68c89c97daa
> 
> 
> 
> This can be fixed by updating the table manually and fixing the broken link
> created so far but it will reappear again when as new SS will be created.
> 
> 
> Does anyone have this issue or know if it is fixed an next releases ?
> 
> 
> I didn't find anyone reporting it so far.
> 
> 
> 
> Regards,
> Marco
> 

Re: Miami and conference

[ilya]

Try reaching out to PMC or Dev list..

On 4/7/17 9:00 AM, Engelmann Florian wrote:
> Hi all,
> 
> is there any official number of registered attendees? How many are expected? 
> Any number from the last conferences?
> 
> All the best,
> Florian
> 

Re: Using CentOS-6.x on KVM-hosts - what are the threats?

[ilya]

Vladimir

It comes down to version of qemu and cpu flags. As you know this is not
related to cloudstack.

Because of major differences in qemu and cpu flags -  safest method is
power down and power up.

QEMUs claim to fame was that it can migrate from AMD to Intel and vice
versa. In practice its hit or miss.

What is guest.cpu.mode set to in your agent.properties?

Also, this maybe a good help for "guest.cpu.mode" capabilities.

https://libvirt.org/formatdomain.html#elementsCPU

Regards
ilya

On 2/25/17 2:29 AM, Vladimir Melnik wrote:
> Thank you for the comment, Simon! The most funny thing is that I've added 3
> new hosts to my infrastructure in December, their hardware is awesome, so
> their hardware refresh is not a matter of the nearest future. :) Anyhow, 6.x
> works great and the only thing I regret is lacking certain features (such as
> IOpS limits).
> 
>  
> 
> I'm also maintaining a cluster of 5 hosts (primary storages aren't local,
> they're connected via GlusterFS & NFS) which've been running 6.x too. I've
> upgraded 3 of hosts from 6.x to 7.x, but when I'm trying to migrate a VM
> from the "old" hosts to the "new" ones, the migration is being timed out and
> the VM is being frozen in the "paused" state. I noticed a difference in the
> CPU-flags set: all the hosts running 7.x have the "nopl" flag, but the hosts
> running 6.x don't. This option appears only after installing 7.x and maybe
> this is the cause. Does anyone have any suggestions on the reason that
> causes freezing the VMs when they've been migrating from the 6.x-powered
> hosts to the 7.x-powered ones? Is that the "nopl" flag? Is that anything
> else? Thanks to all!
> 
> 

Re: Introducing Vishwas

[ilya]

Welcome!

On 2/24/17 1:08 AM, Vishwas Pathak wrote:
> Hello CloudStack team,
> 
> My name is Vishwas Pathak and I am working with Accelerite CloudPlatform 
> testing team. Earlier I have worked in storage systems, cloud computing, 
> virtualization and telecom messaging domains. Excited to be part of this 
> community and happy to contribute.
> 
> Currently I am working on testing the Accelerite CloudPlatform product. So 
> far have worked on Xen server and VMWare based environments. Did some testing 
> for KVM as well. 
> 
> Looking forward to work with all of you.
> 
> 
> Regards,
> Vishwas Pathak
> www.accelerite.com
> 
> 
> 
> 
> 
> 
> DISCLAIMER
> ==
> This e-mail may contain privileged and confidential information which is the 
> property of Accelerite, a Persistent Systems business. It is intended only 
> for the use of the individual or entity to which it is addressed. If you are 
> not the intended recipient, you are not authorized to read, retain, copy, 
> print, distribute or use this message. If you have received this 
> communication in error, please notify the sender and delete all copies of 
> this message. Accelerite, a Persistent Systems business does not accept any 
> liability for virus infected mails.
> 

Re: Intel v3 and v4 CPUs in the same cluster

[ilya]

https://media.readthedocs.org/pdf/cloudstack-installation/4.8/cloudstack-installation.pdf

Page 76 explains it.. you want for CPU instructions sets to match and be
consistent if you plan to run v3 and v4 cpus in the same cluster.

Regards
ilya

On 2/10/17 8:23 AM, Rafael Weingärtner wrote:
> From my experience, yes it would work.
> 
> At the end what matter are the CPU capabilities, meaning the instructions
> tha can be used by operating systems (O.S) and their processes, in this
> case, VMs can be considered processes in the hypervisor (hypervisors are
> OS). We normally do the masking to hide features that may only be available
> in one host or other; when you migrate a VM of a host (without masking), if
> the VM ends up in a host that does not have the same CPU features it may
> crash or present unexpected behaviors.
> 
> On Fri, Feb 10, 2017 at 11:15 AM, Nando Beifiori <nandobeifi...@gmail.com>
> wrote:
> 
>> Hi Rafael,
>>
>> in this case the hypervisor is KVM and I think it supports masking, but let
>> me check if I got it right:
>>
>> if the CPU is not exactly the same the features can be masked using
>> hypervisor's capabilities.
>>
>> Does it mean that if the CPU has the exact same features but different
>> frequency for example it would work?
>>
>> Thanks
>>
>> On 10 February 2017 at 15:46, Rafael Weingärtner <
>> rafaelweingart...@gmail.com> wrote:
>>
>>> It depends on your hypervisor.
>>> For XenServer, you should check the CPU feature you have and if the CPU
>>> supports masking [1].
>>>
>>> xe host-cpu-info
>>>>
>>>
>>> Look for the "features" information, and then check if your hardware guy
>>> can provide a processor with similar features or one that supports
>> masking
>>> the same features as your current servers' CPUs.
>>>
>>> [1] https://support.citrix.com/article/CTX127059
>>>
>>> On Fri, Feb 10, 2017 at 10:28 AM, Nando Beifiori <
>> nandobeifi...@gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> quick but very important question: I have a cluster with Intel Xeon
>>> E5-2667
>>>> v3 CPUs and I need to add another host, but my provider has replaced
>> the
>>> v3
>>>> with the v4 of the same CPU.
>>>>
>>>> The differences are:
>>>>
>>>>- Lithography (22nm > 14nm)
>>>>- Cache size (20MB > 25MB)
>>>>- Max Memory Bandwidth (68 GB/s > 76.8 GB/s)
>>>>- Intel TSX-NI feature which is present in v4 but not in the v3.
>>>>
>>>> Frequency, turbo frequency and cores are exactly the same.
>>>> Would it be possible to add the new host to the same cluster or should
>> I
>>>> create a new cluster or maybe look for a provider that still has the v3
>>>> CPU?
>>>>
>>>> Cloudstack version is 4.9 and hosts use Centos 7.2.
>>>>
>>>> Thanks in advance
>>>>
>>>
>>>
>>>
>>> --
>>> Rafael Weingärtner
>>>
>>
> 
> 
> 

Re: Isolated Network Vlan Tag

[ilya]

Hi Luis

If no one has responded to your inquiry - then yes, just post a message
and someone will get back to you.

I'd suggest you use markmail.org and search existing cloudstack message
threads for questions you might have - as others may have asked the same
question in past.

Regards,
ilya

On 12/10/16 5:21 PM, Luis M wrote:
> Hi forum.
> 
> How does this works, I just have to send an email with my question?
> 
> 
> 
>> On Dec 10, 2016, at 4:28 PM, Gian Paolo Buono <gianpaolo.bu...@gesca.it> 
>> wrote:
>>
>> Hi all,
>>
>> Can I add an isolated network setting the tag (ex: 500) ?
>>
>> Thanks
> 

Re: Router VM: patchviasocket.py timeout issue on 1 out of 4 networks

[ilya]

This will explain a bit more on how this issue came about and how to fix
it..
https://www.mail-archive.com/dev@cloudstack.apache.org/msg71559.html

On 12/12/16 6:31 PM, Simon Weller wrote:
> Can you turn on agent debug mode and take a look at the debug level logs?
> 
> 
> You can do that by running sed -i 's/INFO/DEBUG/g' 
> /etc/cloudstack/agent/log4j-cloud.xml on the host and then restarting the 
> agent.
> 
> 
> - Si
> 
> 
> 
> 
> 
> From: Syahrul Sazli Shaharir 
> Sent: Monday, December 12, 2016 8:21 PM
> To: users@cloudstack.apache.org
> Subject: Router VM: patchviasocket.py timeout issue on 1 out of 4 networks
> 
> Hi,
> 
> I am running latest Cloudstack 4.9.0.1 on CentOS 7 KVM + ceph
> environment. After running for some time, I faced with an issue with
> one out of 4 networks - following a heartbeat-induced reset on all
> hosts, the associated virtual router would not get recreated and
> started properly on any of the 3 hosts I have, even after repeated
> attempts of the following:-
> - destroy-recreate cycles, via Cloudstack UI
> - restartNetwork cleanup=true API calls (failed with errorcode = 530).
> - redownload and reregister system VM template as another entry and
> assign to router VM in global setting (boots the new template OK, but
> still same problem)
> - tweak default system offering for router VM (increased RAM from 256 to 
> 512MB)
> - created new system offering, with RAM tweak, and use of ceph rbd
> store, and assigned it to Cloud.Com-SoftwareRouter as per docs - which
> didnt work for some reason: it kept on using initial default offering
> and created image on local host storage
> - upgrade to latest cloudstack (previously was running 4.8)
> 
> As with a handful of others in this list archives, virsh list and
> dumpxml shows the VM created OK but failed soon after booting, as
> found in the following error in agent.log :-
> 
> 2016-12-13 10:03:33,894 WARN  [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:null) (logid:633e6e03) Timed out:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py
> -n r-668-VM -p 
> %template=domP%name=r-668-VM%eth0ip=10.3.28.10%eth0mask=255.255.255.0%gateway=10.3.28.1%domain=nocser.net%cidrsize=24%dhcprange=10.3.28.1%eth1ip=169.254.0.33%eth1mask=255.255.0.0%type=dhcpsrvr%disable_rp_filter=true%dns1=8.8.8.8%dns2=8.8.4.4%ip6dns1=%ip6dns2=%baremetalnotificationsecuritykey=uavJByNGGjNLrELG-qbdN99__1I3tnp8qa0KbcsKokKJcPB43K9s6oQu2nMLqo3YP8p6jqDy5XT3WWOWBA2yNw%baremetalnotificationapikey=8JH4mdkxsEMhgIBgMonkNXAEKjVOeZnG1m5UVekvvo4v_iXQ4ZS7rh6NNS0qphhc7ZrCauiz23tp2-Wa3AASlg%host=10.2.30.11%port=8080
> .  Output is:
> .
> 2016-12-13 10:05:45,895 WARN  [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:null) (logid:633e6e03) Timed out:
> /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
> vr_cfg.sh 169.254.0.33 -c
> /var/cache/cloud/VR-48ea8a95-6c02-499f-88d3-eae5bf9f9fbe.cfg .  Output
> is:
> 
> As mentioned, this only happens with 1 network (always the same
> network). The other router VMs work OK. Any clues on how to
> troubleshoot this further, would be greatly appreciated.
> 
> Thanks.
> 
> --
> --sazli
> Syahrul Sazli Shaharir 
> 

Re: Good backup solutions for Cloudstack

[ilya]

Marty

If you mean disk names being cryptic - you are correct.

Speaking of NetApp NFS level snapshot backups, it creates a .snapshot
directory with structure for hourly, daily, weekly, monthly, etc..

I had many occasions where a user would mistakenly delete VMs and i had
to reverse the deletion - which was a royal pain but eventually worked.

Regards
ilya

On 11/7/16 3:26 PM, Marty Godsey wrote:
> The only problem with this is if you are dong NFS, the VHDs are named very 
> cryptic so you don't have an idea which VM is which.
> 
> Regards,
> Marty Godsey
> 
> -Original Message-
> From: ilya [mailto:ilya.mailing.li...@gmail.com] 
> Sent: Monday, November 7, 2016 12:25 PM
> To: users@cloudstack.apache.org
> Subject: Re: Good backup solutions for Cloudstack
> 
> Consider using SAN/NAS level snapshots.
> 
> On 11/3/16 9:12 AM, a...@globalchangemusic.org wrote:
>>  
>>
>> How about KVM? 
>>
>> On 2016-11-02 16:47, Sergey Levitskiy wrote: 
>>
>>> Veeam works OK for VMware based implementations. You can tag VMs and based 
>>> on vsphere tag Veeam will automatically pick them up for the backup 
>>> processing.
>>>
>>> On 11/2/16, 4:21 PM, "Asai" <a...@globalchangemusic.org> wrote:
>>>
>>> Hello,
>>>
>>> Can anyone recommend a good backup solution for a Cloudstack deployment? 
>>> What's the best way of backing up VMs and snapshots? I have experience with 
>>> XenServer, but I'm moving into a CS deployment now and am looking for 
>>> recommendations on best practices.
>>>
>>> Thanks
>>> Asai
>>> Network and Systems Administrator
>>> GLOBAL CHANGE MEDIA
>>> http://globalchange.media [1]
>>> Tucson, AZ
>>  
>>
>> Links:
>> --
>> [1] http://globalchange.media
>>

Re: Good backup solutions for Cloudstack

[ilya]

Consider using SAN/NAS level snapshots.

On 11/3/16 9:12 AM, a...@globalchangemusic.org wrote:
>  
> 
> How about KVM? 
> 
> On 2016-11-02 16:47, Sergey Levitskiy wrote: 
> 
>> Veeam works OK for VMware based implementations. You can tag VMs and based 
>> on vsphere tag Veeam will automatically pick them up for the backup 
>> processing.
>>
>> On 11/2/16, 4:21 PM, "Asai"  wrote:
>>
>> Hello,
>>
>> Can anyone recommend a good backup solution for a Cloudstack deployment? 
>> What's the best way of backing up VMs and snapshots? I have experience with 
>> XenServer, but I'm moving into a CS deployment now and am looking for 
>> recommendations on best practices.
>>
>> Thanks
>> Asai
>> Network and Systems Administrator
>> GLOBAL CHANGE MEDIA
>> http://globalchange.media [1]
>> Tucson, AZ
>  
> 
> Links:
> --
> [1] http://globalchange.media
> 

Re: CloudStack Max Number of Hypervisors

[ilya]

There is no cloudstack zone limit i'm aware off.

The zone itself has no limit.

Your limits will be defined by back-end storage and network topology.

If you offer Physical VLANs segregation, then you have to be mindful of
Spanning Tree issues, in that case in my experience, you may want to
limit a zone to 200 nodes and less than 300 VLANs.

If you are running SDN, then this limit no longer applies and you can
span with ease beyond 200 nodes per zone.

As for CloudStack ability to manage many nodes, there are known
implementations with close to 1000 hypervisors on 2 pair cloudstack
setup - and its not the final limit.

With that said, you will have to tweak DB, Java and CloudStack, as
default configs - wont scale.

Perhaps explain what you are trying to accomplish and how your
environment is laid out?



On 10/26/16 12:22 AM, Dag Sonstebo wrote:
> Hi Tyler,
> 
> Ultimately the number of hypervisors a management server can manage depends 
> on the workload it handles – i.e. number of users, number of VMs, how complex 
> configurations your users use etc. As a rule of thumb you should always have 
> two management servers for redundancy – but we do see customers run hundreds 
> of hypervisors with this setup. On top of this keep in mind the management 
> server is stateless – in other words it is easy to just build additional 
> servers once your load increases – given that your load balancer 
> configuration is healthy.
> 
> The same redundancy point goes for your backed MySQL services – you should 
> always run at least a slave + master setup.
> 
> Hope this helps,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
> 
> On 26/10/2016, 04:52, "Tyler Wilson"  wrote:
> 
> Hello All,
> 
> Are there any known issues with scaling issues with the controller
> services? How many hypervisors should I be able to deploy to a zone before
> running into issues and having to start a new zone?
> 
> Thanks for any info!
> 
> 
> 
> dag.sonst...@shapeblue.com 
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>   
>  
> 

Re: Reusing ISO from Secondary Storage

[ilya]

Mount your secondary store NFS.

cd /mnt/my-secondary-nfs-share
find . | grep -i iso


On 10/26/16 11:58 AM, Asai wrote:
> Greetings,
> 
> Noob question.  I had an instance of Cloudstack up and running which I 
> scrapped and started over.  In that instance I had downloaded and registered 
> an ISO which is now in secondary storage.  In my new instance, I would like 
> to reuse that ISO but CloudStack is not seeing it.  How do I retrieve an ISO 
> from secondary storage for reuse?
> Asai
> Network and Systems Administrator
> GLOBAL CHANGE MEDIA
> office: 520.398.2542
> http://globalchange.media
> Tucson, AZ
> 
> 

Re: configdrive in ACS

[ilya]

I guess a known fact, but most people behind RacherOS were original
authors of CloudStack. They should have no issues integrating with
CloudStack - assuming you are Rancher customer - you can ask them to
extend support for it.



On 9/12/16 4:38 AM, Helge Waastad wrote:
> Hi,
> and thanks for answering.
> 
> Yeah, I've read the docs and I've seen the cloudstack cloud-init
> implementation in ubuntu cloud-image
> 
> So since configdrive is not supported I think I need to follow your
> advice and rather make a script fetching needed information from
> cloudstack instead and then start services needed in shell.
> 
> thx,
> 
> hw
> 
> 
> sø., 11.09.2016 kl. 15.30 +, skrev Sergey Levitskiy:
>> ACS uses a different system than configdrive. Userdata and metadata
>> is saved on a vrouter and can be queried from the instance with basic
>> HTTP call. It is very similar to AWS.
>> To save userdata you would use deployVirtualMachine or
>> updateVirtualMachine call
>> Use this guide for details how format encode it:
>> http://docs.cloudstack.apache.org/projects/cloudstack-administration/
>> en/4.8/virtual_machines/user-data.html
>>
>> Here it is an example how you would get VM ID and some random
>> property from metadata and userdata assuming userdata has
>> propertyname:”propertyvalue” style tag inside
>>
>> if [[ $OS == "RHEL" || $OS == "CentOS" ]]; then
>>   DHCPBASE="/var/lib/dhclient/dhclient-*.lease"
>> elif [ $OS == "Ubuntu" ]; then
>>   DHCPBASE="/var/lib/dhcp/dhclient.*.leases"
>> fi
>>
>> DHCPSERVER=`grep dhcp-server-identifier $DHCPBASE |tail -1|awk
>> '{print $3}'|sed 's/;//'`
>> IPADDRESS=`curl -s http://$DHCPSERVER/latest/local-ipv4`
>> PROVIDERID=`curl -s http://$DHCPSERVER/latest/vm-id|sed '/^i-.*/s/i-
>> [0-9]*-//'|sed 's/-VM$//'|sed '/[0-9a-f]\{8\}-[0-9a-f]\{4\}-[0-9a-
>> f]\{4\}-[0-9a-f]\{4\}-[0-9a-f]\{12\}/s/-[0-9a-f]\{4\}-[0-9a-f]\{4\}-
>> [0-9a-f]\{4\}-[0-9a-f]\{12\}$//'`
>>
>> PROPERTYVALUE=`curl -s http://$DHCPSERVER/latest/user-data|grep
>> propertyname|cut -d":" -f2|cut -d'"' -f2`
>>
>>
>>
>>
>>
>>
>> On 9/11/16, 3:36 AM, "Helge Waastad"  wrote:
>>
>> Hi,
>> Im testing out RancherOS in ACS and have a couple of issues.
>> 
>> First, it seems that RancherOS does not support cloudstack
>> datasource for meta/userdata but that I need to take with Rancher
>> guys.
>> 
>> But, in openstack I can always use configdrive to get userdata to
>> my vm.
>> 
>> Is it possible to use configdrive in acs? (i have'nt had any luck
>> yet)
>> 
>> Br hw
>> 
>> 
>> 
>> 
>> Sendt fra Galaxy Tab
>>
>>
>>

Re: Creating a SQL Server 2012 AlwaysOn Availability Group on a cloudstack

[ilya]

Kiril

Perhaps you can educate us on what MS SQL 2012 Cluster requirements are
when it comes to disks?

Is it like oracle rac and expects a shared disk that will be handled by
native clustering software?

Lastly, tell us about what cloudstack implementation you have running,
hypervisor and storage backend.

Regards
ilya

On 8/24/16 2:26 AM, Kiril Churilov wrote:
> Hi guys,
> 
> I'm trying to create  MS sql 2012 std. cluster with AlwaysOn Availability
> Group on a cloudstack platform.
> To build the sql cluster I need to setup  Windows Server 2012 two-node
> cluster first.
> How to organize shared storage for two-node  win. cluster on a cloudstack?
> Does any one build something similar before?
>  Windows Server Failover Clustering (WSFC) with SQL Server on a cloudstack?
> 
> Thank you.
> 
> Regards,
> 
> Kirils.
> 

Re: Cloudstack - volume migration between clusters (primary storage)

[ilya]

Not certain how Xen Storage Migration is implemented in 4.5.2

I'd suspect legacy mode would be

1) copy disks from primary store to secondary NFS
2) copy disks from secondary NFS to new primary store

it might be slow... but if you have enough space - it should work...

My understanding is that NFS is mounted directly on hypervisors. I'd ask
someone else to confirm though...

On 8/24/16 7:20 AM, cs user wrote:
> Hi All,
> 
> Xenserver 6.5, cloudstack 4.5.2. NFS primary storage volumes
> 
> Lets say I have 1 pod, with 2 clusters, each cluster has its own primary
> storage.
> 
> If I migrate a volume from one primary storage to the other one, using
> cloudstack, what aspect of the environment is responsible for this copy?
> 
> I'm trying to identify bottlenecks but I can't see what is responsible for
> this copying. Is it is the xen hosts themselves or the secondary storage vm?
> 
> Thanks!
> 

Re: KVM with vlanid: 4095

[ilya]

And interface config would look like this

root@rn2-gcs-np-pod14-lhv20:/etc/sysconfig/network-scripts# cat ifcfg-ens1f0

DEVICE=ens1f0
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
PEERDNS=no
BRIDGE=brens1f0-4095

It needs to be referenced in cloudstack under physical network - i
assume for guest networks - that will do passthroughs..

On 8/19/16 11:53 AM, Pierre-Luc Dion wrote:
> Hi,
> 
> I've found this new feature from 4.6.0 [1]; Did anyone make it worked? How
> can I setup a network that would do a 802.1q trunk passthrough ? Do I
> create a share network or a regular guest network?  I presume it's not
> working for simple network ?
> 
> I would like to use that capability on 4.9.0 + KVM on Centos 7.
> 
> [1] https://issues.apache.org/jira/browse/CLOUDSTACK-8252
> 
> 
> Thanks !
> 

Re: KVM with vlanid: 4095

[ilya]

We have this working in 4.5 with kvm.

You need to setup a bridge with 4095 in its name..


I looked up my notes and i was setting up like this

root@rn2-gcs-np-pod14-lhv20:/etc/sysconfig/network-scripts# cat
brens1f0-4095
DEVICE=brens1f0-4095
BOOTPROTO=none
ONBOOT=yes
TYPE=Bridge
DELAY=0

note "ens1f0" is the name of interface to be bridged.

On 8/19/16 11:53 AM, Pierre-Luc Dion wrote:
> Hi,
> 
> I've found this new feature from 4.6.0 [1]; Did anyone make it worked? How
> can I setup a network that would do a 802.1q trunk passthrough ? Do I
> create a share network or a regular guest network?  I presume it's not
> working for simple network ?
> 
> I would like to use that capability on 4.9.0 + KVM on Centos 7.
> 
> [1] https://issues.apache.org/jira/browse/CLOUDSTACK-8252
> 
> 
> Thanks !
> 

Re: Fresh 4.9 Install

[ilya]

Marty

i've tested 4.9 RC2 recently, i've noticed few minor UI glitches (but
nothing really major).

With that said, i could not see your screenshots as files attached to
this mailing list - can you upload them elsewhere?

I could be wrong, but i believe users mailings provided by ASF does not
honor attachments.

Personal preference, I use  a nifty app called "Jing" from TechSmith,
its freebie - and allows for direct online posting.

Regards,
ilya

On 8/12/16 7:33 PM, Marty Godsey wrote:
> Correction:
> 
>  
> 
> I also get an error when going to Storage
> 
>  
> 
>  
> 
> Its almost since nothing is configured, it gets unhappy.
> 
>  
> 
>  
> 
> Regards,
> 
> Marty Godsey
> 
>  
> 
> *From:* Marty Godsey [mailto:ma...@gonsource.com]
> *Sent:* Friday, August 12, 2016 10:32 PM
> *To:* users@cloudstack.apache.org
> *Subject:* Fresh 4.9 Install
> 
>  
> 
> Hello,
> 
>  
> 
> On a fresh 4.9 install I get the following error when I click on the
> dashboard:
> 
>  
> 
>  
> 
> At this time I have no zones or storage configured. I also do not get
> any other errors any where else.
> 
>  
> 
> My setup is as follow:
> 
>  
> 
> 1.   Two management servers running Ubuntu 14.04 (get error on both)
> 
> 2.   Separate database server running mysql
> 
>  
> 
> Again nowhere else shows an error.
> 
>  
> 
> Regards,
> 
> Marty Godsey
> 
>  
> 

Re: Guest networking issue

[ilya]

After you get a known functional template from http://openvm.eu, few
more pointers

Note that I dont run basic zone with SG in any envs, so this is purely
trial and error approach.

Can you inspect iptables on the KVM host for any rules that might block
communication? I would also try stopping the iptables on KVM host to see
if it resolves the communication issue.

IIRC, the routing is provided by router VM, if you cant get out - i'd
assume the issue could be there.


On 8/8/16 10:27 PM, Asanka Gunasekara wrote:
> Hi I desperately need some assistance,
> 
> Best Regards
> 
> *Asanka Gunasekara*
> 
> *Asst. Engineering Manager - Systems Infra Services*
> *Global IT Infrastructure Services Division*
> 
> | Informatics International Limited | 89/57 | Jampettah Lane | Colombo 13 |
> Sri Lanka |
> 
> | T: +94-115-794-942 (Dir)| F: +94-112-542-832 | M: +94-768209719 |
> 
> | asank...@informaticsint.com  |
> www.informaticsint.com |
> 
> 
> On 9 August 2016 at 10:22, Asanka Gunasekara 
> wrote:
> 
>> Hi Hope someone can help me on this
>>
>>  Below is my env
>>
>> a) CloudStack 4.9 on CentOs 7.2,
>> b) running on 2 compute nodes and one controller node
>> c) NFS as shared storage primary and secondary
>> d) Basic networking
>> f) Hosts, Guests, system VMs and NFS storage are in same IP range
>>
>> I can create VMs and from UI I can see Guests are getting IPs assigned
>>
>>  1. when the instance boots up it has no IP assigned (ip addr show
>> shows no IP). But I have to do dhclient for the guest to get configured.
>>
>>  2. Even though instance gets an IP after running dhclient, I am
>> unable to reach no further than the host that the instance are in. All the
>> routes are in place. I have created Security Group with allow all to this
>> guest still no luck
>>
>> please let me know if you need any additional information
>>
>> Best Regards
>>
>> *Asanka Gunasekara*
>>
>> *Asst. Engineering Manager - Systems Infra Services*
>> *Global IT Infrastructure Services Division*
>>
>> | Informatics International Limited | 89/57 | Jampettah Lane | Colombo 13
>> | Sri Lanka |
>>
>> | T: +94-115-794-942 (Dir)| F: +94-112-542-832 | M: +94-768209719 |
>>
>> | asank...@informaticsint.com  |
>> www.informaticsint.com |
>>
>>
> 

Re: Guest networking issue

[ilya]

You told us nothing about how CentOS 7.2 template was created.

Can you try to launch console and do
1) ifconfig
2) dhclient eth0 (or any other interface you see in step 1)

try to ping the gateway..

Lastly, ss this a known cloud ready template?

Can you try one of the common cloudstack templates from http://openvm.eu


On 8/8/16 10:27 PM, Asanka Gunasekara wrote:
> Hi I desperately need some assistance,
> 
> Best Regards
> 
> *Asanka Gunasekara*
> 
> *Asst. Engineering Manager - Systems Infra Services*
> *Global IT Infrastructure Services Division*
> 
> | Informatics International Limited | 89/57 | Jampettah Lane | Colombo 13 |
> Sri Lanka |
> 
> | T: +94-115-794-942 (Dir)| F: +94-112-542-832 | M: +94-768209719 |
> 
> | asank...@informaticsint.com  |
> www.informaticsint.com |
> 
> 
> On 9 August 2016 at 10:22, Asanka Gunasekara 
> wrote:
> 
>> Hi Hope someone can help me on this
>>
>>  Below is my env
>>
>> a) CloudStack 4.9 on CentOs 7.2,
>> b) running on 2 compute nodes and one controller node
>> c) NFS as shared storage primary and secondary
>> d) Basic networking
>> f) Hosts, Guests, system VMs and NFS storage are in same IP range
>>
>> I can create VMs and from UI I can see Guests are getting IPs assigned
>>
>>  1. when the instance boots up it has no IP assigned (ip addr show
>> shows no IP). But I have to do dhclient for the guest to get configured.
>>
>>  2. Even though instance gets an IP after running dhclient, I am
>> unable to reach no further than the host that the instance are in. All the
>> routes are in place. I have created Security Group with allow all to this
>> guest still no luck
>>
>> please let me know if you need any additional information
>>
>> Best Regards
>>
>> *Asanka Gunasekara*
>>
>> *Asst. Engineering Manager - Systems Infra Services*
>> *Global IT Infrastructure Services Division*
>>
>> | Informatics International Limited | 89/57 | Jampettah Lane | Colombo 13
>> | Sri Lanka |
>>
>> | T: +94-115-794-942 (Dir)| F: +94-112-542-832 | M: +94-768209719 |
>>
>> | asank...@informaticsint.com  |
>> www.informaticsint.com |
>>
>>
> 

Re: Urgent: Can't deploy VM.

[ilya]

Is this still an issue?

Your logs arent giving us anything useful.

retry the start vm and if it fails, run below command:

> grep -E 'ERROR' /var/log/cloudstack/management/management-server.log  | grep 
> 'Failed to start instance VM' | tail -1

Response would look like:

> 2016-08-02 23:54:23,724 ERROR [c.c.v.VirtualMachineManagerImpl] 
> (Work-Job-Executor-60:ctx-b3db4d1a job-143982/job-143985 ctx-53d5067c) Failed 
> to start instance VM[User|i-109-13930-ff]

where as "i-109-13930-ff" is cloudstack instancename for the vm that
failed to start.

note the jobid and ctx ids and only grep for those lines.

Example:

> grep -E 'ctx-b3db4d1a|job-143982|job-143985' 
> /var/log/cloudstack/management/management-server.log


Look through the error - it will tell you what went wrong, if it makes
no sense - paste it to pastebin and share the link.

Regards
ilya



On 8/5/16 9:16 AM, 조대형 wrote:
> HI, I am using Cloudstack 4.7.
> All of sudden, I can't deploy and terminate VM.
> What should I do to fix this issue?
> Anyone who got this error before?
> Please, help me.
> 
> I found this error;
> 
> tail -f /var/log/cloudstack/management/management-server.log | grep -i -E
> 'exception|unable|fail|invalid|leak|warn|error'
> 2016-08-06 00:54:52,086 WARN  [c.c.v.VirtualMachinePowerStateSyncImpl]
> (DirectAgentCronJob-33:ctx-b0f97ee6) (logid:54175709) VM state was updated
> but update time is null?! vm id: 979
> 2016-08-06 00:54:53,122 WARN  [c.c.v.VirtualMachinePowerStateSyncImpl]
> (DirectAgentCronJob-66:ctx-a1074eb7) (logid:0dfd1efd) VM state was updated
> but update time is null?! vm id: 1288
> 2016-08-06 00:54:54,478 WARN  [c.c.v.VirtualMachinePowerStateSyncImpl]
> (DirectAgentCronJob-132:ctx-3cf7c07a) (logid:697a13ab) VM state was updated
> but update time is null?! vm id: 1287
> [root@cloud ~]# tail -f
> /var/log/cloudstack/management/management-server.log | grep -i -E
> 'exception|unable|fail|invalid|leak|warn|error'
> 2016-08-06 01:06:25,217 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
> (Timer-1:ctx-3562d0cc) (logid:50b13c81) Task (job-11252) has been pending
> for 3305 seconds
> 2016-08-06 01:06:25,217 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
> (Timer-1:ctx-3562d0cc) (logid:50b13c81) Task (job-11255) has been pending
> for 3264 seconds
> 

Re: networking problem with Windows 7 guest

[ilya]

Hi

Where are you getting you templates from?

I suspect the issue might be with how templates have been created. Try
launching console sessions to both windows and linux and see if you can
ping the gateway.

Regards
ilya

On 8/5/16 7:23 AM, Xiaoming Yang wrote:
> Hi All,
> 
> I have set up my 'cloud' with a basic zone automatically using cloudmonkey 
> following the sample script in page 
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+cloudmonkey+CLI#CloudStackcloudmonkeyCLI-Installation
> 
> After that, I have done following tests:
> 
> 1.  Add two ISO images as templates, RHEL6.6 and Win7.
> 
> 2.  Add ingress & egress rules allowing 'ping' and 'ssh'
> 
> 3.  Create a RHEL instance and test 'ping' & 'ssh'.
> 
> 4.  Create a Win7 instance and test 'ping'
> 
> Before step 4, thing are going as expected. I can ping rhel VM from external 
> of the 'cloud', and ping external machines from within the rhel VM.
> However, I couldn't do the 'ping' with the Win7 VM. This is strange. Both VMs 
> are in the same zone, and I created them the same way.
> Does anyone know anything about this 'problem' or did I miss anything? Any 
> help/tips are appreciated. BTW, I use 4.8 releases.
> 
> 
> Regards,
> 
> Xiaoming
> 
> 

Re: Overprovising CPU with XenServer 6.5SP1

[ilya]

With release of ACS 4.3, cloudstack keeps track of previous
overprovisioning values for CPU and MEM. That is done to guarantee the
resources you initially promised when VM was deployed.

I know some people will question this decision, nevertheless, it makes
sense to others - as the feature was implemented.

In order to re-balance all existing VMs CPU over provisioning value with
new values, you need to run a sql on cloudstack mysql db.


In my experience, it would also require restart of CloudStack MS servers
for change to kick in right away.

Backup your mysql db first.

Here is an example of the SQL query one might run:

> update user_vm_details set value=2.25 where name="cpuOvercommitRatio" and 
> vm_id in (SELECT count(id) FROM cloud.vm_instance where hypervisor_type="KVM" 
> and state not like "Expunging" and type="User" and pod_id=13);

Note this query increases CPU OverCommit value to 2.25 on all VMs in
pod_id=13. If you want to do it zonewide replace pod_id with
data_center_id="x".

You can look up the zone id in data_center table.

Regards
ilya


On 8/8/16 5:55 AM, Stephan Seitz wrote:
> Hi
> 
> cpu overprovision factor is only recalculated for vms started *after*
> the factor has been changed.
> The more vms you're going to stop/start, the more accurate your
> calculation is getting.
> 
> Please refer to http://docs.cloudstack.apache.org/projects/cloudstack-a
> dministration/en/4.8/hosts.html#over-provisioning-and-service-offering-
> limits
> 
> cheers,
> 
> - Stephan
> 
> 
> Am Montag, den 08.08.2016, 15:22 +0300 schrieb Mindaugas Milinavičius:
>> Hello,
>>
>> Does anyone did overprovising with CPU and Xen? I tryed to change in
>> global
>> settings, but in dashboard anyway shoing like 1:1...
>>
>>
>>
>> Pagarbiai
>> Mindaugas Milinavičius
>> UAB STARNITA
>> Direktorius
>> http://www.clustspace.com
>> LT: +37068882880
>> RU: +7913933
>>
>> Tomorrow's possibilities today
>> <http://www.clustspace.com/>
>>
>>- 1 core CPU, 512MB RAM, 20GB (€ 5.00)
>>- 1 core CPU, 1GB RAM, 30GB (€ 10.00)
>>- 2 core CPU, 2GB RAM, 40GB (€ 20.00)
>>- 2 core CPU, 4GB RAM, 60GB (€ 40.00)
>>- 4 core CPU, 8GB RAM, 80GB (€ 80.00)
>>- 8 core CPU, 16GB RAM, 160GB (€ 160.00)

Re: Mess after volume migration.

[ilya]

this happened to us on non XEN hypervisor as well.

CloudStack has a timeout for a long running jobs - which i assume in
your case - it has exceeded.

Changing volumes table should be enough by referencing proper pool_id.
Just make sure that data size matches on both ends.

consider changing
"copy.volume.wait" (if that does not help) also "vm.job.timeout"


Regards
ilya

On 8/8/16 3:54 AM, Makrand wrote:
> Guys,
> 
> My setup:- ACS 4.4.2. Hypervisor: XENserver 6.2.
> 
> I tried moving a volume in running VM from primary storage A to primary
> storage B (using GUI of cloudstack). Please note, primary storage A LUN
> (LUN7)is coming out of one storage box and  primary storage  B LUN (LUN14)
> is from another.
> 
> For VM1 with 250GB data volume (51 GB used space), I was able to move this
> volume without any glitch in about 26mins.
> 
> But for VM2 with 250Gb data volume (182 GB used space), the migration
>  continued for about ~110 mins and then failed with follwing exception in
> very end with message like:-
> 
> 2016-08-06 14:30:57,481 WARN  [c.c.h.x.r.CitrixResourceBase]
> (DirectAgent-192:ctx-5716ad6d) Task failed! Task record:
> uuid: 308a8326-2622-e4c5-2019-3beb
> 87b0d183
>nameLabel: Async.VDI.pool_migrate
>  nameDescription:
>allowedOperations: []
>currentOperations: {}
>  created: Sat Aug 06 12:36:27 UTC 2016
> finished: Sat Aug 06 14:30:32 UTC 2016
>   status: failure
>   residentOn: com.xensource.xenapi.Host@f242d3ca
> progress: 1.0
> type: 
>   result:
>errorInfo: [SR_BACKEND_FAILURE_80, , Failed to mark VDI hidden
> [opterr=SR 96e879bf-93aa-47ca-e2d5-e595afbab294: error aborting existing
> process]]
>  otherConfig: {}
>subtaskOf: com.xensource.xenapi.Task@aaf13f6f
> subtasks: []
> 
> 
> So cloudstack just removed the JOB telling it failed, says the mangement
> server log.
> 
> A) But when I am checking it at hyeprvisor level, the volume is on new SR
> i.e. on LUN14. Strange huh? So now the new uuid for this volume from XE cli
> is like
> 
> [root@gcx-bom-compute1 ~]# xe vbd-list
> vm-uuid=3fcb3070-e373-3cf9-d0aa-0a657142a38d
> uuid ( RO) : f15dc54a-3868-8de8-5427-314e341879c6
>   vm-uuid ( RO): 3fcb3070-e373-3cf9-d0aa-0a657142a38d
> vm-name-label ( RO): i-22-803-VM
>  vdi-uuid ( RO): cc1f8e83-f224-44b7-9359-282a1c1e3db1
> empty ( RO): false
>device ( RO): hdb
> 
> B) But luckily I had the entry taken before migration  and it shows like:-
> 
> uuid ( RO) : f15dc54a-3868-8de8-5427-314e341879c6
> vm-uuid ( RO): 3fcb3070-e373-3cf9-d0aa-0a657142a38d
> vm-name-label ( RO): i-22-803-VM
> vdi-uuid ( RO): 7c073522-a077-41a0-b9a7-7b61847d413b
> empty ( RO): false
> device ( RO): hdb
> 
> C) Since this failed at cloudstack, the DB is still holding old value.
> Here is current volume table entry in DB
> 
> id: 1004
>> account_id: 22
>>  domain_id: 15
>>pool_id: 18
>>   last_pool_id: NULL
>>instance_id: 803
>>  device_id: 1
>>   name:
>> cloudx_globalcloudxchange_com_W2797T2808S3112_V1462960751
>>   uuid: a8f01042-d0de-4496-98fa-a0b13648bef7
>>   size: 268435456000
>> folder: NULL
>>   path: 7c073522-a077-41a0-b9a7-7b61847d413b
>> pod_id: NULL
>> data_center_id: 2
>> iscsi_name: NULL
>>host_ip: NULL
>>volume_type: DATADISK
>>  pool_type: NULL
>>   disk_offering_id: 6
>>template_id: NULL
>> first_snapshot_backup_uuid: NULL
>>recreatable: 0
>>created: 2016-05-11 09:59:12
>>   attached: 2016-05-11 09:59:21
>>updated: 2016-08-06 14:30:57
>>removed: NULL
>>  state: Ready
>> chain_info: NULL
>>   update_count: 42
>>  disk_type: NULL
>> vm_snapshot_chain_size: NULL
>> iso_id: NULL
>> display_volume: 1
>> format: VHD
>>   min_iops: NULL
>>   max_iops: NULL
>>  hv_ss_reserve: 0
>> 1 row in set (0.00 sec)
>>
> 
> 
> So the path variable shows value as 7c073522-a077-41a0-b9a7-7b61847d413b
> and pool id as 18.
> 
> The VM is running as of now, but I am sure the moment I will reboot, this
> volume will be gone or worst VM won't boot. This is production VM BTW.
> 
> D) So I think I need to edit volume table for path and pool_id parameters
> and need to place new values in place and then reboot VM. Do I need to make
> any more changes in DB in some other tables for same? Any comment/help is
> much appreciated.
> 
> 
> 
> 
> --
> Best,
> Makrand
> 

Re: Meet BlueOrangutan

[ilya]

1 word - awesome!

On 8/6/16 2:14 AM, Rohit Yadav wrote:
> All,
> 
> 
> Meet blueorangutan [1], a Github bot account that will help us automate 
> CloudStack (PR) testing [2][3] among other things.
> 
> 
> It works by polling Github notifications for the apache/cloudstack repository 
> every minutes and then reacts to comments. We can post comments on a 
> apache/cloudstack PR and ask @blueorangutan to perform certain build jobs 
> such as building packages, then running Trillian [2] tests (across a set of 
> hypervisors) using those packages, and finally report us the results.
> 
> 
> Since, the task of building packages and testing them are expensive. A 
> typical packaging job may take up to 30 minutes, a typical Trillian [2][3] 
> environment can take about 30 minutes to build/deploy a zone, and a Trillian 
> (smoke) test run may take hours while an exhaustive Trillian 
> (component+smoke) test run may take 3-4 days. Due to these reasons, for now 
> the '@blueorangutan test' task is restricted to a selected Github users (my 
> colleagues at ShapeBlue). Running Trillian test for each PR may be expensive, 
> we may consider batching smaller thoroughly reviewed PRs, then create 
> packages for a set of PRs and test them all at once as well.
> 
> 
> The task to build (centos6, centos7, debian) packages is relatively less 
> expensive and anyone can request blueorangutan to build packages now by 
> commenting '@blueorangutan package' on a PR.
> 
> 
> The great blue ape aims to be a quite friendly and polite bot, go on any 
> apache/cloudstack PR and try:
> 
> '@blueorangutan hello'
> 
> 
> Or, build package for a PR:
> 
> '@blueorangutan package'
> 
> 
> Or, if it helps you build packages you may say:
> 
> '@blueorangutan thanks'
> 
> 
> This is an experiment and I hope to improve this over time, any feedback and 
> comments are welcome.
> 
> 
> [1] https://github.com/blueorangutan
> 
> [2] 
> http://www.shapeblue.com/trillian-flexible-on-demand-individual-cloud-environment-creation/
> 
> [3] 
> http://www.shapeblue.com/cloudstack-test-automation-with-trillian-and-jenkins/
> 
> Regards.
> 
> rohit.ya...@shapeblue.com 
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>   
>  
> 
> 

Re: Architecture Advice

[ilya]

Matthew,

Noticed that you are on users list, if you get no response, try asking
on dev list.

Also, perhaps refine the subject to VR VPN + LDAP access. Lastly, there
is StrongSwan initiative to replace OpenSwan, but nothing about LDAP
integration that i could find.

Regards
ilya

On 8/2/16 12:52 PM, Matthew Smart wrote:
> Ilya,
> 
> Thanks for the response. For the most part, our deployment is much
> simpler than yours. We allow only our senior sysadmins access to the
> Cloudstack UI (and only have 2 senior sysadmins currently). This access
> is already tied to LDAP and working perfectly. I don't mind using a vm
> for VPN since we have sysadmin staff with direct physical access to the
> datacenter 24/7. Worst case in an outtage they can connect directly to
> the bare metal servers and interface with a VM through the hypervisor
> vnc port just like the Cloudstack Console Proxy does.
> 
> What we are stumbling on is allowing our development staff, sysadmins,
> and clients to access the vms directly via ssh and other access
> protocols. I have to allow them the ability to remote into vms to
> perform maintenance, configuration, and troubleshooting but have to keep
> these networks completely segregated and managed by our centralized LDAP
> system. This access is currently facilitated in our non-cloudstack
> environment by allowing them to VPN into segregated networks and
> directly access the vms but we do so by allowing our VPN cluster to
> access ALL segregated networks. This creates a single point of
> vulnerability in that if an attacker gains access to a server in the VPN
> cluster they have penetrated our segregation and can access all networks.
> 
> My plan was to use the built in VPN capabilities of the VRouter
> instances to provide for a more secure asset segregation while allowing
> stakeholders the necessary access to their vms. The stumbling point
> right now is how we manage the vpns for the 50-60 separate networks we
> will have when this is rolled out. From what I can find, the current VPN
> implementation allows for the manual creation of 8 VPN users for each
> Cloudstack Account and I cannot find anything to indicate whether the
> VPN users can be managed via LDAP the way that the Cloudstack UI users are.
> 
> Does anyone have any guidance on the capabilities of the VRouter VPN
> offering? Am I correct in my determination that there is not currently
> any way to configure it to pull auth and access rights from LDAP?
> 
> Thanks,
> 
> Matthew Smart
> President
> Smart Software Solutions Inc.
> 108 S Pierre St.
> Pierre, SD 57501
> 
> Phone: (605) 280-0383
> Skype: msmart13
> Email: msm...@smartsoftwareinc.com
> 
> On 07/29/2016 02:30 AM, ilya wrote:
>> Matthew,
>>
>> Interesting challenge, i operate in slightly different environment -
>> let me explain how it works in places i've been too in past and you can
>> decide if its something you see being a fit.
>>
>> Since data center access is treated as top tier - access to it must be
>> guaranteed at all times - especially to sysadmin. Hence, i'm personally,
>> hesitant placing it on a VM - managed by cloudstack, openstack or vmware
>> or any virtual technology..
>>
>> I'd prefer for it to be a physical redundant VPN appliance - but its
>> just me, being overly paranoid, bitten by many outages - and probably
>> not cloudy enough.
>>
>> With that said, the VPN profile - will inherit a configuration that can
>> access whatever number of VLANs you have to offer - on the network
>> layer. For example, i'd create a Admin network that can access all
>> networks underneath that is bound to my VPN users.
>>
>> As for cloudstack access, i see few ways of solving your challenge - but
>> i also believe i may not fully understand you design.
>>
>> For example, in my environment, i may have close to 100 cloud admins.
>> These are the people that tend to different environments across many
>> datacenters doing different things. Some fix hypervisors, other deal
>> with network and vms or do capacity planning.
>>
>> When they login to cloudstack to perfom management task - select few -
>> that we may trust - get root admin priveleges. They can access all
>> cloudstack entities below ROOT domain - there are no restrictions. This
>> is something that is available now cloudstack.
>>
>> However, i may also have 98 users that i dont trust as much and want to
>> limit what they can do, for that - we will leverage another feature
>> called Dynamic CloudStack Roles A.K.A. RBAC.
>>
>> link: http://www.shapeblue.com/cloudstack-101/ - scroll down to
>> Management
>>
>> What RBAC gets

Re: LDAP (Active Directory) password concerns

[ilya]

Marty see response in-line

On 7/31/16 11:32 PM, Marty Godsey wrote:
> The password has been changed. If I try to log onto a machine in the domain 
> with the old password it tells me the password is incorrect. 
correct behavior

If I use the new one, it logs me into the machine.
also correct behavior


There are only three accounts in the ACS instance: admin, bare-metal and
testallow. Testallow is the LDAP account.

not following where the issue might be
> 
> 
> Regards,
> Marty Godsey
> 
> -Original Message-
> From: ilya [mailto:ilya.mailing.li...@gmail.com] 
> Sent: Monday, August 1, 2016 2:29 AM
> To: users@cloudstack.apache.org
> Subject: Re: LDAP (Active Directory) password concerns
> 
> Do you happen to have local account as well as ldap account set?
> 
> It usually follows one authentication method (ldap) followed by another 
> (local). Please confirm the passwords are different.
> 
> I will be testing ldap this week and will let you know if i see this issue. 
> I've used it in past, I'd be surprised to see this behavoiur, last i recall, 
> we dont cache - and do a lookup to LDAP each time user tries to 
> authenticate.. You should see this in the logs..
> 
> 
> Regards,
> ilya
> 
> On 7/31/16 11:01 PM, Marty Godsey wrote:
>> Hello,
>>
>> I have a lab CloudStack that is authenticating to an active directory and it 
>> works great accept one thing. If I change the password on the AD user, ACS 
>> still allows the user to log into the ACS portal with the old AND the new 
>> password...
>>
>> Is there a refresh interval for LDAP accounts? Does it store a hash in the 
>> ACS database? Did I miss a setting?
>>
>> Regards,
>> Marty Godsey
>>
>>

Re: LDAP (Active Directory) password concerns

[ilya]

I must also mention, i dont use Active Directory..

On 7/31/16 11:29 PM, ilya wrote:
> Do you happen to have local account as well as ldap account set?
> 
> It usually follows one authentication method (ldap) followed by another
> (local). Please confirm the passwords are different.
> 
> I will be testing ldap this week and will let you know if i see this
> issue. I've used it in past, I'd be surprised to see this behavoiur,
> last i recall, we dont cache - and do a lookup to LDAP each time user
> tries to authenticate.. You should see this in the logs..
> 
> 
> Regards,
> ilya
> 
> On 7/31/16 11:01 PM, Marty Godsey wrote:
>> Hello,
>>
>> I have a lab CloudStack that is authenticating to an active directory and it 
>> works great accept one thing. If I change the password on the AD user, ACS 
>> still allows the user to log into the ACS portal with the old AND the new 
>> password...
>>
>> Is there a refresh interval for LDAP accounts? Does it store a hash in the 
>> ACS database? Did I miss a setting?
>>
>> Regards,
>> Marty Godsey
>>
>>

Re: LDAP (Active Directory) password concerns

[ilya]

Do you happen to have local account as well as ldap account set?

It usually follows one authentication method (ldap) followed by another
(local). Please confirm the passwords are different.

I will be testing ldap this week and will let you know if i see this
issue. I've used it in past, I'd be surprised to see this behavoiur,
last i recall, we dont cache - and do a lookup to LDAP each time user
tries to authenticate.. You should see this in the logs..


Regards,
ilya

On 7/31/16 11:01 PM, Marty Godsey wrote:
> Hello,
> 
> I have a lab CloudStack that is authenticating to an active directory and it 
> works great accept one thing. If I change the password on the AD user, ACS 
> still allows the user to log into the ACS portal with the old AND the new 
> password...
> 
> Is there a refresh interval for LDAP accounts? Does it store a hash in the 
> ACS database? Did I miss a setting?
> 
> Regards,
> Marty Godsey
> 
> 

Re: Nuage Plugin

[ilya]

Marty

Nuage is not an open source platform and requires special hardware
gateway along with few other nuage components that must be installed as
virtual appliances. Otherwise it works very well.

Regards
ilya

On 7/30/16 7:03 AM, Marty Godsey wrote:
> Has anyone used or using the Nuage Plugin? I also can't get a clear picture 
> on whether or not there VSP is FOSS or must be licensed. Anyone ever use it 
> or have an idea?
> 
> Regards,
> Marty Godsey
> 
> 

Re: CS 4.8 VMware - Virtual Router stuck at starting

[ilya]

Jacob

So you are setting a basic zone, which means you have a single network
for both hypervisors and guest VMs (or everything). In that case - the
control network should 0.0.0.0 - since there are no other networks. I
was under assumption you are using advanced zone - but its more clear now.

I'd suggest you start off with 4.5 for the time being - while we raise a
blocker issue for 4.8 (and possibly 4.9).

I'll see if i can spend some cycles to investigate this.

Regards
ilya



On 7/29/16 7:20 AM, Jacob Seeley wrote:
> ilya,
> 
> I'm using a Basic zone. Here is the workflow I'm using with actual IP 
> addresses. Any fields I left out you can assume I leave blank.
> 
> Add Zone
> 
> Zone Type: Basic
> Name: ZONE1
> IPv4 DNS: 10.70.116.20
> Internal DNS 1: 10.70.116.20
> Hypervisor: VMware
> Network Offering: DefaultSharedNetworkOffering
> 
> Physical Network
> 
> Management: vSwitch Name: vSwitch0
> Guest: vSwitch Name: vSwitch0
> 
> Pod name: POD1
> Reserved system gateway: 10.70.116.1
> Reserved system netmask: 255.255.255.0
> Start Reserved system IP: 10.70.116.60
> End Reserved system IP: 10.70.116.79
> 
> Guest Gateway: 10.70.116.1
> Guest Netmask: 255.255.255.0
> Guest start IP: 10.70.116.80
> Guest end IP: 10.70.116.99
> 
> The rest is Storage and is probably irrelevant here.
> 
> After I go through the wizard of adding a zone, it asks me to enable it, 
> which I do. Without any further action, 2 System VMs (Console and Secondary) 
> are created. The default CentOS template is downloaded. Both System VMs 
> receive 2x IP addresses, one on the Pod network and one on the guest network.
> 
> System Storage VM
> Public IP address: 10.70.116.81
> Private IP address: 10.70.116.73
> Gateway: 10.70.116.1
> 
> Console Proxy VM
> Public IP address: 10.70.116.80
> Private IP address: 10.70.116.74
> Gateway: 10.70.116.1
> 
> Only when I initiate my first VM from template (or even ISO) is a Virtual 
> Router deployed. Like mentioned before, it gets two NICS with the first one 
> being of Traffic Type Guest and an IP address of 10.70.116.92 and a second 
> NIC of Traffic Type Control and no IP address assigned (it reports 0.0.0.0). 
> Ultimately the virtual router gets deployed on the hypervisor (VMware) but 
> it's useless. The instance I tried deploying ultimately 
> fails. I suspect that this is the problem or a problem. The virtual router 
> gets an IP address on the guest network but not the management network.
> 
> Regarding the cloud agent/service (/etc/init.d/cloud) on the virtual router. 
> I mentioned earlier that I found that /etc/init.d/cloud on the virtual router 
> fails. I found this happens because /usr/local/cloud/systemvm never gets 
> populated on the virtual router. Further down the rabbit hole I go, I see 
> there is a script, /opt/cloud/bin/patchsystemvm.sh, that is responsible for 
> mounting the systemvm.iso and unzipping the contents to 
> /usr/local/cloud/systemvm. Both System VMS (console and secondary) do this 
> but not the virtual router. From what I can tell, the reason for this as 
> follows. If you look at the script (found here: 
> https://github.com/apache/cloudstack/blob/master/systemvm/patches/debian/config/opt/cloud/bin/patchsystemvm.sh)
> 
> There is a function of the script called patch_console_proxy. This function 
> gets called only if the following is satisfied: if [ "$TYPE" == 
> "consoleproxy" ] || [ "$TYPE" == "secstorage" ] && [ -f 
> ${PATCH_MOUNT}/systemvm.zip ]
> 
> I've noticed that the value for TYPE in every case I've tried this with the 
> virtual router is equal to dhcpsrv. According to that script, the function 
> that gets called for TYPE=dhcpdsrv is dhcpsrvr_svcs. That function does the 
> following:
> 
> dhcpsrvr_svcs() {
>chkconfig cloud off
>chkconfig cloud-passwd-srvr on ; 
>chkconfig haproxy off ; 
>chkconfig dnsmasq on
>chkconfig ssh on
>chkconfig nfs-common off
>chkconfig portmap off
>chkconfig keepalived off
>chkconfig conntrackd off
>echo "ssh dnsmasq cloud-passwd-srvr apache2" > 
> /var/cache/cloud/enabled_svcs
>echo "cloud nfs-common haproxy portmap" > /var/cache/cloud/disabled_svcs
> } 
> 
> Here you can see that it turns off the cloud service. As far as I can tell, 
> my system router is executing this function, so this is expected behavior. 
> This tells me that the service cloud is to never run when TYPE=dhcpsrv.
> 
> As you mentioned before, I've since tried manually assigning an IP address 
> NIC1 on the virtual router but it doesn't seem to help or do anything.
> 
> Without ever having a working setup of CloudStack bef

Re: Architecture Advice

[ilya]

Matthew,

Interesting challenge, i operate in slightly different environment -
let me explain how it works in places i've been too in past and you can
decide if its something you see being a fit.

Since data center access is treated as top tier - access to it must be
guaranteed at all times - especially to sysadmin. Hence, i'm personally,
hesitant placing it on a VM - managed by cloudstack, openstack or vmware
or any virtual technology..

I'd prefer for it to be a physical redundant VPN appliance - but its
just me, being overly paranoid, bitten by many outages - and probably
not cloudy enough.

With that said, the VPN profile - will inherit a configuration that can
access whatever number of VLANs you have to offer - on the network
layer. For example, i'd create a Admin network that can access all
networks underneath that is bound to my VPN users.

As for cloudstack access, i see few ways of solving your challenge - but
i also believe i may not fully understand you design.

For example, in my environment, i may have close to 100 cloud admins.
These are the people that tend to different environments across many
datacenters doing different things. Some fix hypervisors, other deal
with network and vms or do capacity planning.

When they login to cloudstack to perfom management task - select few -
that we may trust - get root admin priveleges. They can access all
cloudstack entities below ROOT domain - there are no restrictions. This
is something that is available now cloudstack.

However, i may also have 98 users that i dont trust as much and want to
limit what they can do, for that - we will leverage another feature
called Dynamic CloudStack Roles A.K.A. RBAC.

link: http://www.shapeblue.com/cloudstack-101/ - scroll down to Management

What RBAC gets is an ability to define you won custom role within
cloudstack to perform only specific operations based on fairly granular
cloudstack API. For example, you may want a user who needs to be able to
READ content from CloudStack - but not make any changes.
You would create a role with "List*" priveleges, assing an account and
user on ROOT domain. This would be equivalent of read-only-admin user.

Other admins, could do VM stop, start, reboot, snapshot and read and
change some  settings - you can create a Power User role to do that as
well and since they are sysadmin users - you will assign them to ROOT
domain - so they can see all your customers within ACS.

There is no limit as to how granular you can be in terms of access to
cloudstack. If there is an API that does it - you can decide how and who
uses it.

You can also tie your cloudstack with LDAP group, but you still have to
import your users into cloudstack once - there is an import api command
for that. These users can be tied to specific account and role of your
choosing to only perform specific operations.

Lastly, RBAC has been committed to master branch and i believe it maybe
part of 4.9 release that community is testing now. However, if you feel
you want to be on older - more stable release - you can backport the
commits to your own branch and rebuild from source. We had this feature
backported to 4.5.2 - which we find stable for our needs.

Hope i answered some of your questions and VPN can be addressed by
someone else.

Regards
ilya
On 7/28/16 11:49 AM, Matthew Smart wrote:
> Not sure if this is the right place for this question but I am in the
> process of migrating my datacenter to cloudstack from a manually managed
> virtualization cluster. I am doing this because we need to implement
> full segregation between assets owned by different entities and managing
> that manually would be highly inefficient.
> 
> I have everything configured and working exactly the way I want it from
> a segregation standpoint. When fully migrated we will have around 50
> separate accounts all segregated onto their own vlans. The stumbling
> block for me now is VPN access. We do not operate a public cloud. A
> small number of sysadmins in my organization are responsible for all
> management and administration of all assets hosted in the datacenter.
> 
> Afaik, to use the VPN capability of the VRouter I would have to create
> users for each sysadmin in all 50 accounts and then propagate any
> changes to access rights via the api or manually through the UI. Our
> current setup has 7 segregated vlans that are accessible via a single
> OpenVPN gateway that queries my ldap server to determine access rights
> and pushes network routes when a user authenticates.
> 
> I would like to reproduce this capability in Cloudstack but am faltering
> at determining how it could be done. I would prefer to keep all assets
> including the Master VPN gateway as vms inside my Cloudstack environment
> and really don't want to incur the overhead of adding an OpenVPN VM to
> each account. I also can't really just create a shared network and give
> e

Re: CS 4.8 VMware - Virtual Router stuck at starting

[ilya]

Daren

I'm also running 4.5.2 - and like the stability we get with it.

For the features we need, 4.5.2 - has everything that is required, so I
dont see huge benefit of upgrading to latest ACS ATM. Also, our
environments are very large and complex - so upgrade is not something I
can take lightly.

With that said, i do have a small 8 node Lab environment i can try the
upgrade on, it consists of 4 ESXi and 4 KVM nodes - so it should be a
fair test.

Lets wait for Jacob to respond with his test of setting up IP/Netmask
for eth1 router vm, if it does not help, i'll try to upgrade to see if i
can reproduce the issue.

Regards
ilya

On 7/28/16 9:43 PM, Darren Tang wrote:
> Hi ilya:
>  I can confirm that issus,  please check :
> https://issues.apache.org/jira/browse/CLOUDSTACK-9144
>  When we deployed cloudstack(4.6/4.7/4.8)  with vmware(5.x/6.0) in basic
> zone,  The VR is nerver leaves the "starting" state.  fell back to 4.5 is
> fine.
>  Maybe you can test it by yourself.
> 
> 2016-07-29 3:24 GMT+08:00 ilya <ilya.mailing.li...@gmail.com>:
> 
>> I guess it would help to know what type of zone you use?
>>
>> Is it advanced, isolated vpc or shared network? what type of isolation?
>> or perhaps basic zone?
>>
>> Lastly, try stopping the iptables and restarting cloud agent (via stop
>> and start)
>>
>> Please see my response in-line
>>
>> On 7/28/16 6:58 AM, Jacob Seeley wrote:
>>> Hi ilya,
>>>
>>> Funny you brought up debugging the router VM. After I responding
>> yesterday, I did just that and I did find some odd things.
>>> Just to be clear (I think we're on the same page), since I'm not the OP
>> of this thread, the virtual router always gets deployed and it starts up
>> just fine; however, CloudStack reports that it's always stuck in starting.
>> VMs that get deployed ultimately fail. CloudStack reports the router
>> version as UNKNOWN.
>>> Before I provide what I found debugging the router VM, I'll address some
>> of your points.
>>>
>>> ### FOLLOW-UP QUESTIONS ###
>>>
>>> " Another reason would be an issue of hypervisor accessing the NFS mount
>> used for secondary storage."
>>> I don't believe this is an issue. The hypervisor (VMware) does mount the
>> secondary storage via NFS just fine. If this were an issue, I would think
>> the Secondary Storage and Console VMs would not deploy.
>>>
>>> " Use console of vCenter to see what is happening on router vm. You can
>> login locally with root/password and see the content of /var/log/cloud.out
>> file, paste it on pastebin - if it makes no sense to you..."
>>> It looks like to me that /var/log/cloud.out is only logged to when
>> $CLOUD_DEBUG is set to a non-zero length in the /etc/init.d/cloud script.
>> As such, there isn't even a file for /var/log/cloud.out. Even when I set
>> that variable, I never get anything logged to /var/log/cloud.out. However,
>> there is a /var/log/cloud.log. Here is the contents of that:
>> http://pastebin.com/aaTsRKZE
>>>
>>> " you can also run /etc/init.d/cloud stop and start.. that will give you
>> a fresh start on logs.."
>>> The service is in a failed state. It's worth noting that this service is
>> in a started state on the Console and Secondary Storage VMs.
>>
>> this is concerning - see you did "sh -x", read on..
>>
>>>
>>> " also, confirm that management server can talk to VR on POD IP
>>> (management) on port 3922.."
>>> It appears this is not an issue; see below:
>>
>> 3922 from MS to VR - this is the SSH daemon on VR with private key
>> 8250 from VR to MS - cloudstack java agent on VR talking to MS
>>
>>
>>>
>>> root@r-4-VM:~# telnet 10.70.110.101 8250
>>> Trying 10.70.110.101...
>>> Connected to 10.70.110.101.
>>> Escape character is '^]'.
>>>
>>
>>
>>> ### ROUTE VM DEBUG ###
>>>
>>> Here is what I found with router VM gets deployed (please tell me if
>> anything seems off):
>>> 2 NICs; only one NIC gets an IP  address. CloudStack NIC1 shows an IP
>> address coming from the defaultGuestNetwork. NIC2 is traffic type Control
>> but has an IP address of 0.0.0.0
>>
>> It is an issue for concern to see 0.0.0.0 assigned to eth1
>>
>> Lets assume NIC1 (as eth0) and NIC2 (as eth1).
>>
>> 1) we should not be getting 0.0.0.0 for eth1 - aka control network. This
>> IP should be coming from the POD network range -> when you added a pod -
>> i assume you did it as part of

Re: CS 4.8 VMware - Virtual Router stuck at starting

[ilya]

I guess it would help to know what type of zone you use?

Is it advanced, isolated vpc or shared network? what type of isolation?
or perhaps basic zone?

Lastly, try stopping the iptables and restarting cloud agent (via stop
and start)

Please see my response in-line

On 7/28/16 6:58 AM, Jacob Seeley wrote:
> Hi ilya,
> 
> Funny you brought up debugging the router VM. After I responding yesterday, I 
> did just that and I did find some odd things. 
> Just to be clear (I think we're on the same page), since I'm not the OP of 
> this thread, the virtual router always gets deployed and it starts up just 
> fine; however, CloudStack reports that it's always stuck in starting. VMs 
> that get deployed ultimately fail. CloudStack reports the router version as 
> UNKNOWN.
> Before I provide what I found debugging the router VM, I'll address some of 
> your points.
> 
> ### FOLLOW-UP QUESTIONS ###
> 
> " Another reason would be an issue of hypervisor accessing the NFS mount used 
> for secondary storage."
> I don't believe this is an issue. The hypervisor (VMware) does mount the 
> secondary storage via NFS just fine. If this were an issue, I would think the 
> Secondary Storage and Console VMs would not deploy.
> 
> " Use console of vCenter to see what is happening on router vm. You can login 
> locally with root/password and see the content of /var/log/cloud.out file, 
> paste it on pastebin - if it makes no sense to you..."
> It looks like to me that /var/log/cloud.out is only logged to when 
> $CLOUD_DEBUG is set to a non-zero length in the /etc/init.d/cloud script. As 
> such, there isn't even a file for /var/log/cloud.out. Even when I set that 
> variable, I never get anything logged to /var/log/cloud.out. However, there 
> is a /var/log/cloud.log. Here is the contents of that: 
> http://pastebin.com/aaTsRKZE
> 
> " you can also run /etc/init.d/cloud stop and start.. that will give you a 
> fresh start on logs.."
> The service is in a failed state. It's worth noting that this service is in a 
> started state on the Console and Secondary Storage VMs.

this is concerning - see you did "sh -x", read on..

> 
> " also, confirm that management server can talk to VR on POD IP
> (management) on port 3922.."
> It appears this is not an issue; see below:

3922 from MS to VR - this is the SSH daemon on VR with private key
8250 from VR to MS - cloudstack java agent on VR talking to MS


> 
> root@r-4-VM:~# telnet 10.70.110.101 8250
> Trying 10.70.110.101...
> Connected to 10.70.110.101.
> Escape character is '^]'.
> 


> ### ROUTE VM DEBUG ###
> 
> Here is what I found with router VM gets deployed (please tell me if anything 
> seems off):
> 2 NICs; only one NIC gets an IP  address. CloudStack NIC1 shows an IP address 
> coming from the defaultGuestNetwork. NIC2 is traffic type Control but has an 
> IP address of 0.0.0.0

It is an issue for concern to see 0.0.0.0 assigned to eth1

Lets assume NIC1 (as eth0) and NIC2 (as eth1).

1) we should not be getting 0.0.0.0 for eth1 - aka control network. This
IP should be coming from the POD network range -> when you added a pod -
i assume you did it as part of Add Zone wizard...

To see the PODIP range, goto UI
Infrastructure, Zones, Your Zone, Physical Network, Physical Network 1
(assume you did not create anything special), Management, IP Ranges ->
you should see a range defined there and it should not be 0.0.0.0...

> From the CloudStack management server, I cannot SSH into the router VM on 
> NIC1. I've found this is because of iptables rules on the router VM. If I 
> issue a /etc/init.d/iptables-persistent flush on the router VM, I can SSH 
> into the router VM using the SSH key at port 3922.
> The service "cloud" is in a failed state. Looking at the cloud init script, I 
> see the following:
> 
> CMDLINE=$(cat /var/cache/cloud/cmdline)
> 
> TYPE="router"
> for i in $CMDLINE
>   do
> # search for foo=bar pattern and cut out foo
> FIRSTPATTERN=$(echo $i | cut -d= -f1)
> case $FIRSTPATTERN in 
>   type)
>   TYPE=$(echo $i | cut -d= -f2)
>   ;;
> esac
> done
> 
> The file cat /var/cache/cloud/cmdline exist; here are the contents:
> 
> template=domP name=r-4-VM eth0ip=10.70.116.75 eth0mask=255.255.255.0 
> gateway=10.70.116.1 domain=vit.vertitechit.com cidrsize=24 
> dhcprange=10.70.116.1 eth1ip=0.0.0.0 eth1mask=0.0.0.0 mgmtcidr=10.70.110.0/24 
> localgw=10.70.116.1 sshonguest=true type=dhcpsrvr disable_rp_filter=true 
> extra_pubnics=2 dns1=10.70.10.21 
> baremetalnotificationsecuritykey=nu1HfF_DpC-gK-G_3y1u54Snb9ruROq-qldOvhnHj4EMypguvtfQu0o18eY3gs81iPZMD2Du1QOUAG5KOfMYXQ
>  
> baremetalnotificationapikey=CKZoOXffpY5ihjvzly

Re: CS 4.8 VMware - Virtual Router stuck at starting

[ilya]

Hi Jacob

I gave this a second read - if your issue is Router VM in starting mode
- but not started - it means cloudstack agent on routerVM cannot talk to
management server on 8250 over POD network.

Another reason would be an issue of hypervisor accessing the NFS mount
used for secondary storage.

Use console of vCenter to see what is happening on router vm. You can
login locally with root/password and see the content of
/var/log/cloud.out file, paste it on pastebin - if it makes no sense to
you...

you can also run /etc/init.d/cloud stop and start.. that will give you a
fresh start on logs..

also, confirm that management server can talk to VR on POD IP
(management) on port 3922..

Regards
ilya

On 7/27/16 9:34 AM, Jacob Seeley wrote:
> ilya,
> 
> Here are the contents of the secondary storage:
> 
> .
> ./template
> ./template/tmpl
> ./template/tmpl/1
> ./template/tmpl/1/8
> ./template/tmpl/1/8/49a4c4ee-ef06-4474-92c3-1d8efb082266.ova
> ./template/tmpl/1/8/template.properties
> ./template/tmpl/1/8/systemvm64template-4.6.0-RC20151104T1522-4.6.0-vmware.ovf
> ./template/tmpl/1/8/systemvm64template-4.6.0-RC20151104T1522-4.6.0-vmware-disk3.vmdk
> ./template/tmpl/1/7
> ./template/tmpl/1/7/template.properties
> ./template/tmpl/1/7/0098d168-4985-3b33-9840-eb5848d2f385.ova
> ./template/tmpl/1/7/CentOS5.3-x86_64.ovf
> ./template/tmpl/1/7/CentOS5.3-x86_64-disk1.vmdk
> ./template/tmpl/1/7/CentOS5.3-x86_64.mf
> ./systemvm
> ./systemvm/systemvm-4.8.0.1.iso
> ./systemvm/.lck-bf162a01
> ./snapshots
> ./volumes
> 
> I've noticed that both the Secondary Storage VM and Console Proxy VM mount 
> this ISO and as stated before, they come up just fine.
> 
> Regards,
> 
> Jacob Seeley
> Sr. Infrastructure Engineer
> VertitechIT
> 413-268-1631
> 
> www.vertitechit.com
> 
> -Original Message-
> From: ilya [mailto:ilya.mailing.li...@gmail.com] 
> Sent: Wednesday, July 27, 2016 3:22 AM
> To: users@cloudstack.apache.org
> Subject: Re: CS 4.8 VMware - Virtual Router stuck at starting
> 
> Jacob
> 
> The upgrade usually occurs though systemvm.iso - that is generated by 
> cloudstack on the first start.
> 
> Please show the content of your secondary store specifically
> 
> /mnt/[secondary-storage]/systemvm
> 
> Regards
> ilya
> 
> On 7/25/16 11:19 AM, Jacob Seeley wrote:
>> Here is a pastebin snippet the management-server.log - 
>> http://pastebin.com/GCLm53Gz
>>
>> Hopefully the relevant data is in there.
>>
>> I made sure to start from scratch for this example. Everything from the 
>> vSphere ESXi to the vCenter to the CentOS 7 with CloudStack install is 
>> fresh. I deployed a new instance in CloudStack, a VM internally named 
>> i-2-3-VM with an IP address of 192.168.0.78. This prompted CloudStack to 
>> deploy a VR. The VR is called r-4-VM with an IP address of 192.168.0.79.
>>
>> Thank you,
>>
>> Jacob Seeley
>> Sr. Infrastructure Engineer
>> VertitechIT
>> 413-268-1631
>>
>> www.vertitechit.com
>>
>> -Original Message-
>> From: Suresh Sadhu [mailto:suresh.sa...@accelerite.com]
>> Sent: Monday, July 25, 2016 1:37 AM
>> To: users@cloudstack.apache.org
>> Subject: Re: CS 4.8 VMware - Virtual Router stuck at starting
>>
>> please upload the logs in the issue.
>>> On Jul 5, 2016, at 8:46 AM, Darren Tang <darrentang...@gmail.com> wrote:
>>>
>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9144
>>>
>>> 2016-07-04 19:41 GMT+08:00 Glenn Wagner <glenn.wag...@shapeblue.com>:
>>>
>>>> Hi,
>>>>
>>>> What template are you using to start your first VM? - the default 
>>>> vmware template?
>>>> If you look in vcenter , what does the console show you ?
>>>>
>>>>
>>>> Glenn
>>>>
>>>>
>>>>
>>>> glenn.wag...@shapeblue.com
>>>> www.shapeblue.com
>>>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West, Cape Town 
>>>> 7130South Africa @shapeblue
>>>>
>>>>
>>>>
>>>>
>>>> -Original Message-
>>>> From: Pascal R. [mailto:repa...@gmail.com]
>>>> Sent: Monday, 04 July 2016 1:26 PM
>>>> To: users@cloudstack.apache.org
>>>> Subject: CS 4.8 VMware - Virtual Router stuck at starting
>>>>
>>>> hi,
>>>>
>>>> we have a CS4.8 deployment with VMWare 5.5.
>>>>
>>>> When trying to launch the first VM, the VS is created. VS starts up, 
>>>> but in CS, it stuck with "starting" state.
>>>>
>>>> i can't find any usefull information in the logs.
>>>>
>>>> any hint?
>>>>
>>
>>
>>
>>
>> DISCLAIMER
>> ==
>> This e-mail may contain privileged and confidential information which is the 
>> property of Accelerite, a Persistent Systems business. It is intended only 
>> for the use of the individual or entity to which it is addressed. If you are 
>> not the intended recipient, you are not authorized to read, retain, copy, 
>> print, distribute or use this message. If you have received this 
>> communication in error, please notify the sender and delete all copies of 
>> this message. Accelerite, a Persistent Systems business does not accept any 
>> liability for virus infected mails.
>>

Re: CS 4.8 VMware - Virtual Router stuck at starting

[ilya]

Jacob

The upgrade usually occurs though systemvm.iso - that is generated by
cloudstack on the first start.

Please show the content of your secondary store specifically

/mnt/[secondary-storage]/systemvm

Regards
ilya

On 7/25/16 11:19 AM, Jacob Seeley wrote:
> Here is a pastebin snippet the management-server.log - 
> http://pastebin.com/GCLm53Gz
> 
> Hopefully the relevant data is in there.
> 
> I made sure to start from scratch for this example. Everything from the 
> vSphere ESXi to the vCenter to the CentOS 7 with CloudStack install is fresh. 
> I deployed a new instance in CloudStack, a VM internally named i-2-3-VM with 
> an IP address of 192.168.0.78. This prompted CloudStack to deploy a VR. The 
> VR is called r-4-VM with an IP address of 192.168.0.79.
> 
> Thank you,
> 
> Jacob Seeley
> Sr. Infrastructure Engineer
> VertitechIT
> 413-268-1631
> 
> www.vertitechit.com
> 
> -Original Message-
> From: Suresh Sadhu [mailto:suresh.sa...@accelerite.com] 
> Sent: Monday, July 25, 2016 1:37 AM
> To: users@cloudstack.apache.org
> Subject: Re: CS 4.8 VMware - Virtual Router stuck at starting
> 
> please upload the logs in the issue.
>> On Jul 5, 2016, at 8:46 AM, Darren Tang <darrentang...@gmail.com> wrote:
>>
>> https://issues.apache.org/jira/browse/CLOUDSTACK-9144
>>
>> 2016-07-04 19:41 GMT+08:00 Glenn Wagner <glenn.wag...@shapeblue.com>:
>>
>>> Hi,
>>>
>>> What template are you using to start your first VM? - the default 
>>> vmware template?
>>> If you look in vcenter , what does the console show you ?
>>>
>>>
>>> Glenn
>>>
>>>
>>>
>>> glenn.wag...@shapeblue.com
>>> www.shapeblue.com
>>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West, Cape Town 
>>> 7130South Africa @shapeblue
>>>
>>>
>>>
>>>
>>> -Original Message-
>>> From: Pascal R. [mailto:repa...@gmail.com]
>>> Sent: Monday, 04 July 2016 1:26 PM
>>> To: users@cloudstack.apache.org
>>> Subject: CS 4.8 VMware - Virtual Router stuck at starting
>>>
>>> hi,
>>>
>>> we have a CS4.8 deployment with VMWare 5.5.
>>>
>>> When trying to launch the first VM, the VS is created. VS starts up, 
>>> but in CS, it stuck with "starting" state.
>>>
>>> i can't find any usefull information in the logs.
>>>
>>> any hint?
>>>
> 
> 
> 
> 
> DISCLAIMER
> ==
> This e-mail may contain privileged and confidential information which is the 
> property of Accelerite, a Persistent Systems business. It is intended only 
> for the use of the individual or entity to which it is addressed. If you are 
> not the intended recipient, you are not authorized to read, retain, copy, 
> print, distribute or use this message. If you have received this 
> communication in error, please notify the sender and delete all copies of 
> this message. Accelerite, a Persistent Systems business does not accept any 
> liability for virus infected mails.
> 

Re: Latest stable ACS for VMware

[ilya]

i can only speak of 4.5.2 - i know envs with 300+ hypervisors/zone.. its
stable with no known issues... 4.7 has complete vDS support - if its
something you care about - but i've not used it - so i cant recommend it

I'd strongly suggesting having KVM side by side, so much easier to deal
with... drop VMware it - if you can... migration will be painful.

Things i dont like about vmware (applies to medium & large envs):
license fees - insane
vcenter occasionally malfunctions
in rare instances - disks can get corrupted
you have no flexibility as to what you can integrate with - unless its
from vmware or its top tier vendor
most users use fraction of features vmware offers - hence KVM will do
equally well




On 7/26/16 11:59 PM, Cristian Ciobanu wrote:
> Hello,
> 
>  
> 
>  Just a short question, can i know what is the latest and sable ACS
> version that can be used with Vmware ( also if is used by someone in PROD )
> 
>  
> 
>   Thank you!
> 
>  
> 
> Regards,
> 
> Cristian
> 
>   www.istream.today
> 
>  
> 
> 

Re: State of the S3 secondary storage

[ilya]

We've been kicking around idea internally on abstracting secondary
storage entirely and offload it to cloudstack agent.

Our biggest gripe is NFS.

If we can expose S3 (over HTTPs) like service and offload to cloudstack
agent - we will remove large dependency.

The only issue we see - is vmware, in which case it will have to be
backed by NFS...





On 7/19/16 2:37 PM, Nux! wrote:
> Glusterfs ;)
> 

Re: ScaleIO + ACS + HA

[ilya]

You would need to write a scaleio driver for cloudstack - that can mount
the disks on demand.

If you are a paying EMC customer you can ask them to write it for you.


On 7/13/16 6:11 AM, Mindaugas Milinavičius wrote:
> Hello,
> 
> does anyone have solution for ScaleIO and ACS high availability?
> 
> As we understanding, we can create volume ant map only on 1 server. If this
> server goes down - problem with storage will be for all hosts...
> 
> How do we do it:
> 
> Creating domain > adding SDS > Creating volume > mapping on the server and
> creating NFS > Adding to ACS primary storage.
> 
> Any solution?
> 

Re: Amazon Web Services Interface

[ilya]

Влад

As Eric mentioned, AWS integration piece has been replace with
cloud-ec2stack sometime ago - as it was a rather painful piece of code
to maintain as part of cloudstack bundle.

Instead - it now lives its own live.

https://github.com/apache/cloudstack-ec2stack

Regards
ilya


On 7/11/16 10:23 AM, Vladislav Nazarenko wrote:
> Hello All,
> 
> I'm trying to activate Amazon Web Services Interface according to this
> description:
> http://docs.cloudstack.apache.org/projects/cloudstack-installation/en/4.6/optional_installation.html#amazon-web-services-interface
> 
> 
> After enabling the services in global options and restarting management
> service I can not see any process listening on 7080, as mentioned in the
> description.
> Moreover 7080 ist not mentioned in /etc/cloudstack/management/server.xml
> 
> Probably I'm just missing a section in server.xml?
> 
> Can somebody share a working server.xml file with me or point me the
> direction :)
> 
> appreciate your help!
> Vlad
> 

Re: Is com.cloud.hypervisor.kvm.resource.KVMHAChecker used by CloudStack?

[ilya]

Rohan

As of now:
Disconnect the primary NFS from your KVM and see what happens.

In the future release:

Also, HA piece is being rewritten now. The specs are posted by John
Burwell (and me to a smaller extent) if you search cloudstack mailing
lists via markmail.org for "KVM HA" you can see the thread with many
details.

In summary, we will be changing the behavior to something more precise -
similar to how VmWare does it.

Example: host A, B and C are part of 1 cluster that use a common
clustered storage

host A hangs and halts the VMs ability to write to disk (or crash the vms)

CloudStack MS will retreive the list of volumes used by VMs for host A
ask the neighbor host B to check for when the last write has been
performed.

If all VMs with their disks have no disk activity for predefined
interval (several intervals), cloudstack MS will use IMPI interface to
shoot the node in the head.

This is a very high level overview - there is alot more to this with
many safeguards and tun-able parameters.

Regards
ilya


On 7/11/16 5:33 PM, Rohan T wrote:
> Hi All,
> 
> Having been smashed by the unexpected behaviour of the KVM Heartbeat / HA
> process, we've been working through the logic of the process, and  I now
> believe the intent of the process is sumarised by:
> 
> 
> =
> The heartbeat process consists of 3 parts:
> 
> 1. a shell script that's distributed to each of the hypervisors during the
> CloudStack installation process:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
> 2. Two java classes, built into CloudStack
> com.cloud.hypervisor.kvm.resource.KVMHAMonitor
> com.cloud.hypervisor.kvm.resource.KVMHAChecker
> 
> Behaviour
> 
> Each of the classes periodically calls the kvmheartbeat.sh script with
> different arguments, the script is used to confirm the existence of NFS
> mounts,  remount any that are missing, clean up (i.e. kill) VMs in
> indeterminate state, read and write heartbeats to NFS volumes and force the
> host hypervisor to reboot (as part of a "shoot the node in the head"
> approach to restoring sanity to the cluster).
> 
> The KVMHAMonitor script writes a timestamp to each of the NFS volumes
> (pools), each minute,  if this process times out  (4 times), then calls the
> script once more to force a spontaneous reboot of the host (via: echo b >
> /proc/sysrq_trigger).
> 
> The KVMHAChecker is responsible for triggering the script to read the
> heartbeat value and compare with the current timestamp. Where ALL NFS
> volumes are determined to be "DEAD" (i.e timestamp is older than 60
> seconds),
> 
> 
> 
> Is my understanding correct?
> 
> The problem is, when testing this logic in my test lab (currently 4.4.4,
> but there's been no significant updates committed to these files since),
> I've been unable to see any evidence of the KVMHAChecker actually
> executing!  I see plenty of evidence of heartbeat writes (and of hypervisor
> reboots triggered when this process timesout).
> 
> 
> Thanks,
> Rohan
> 

Re: Opportunity to contribute in Apache CloudStack

[ilya]

Hi Janesh

Use case:
Manage different cloudstacks from single end point

Requirements:
When cloudstack is used in large scale, there many different types of
environments that would need to be supported - but on the high level,
there might be:
Development CloudStack Environments - that adhere to specific level of
configuration and content
Production CloudStack Environments - that adhere to specific level of
configuration and content

The way one might design something likes this:
1) an interface to register all cloudstack environments
2) ability to tag each environment (dev, test or prod)
3) apple specific configuration templates for tag above
4) as configuration change is made - cloudstack manager would reach out
to each applicable cloudstack environment and apply the change (if
necessary)
example:
dev environments, i may need to have specific service offerings 
with
specific templates and configurations
test environment, i would stage pre-prod settings or something 
that i
would eventually promote to production
prod environment, will only have set of approved settings, 
templates,
offerings, users etc...

CloudStack manager would need to support user with different level of
access.
1) admin - can do everything
2) user - can do specific things
3) read only user - can see things but not change

Besides managing the settings, the health of cloudstack zones (and
hosts) and its capacity would be displayed as well.
For example, you might have 20 zones in 1 cloudstack environments, zone
can be 90% used in terms of resources, contain about 1000 hypervisors
and have 30 hypervisors in bad state.


All of this is doable via cloudstack APIs at the moment on each
cloudstack management server, however, there is nothing that connects
and lets you manage different kind of environments that can be grouped
by tag (i.e. dev, test and prod).

Please let me know if this is clear or you need more details,

Regards
ilya


On 7/10/16 11:55 AM, Jainesh Patel wrote:
> Hello ilya,
> 
> Can you please tell us more about the "cloudstack manager" project, its
> use cases and requirements?  It looks like we would like to work on it.
> 
> Regards,
> Jainesh Patel
> LinkedIn <https://in.linkedin.com/pub/jainesh-patel/aa/196/5b2>
> 
> On Sat, Jul 9, 2016 at 11:32 AM, ilya <ilya.mailing.li...@gmail.com
> <mailto:ilya.mailing.li...@gmail.com>> wrote:
> 
> I agree that usage UI would be beneficial.
> 
> As other folks mentioned on this thread, cloudstack collects the usage
> metrics for each VM via cloud-usage service stored on cloud_usage
> database. We've been lacking this for some time now.
> 
> One other nice feature that would help apache cloudstack project alot -
> is "cloudstack manager".
> 
> Conceptually - this means you have many cloudstack management servers
> across the globe in different data centers, but there is nothing right
> now that ties them all together.
> 
> As the result - each cloudstack environment is managed as its own
> entity. Most companies then write their own "managers" or have scripts
> that keep cloudstacks in-sync and healthy. If you have large
> environments with thousands of hypervisors, many users and countless VMs
> - it can go wild pretty quickly.
> 
> What does it mean (for new comers), cloudstack has many constructs:
> Global Settings
> Service, Disk Offerings, Templates
> Zone Settings, Cluster Settings
> Storage Settings
> Healths of hypervisors, system and router vms
> Users, accounts and domains
> Virtual Machines
> ...and more
> 
> All of the above - could be managed via CloudStack API. However, its
> painful to keep all environments consistent.  Keeping your environments
> consistent and having a single view - great simplifies the management of
> dispersed cloudstack environments.
> 
> Either way - both are great projects - i can certainly help with use
> cases and requirements for "cloudstack manager".
> 
> I'm pretty certain other folks in the community would help with "usage"
> project.
> 
> Let us know what you would like to pursue.
> 
> Again, thank you for your participation
> 
> Regards,
> ilya
> 
> 
> On 7/8/16 3:30 AM, Erik Weber wrote:
> > Pricing is hard to do, since there are so many different ways an
> > organization might be doing their sales, so I would not focus on
> that for
> > the first release, but rather do it later.
> > Make a minimum viable product/solution and take it from there.
> >
> 
> 

Re: Opportunity to contribute in Apache CloudStack

[ilya]

I agree that usage UI would be beneficial.

As other folks mentioned on this thread, cloudstack collects the usage
metrics for each VM via cloud-usage service stored on cloud_usage
database. We've been lacking this for some time now.

One other nice feature that would help apache cloudstack project alot -
is "cloudstack manager".

Conceptually - this means you have many cloudstack management servers
across the globe in different data centers, but there is nothing right
now that ties them all together.

As the result - each cloudstack environment is managed as its own
entity. Most companies then write their own "managers" or have scripts
that keep cloudstacks in-sync and healthy. If you have large
environments with thousands of hypervisors, many users and countless VMs
- it can go wild pretty quickly.

What does it mean (for new comers), cloudstack has many constructs:
Global Settings
Service, Disk Offerings, Templates
Zone Settings, Cluster Settings
Storage Settings
Healths of hypervisors, system and router vms
Users, accounts and domains
Virtual Machines
...and more

All of the above - could be managed via CloudStack API. However, its
painful to keep all environments consistent.  Keeping your environments
consistent and having a single view - great simplifies the management of
dispersed cloudstack environments.

Either way - both are great projects - i can certainly help with use
cases and requirements for "cloudstack manager".

I'm pretty certain other folks in the community would help with "usage"
project.

Let us know what you would like to pursue.

Again, thank you for your participation

Regards,
ilya


On 7/8/16 3:30 AM, Erik Weber wrote:
> Pricing is hard to do, since there are so many different ways an
> organization might be doing their sales, so I would not focus on that for
> the first release, but rather do it later.
> Make a minimum viable product/solution and take it from there.
> 

Re: Opportunity to contribute in Apache CloudStack

[ilya]

Hi TheAtom Team

Thanks for selecting Apache CloudStack as a development platform.

We do have large number of initiative you can try to pursue.

Perhaps you can explain what skill set your team has and what would be
your desired trends - so we can help you better in selecting a new endeavor.

Regards
ilya

On 7/6/16 3:05 AM, Jainesh Patel wrote:
> Hello,
> 
> We are a group of students that are currently pursuing our undergraduate
> degree in Computer Science from Pune Insititute of Computer
> Technology(PICT), Maharashtra, India. We will be graduating in June 2017
> and are currently in our final year. For our B.E project, we have selected
> the domain as Cloud Computing and would be very interesting in working with
> open source cloud computing software, which is where we stumbled upon
> Apache CloudStack.
> 
> It will be a great learning opportunity for us to work with Apache
> CloudStack and in turn work with you. We would appreciate if you could
> steer us towards the direction of choosing the right topic and working
> towards culminating a project in the same, which would be helpful for the
> community.
> 
> Following are the few details which include information about us, which
> would help you in making an informed decision:
> 
> 1) Group Name- TheAtom
> 
> 2) Group Members:
> Shubham Mulay ( shubhammu...@gmail.com )
> Faizaan Shaikh ( faizaanshai...@gmail.com )
> Jainesh Patel ( jainesh...@gmail.com )
> 
> 3) We have two mentors working with us, who will be guiding throughout the
> process,
> Dhruvesh Rathore ( dhruves...@hotmail.com )
> Prerit Auti ( prerita...@gmail.com )
> 
> 4) Development time : 6 to 7 months from Aug '16 to Feb '17.
> 
> We would love to hear from you about any ideas that you see fit for us to
> pursue and which are feasible in the specified time frame. Hoping to hear
> from you soon, and thanking you in anticipation.
> 
> Regards,
> TheAtom
> 

Re: Issue with 'stuck' virtual routers

[ilya]

Hi Doug

Do you have primary storage id 18 available?

# cloudmonkey list storagepools id=18

I can only assume cloudstack tries to clean up after it self and fails -
because storage pool 18 is not available.

Are your running local storage zone or clustered?

Lastly, your logs would indicate the issue more clearly - as to why its
not able to expunge.

Regards
ilya

On 7/5/16 9:15 AM, Douglas Land wrote:
> We pulled a host from the pool for upgrades, and in the process seems to
> have gotten a virtual router in an odd state. It's showing as destroyed in
> the UI, but cloudmonkey says it's still expunging.
> 
> This host has been completely rebuild including completely redisked. On the
> management node I found:
> 
> mysql> select * from op_ha_work
> -> ;
> ++-+---+--+---++-+-+---+---+---+-+-+
> | id | instance_id | type  | vm_type  | state | mgmt_server_id
> | host_id | created | tried | taken | step  | time_to_try |
> updated |
> ++-+---+--+---++-+-+---+---+---+-+-+
> |  1 |  13 | Migration | DomainRouter | Expunging |   NULL
> |  24 | 2016-07-01 14:34:17 | 0 | NULL  | Migrating |  142034 |
> 205 |
> |  4 |  78 | Migration | DomainRouter | Destroyed |   NULL
> |  24 | 2016-07-01 14:34:17 | 0 | NULL  | Migrating |  142092 |
>  68 |
> ++-+---+--+---++-+-+---+---+---+-+-+
> 
> I removed those  entries, but when the hosts persist. Via cloudmonkey it
> shows expunging:
> {
>   "count": 1,
>   "router": [
> {
>   "account": "engineering",
>   "created": "2014-09-05T03:56:07+0200",
>   "dns1": "172.16.8.46",
>   "dns2": "172.16.8.47",
>   "domain": "engineering",
>   "domainid": "1da498ba-5646-4cc3-a704-a20ebe12f518",
>   "id": "dc48a402-41d8-4e93-b441-4b34eb83a4c8",
>   "isredundantrouter": true,
>   "name": "r-78-VM",
>   "nic": [],
>   "podid": "f53afa8d-51ff-484d-9a88-52e979aeb688",
>   "redundantstate": "UNKNOWN",
>   "requiresupgrade": false,
>   "role": "VIRTUAL_ROUTER",
>   "serviceofferingid": "ed6b13d0-3e74-4aa5-a6b7-a5d2ac6c4a6c",
>   "serviceofferingname": "System Offering For Software Router",
>   "state": "Expunging",
>   "templateid": "bb3f7e4e-d7f6-4a72-a752-12c3221e43e9",
>   "version": "4.4.1",
>   "zoneid": "3467ff63-b582-4ace-9fda-8d5851bd8753",
>   "zonename": "Oakland"
> }
>   ]
> }
> 
> If I try to destroy the host from the api I get:
> 
> Async job cf08d7fa-1609-4d0e-b33c-63cc38f7e897 failed
> Error 530, Unable to locate datastore with id 18
> {
>   "accountid": "e3389462-6020-425a-9b9e-57141d58e1ab",
>   "cmd": "org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd",
>   "created": "2016-07-05T17:23:53+0200",
>   "jobid": "cf08d7fa-1609-4d0e-b33c-63cc38f7e897",
>   "jobprocstatus": 0,
>   "jobresult": {
> "errorcode": 530,
> "errortext": "Unable to locate datastore with id 18"
>   },
>   "jobresultcode": 530,
>   "jobresulttype": "object",
>   "jobstatus": 2,
>   "userid": "xxx"
> }
> 
> I'm guessing I need to remove all references for the routers from the
> database. Does anyone know what table(s) that's stored in?
> 

Re: HA in management and DB For CLOUDstack

[ilya]

You should be "ok", however, assuming DB goes out on site 1 and you use
replicated DB on site 2 over VPN - there might be a considerable latency
between the MS on site 1 and DB on site 2 - which can work against you.

I dont know anything about your setup, is it KVM, VMware or XEN?

i'd setup

Site 1:
2 MS host (fronted by VIP)
2 DB hosts (replicated to each other and across the VPN as remote slave)

Site 2:
2 MS host (fronted by VIP)
2 DB hosts (replicated to each other and across the VPN as remote slave)

Also, i would probably not use CloudStack DB HA feature - and setup my
own mysql cluster (my opinion).

Regards,
ilya

On 7/1/16 11:33 AM, Ghaith Bannoura wrote:
> Hello All,
> 
> I have two sites that connected together using VPN  and I need to create on 
> each office one management and DB server, is it applicable to create each 
> site DB server with replication to the other one and each Management server 
> connect to the DB that its exist in the same site (each site have their 
> management and DB server with DB replication between each other )  ?
> 
> 
> 
> Best Regards,
> 
> Ghaith Bannoura
> 
> 
> 

Re: Cloustack with VMWare, Console not working

[ilya]

I suspect an issue between ESXi host and Console Proxy VM via MGMT
interface.

Just to confirm, try  watching netstat on console session
# watch 'netstat -antup | grep TIME'

Initiate a console from cloudstack - see if you have anything in
TIME_WAIT state.

Regards
ilya

On 6/30/16 9:06 AM, Pascal R. wrote:
> Hi List!
> 
> 
> 
> i'm testing arround with Cloudstack here, Cloudstack 4.8, vSphere 5.5 and
> ESXi 5.5
> 
> 
> 
> Everything works so far, all systems vm's are launched and working, i'm
> able to provision new vm's.
> 
> 
> 
> but the console is not working, i always get:
> 
> 
> 
> Quote
> 
> 
> Unable to start console session as connection is refused by the machine you
> are accessing
> 
> 
> 
> 
> 
> The Ports are open on the ESXi Hosts.
> 
> 
> Any hint ?
> 

Re: High Availability ACS

[ilya]

HA is being rewritten from the group up. It still at least few months
away from hitting master. Nevertheless, depending on your setup -
cloudstack build in HA might work.

On 6/28/16 2:18 AM, Mohd Zainal Abidin Rabani wrote:
> Hi,
> 
>  
> 
> Does the version 4.8 have success HA? Last time we try use version 4.7 but
> HA not quite working as expected. I'm hoping version 4.8 fix already issue
> on HA. What is method ACS use for HA?
> 
>  
> 
> Thanks.
> 
> 

Re: Help with Usage data - There is no data from Cloudmonkey of types: 4,5,10,21,22,23 or 24 (Network and VM Disk data)

[ilya]

Cory

Just curious if you made any headway on this issue or still need help.

Regards
ilya

On 6/17/16 8:38 AM, Cory Fuchs wrote:
> Hello Everyone,
> 
> 
> I hope that someone can point me in the direction of how to resolve my issue 
> and let me know what I am doing wrong.
> 
> 
> Cloudstack version: 4.5.2
> 
> 
> Running the following command from Cloudmonkey returns no results.
> 
> 
>  list usagerecords startdate=2016-06-16 enddate=2016-06-17 type=4
> 
> 
> Changing the type to any of the following returns no results: 4, 5, 10, 21, 
> 22, or 23
> 
> Changing the type to any of the following does return results: 1, 2, 3, 6, 7, 
> 8, 9, 11, 12, 13, 14, or 25
> 
> 
> This tells me the usage job is running and collecting data.  Just not on the 
> Network or VM Disk data.
> 
> 
> Looking at /var/log/cloudstack/usage/usage.log
> 
> 
> Grepping for an accountId that I know has usage on.  I get lines like the 
> following.
> 
> 
>  2016-06-17 00:15:03,347 DEBUG [cloud.usage.UsageManagerImpl] 
> (Usage-Job-1:null) creating networkHelperEntry...accountId: 7 in zone: 1; 
> abr: 142919240951; abs: 40911793471; curABS: 40911793471; curABR: 
> 142919240951; ubs: 0; ubr: 0
> 
>  2016-06-17 00:15:03,348 DEBUG [cloud.usage.UsageManagerImpl] 
> (Usage-Job-1:null) getting current accounted bytes for... accountId: 7 in 
> zone: 1; abr: 39361808078; abs: 1006529247
> 
>   2016-06-17 00:15:03,348 DEBUG [cloud.usage.UsageManagerImpl] 
> (Usage-Job-1:null) creating networkHelperEntry... accountId: 7 in zone: 1; 
> abr: 39361808078; abs: 1006529247; curABS: 1006529247; curABR: 39361808078; 
> ubs: 0; ubr: 0
> 
> 
> 
> Grepping for account 7.  I get some of the following lines:
> 
> 
>  2016-06-17 00:15:13,976 DEBUG [usage.parser.NetworkUsageParser] 
> (Usage-Job-1:null) No usage record (0 bytes used) generated for account: 7
> 
>  2016-06-17 00:15:13,976 DEBUG [usage.parser.NetworkUsageParser] 
> (Usage-Job-1:null) No usage record (0 bytes used) generated for account: 7
> 
>  2016-06-17 00:15:13,976 DEBUG [usage.parser.VmDiskUsageParser] 
> (Usage-Job-1:null) Parsing all Vm Disk usage events for account: 7
> 
>  2016-06-17 00:15:14,074 DEBUG [usage.parser.VmDiskUsageParser] 
> (Usage-Job-1:null) No vm disk usage record (0 bytes used) generated for 
> account: 7
> 
>  2016-06-17 00:15:14,074 DEBUG [usage.parser.VmDiskUsageParser] 
> (Usage-Job-1:null) No vm disk usage record (0 bytes used) generated for 
> account: 7
> 
> 
> Where can I look or what am I doing wrong?
> 
> I am mainly looking for network usage.
> 
> 
> Thank you to anyone that can assist.
> 
> 
> Cory Fuchs
> 
> 
> 

Re: Storage Performance

[ilya]

I see in the code setCacheMode for KVM is there - i'm not certain how to
invoke it to use writeback.

https://git-wip-us.apache.org/repos/asf?p=cloudstack.git=search=HEAD=grep=setCacheMode

On 6/6/16 7:26 AM, Vladimir Melnik wrote:
> Dear colleagues,
> 
> I've found why guest's storage performance was much less than host's 
> performance (the mistake was too stupid to tell about it, really).
> 
> But I'd like to ask one more question if you don't mind. :) I played with 
> various KVM options (cache, io and so on...) and now I can say that I got 
> most IOpS with cache=writeback. Tests are being performed with the fio 
> utility, here is the jobfile contents:
>   [readtest]
>   blocksize=4k
>   filename=/dev/vdb
>   rw=randread
>   direct=1
>   buffered=0
>   ioengine=libaio
>   iodepth=32
>   size=20%
> 
>   [writetest]
>   blocksize=4k
>   filename=/dev/vdb
>   rw=randwrite
>   direct=1
>   buffered=0
>   ioengine=libaio
>   iodepth=32
>   size=20%
> 
> Is there any way to make ACS run create KVM domains with cache=writeback 
> instead of cache=none without patching the source code?
> 
> Thank you!
> 

Re: cloudmonkey config file get reset to default settings

[ilya]

Rohit,

Its usually shell scripts that wrap around cloudmonkey. I hit this issue
when i have to run parallel tasks using multiple envs.

One other somewhat ugly work around is to make config file immurable
with chattr command, but then - it prints errors as its unable to write
to config file. Not pleasant - but does not mess up my config file.

I'm aware of workaround on making config files per env, but as you might
know - its painful to manage dispersed configs - epsecially when the
count is greater than 10..

Thanks,
ilya




On 5/25/16 10:57 PM, Rohit Yadav wrote:
> Whenever a set command is called, it would save/update the config file. When 
> you run set profile xyz; it needs to make that profile the default profile 
> and update other parameters associated with the profile which may be set as 
> well (such as url, username, password, apikey, secretkey etc). When 
> cloudmonkey is running, and you replace the config file; on calling 'set' it 
> would save the config file based on its in-memory config dictionary.
> 
> I'll see what I can do, in general you should be replacing or changing the 
> cloudmonkey config file outside of cloudmonkey itself. If you want to create 
> new profile, set new rules; you should call cloudmonkey set   
> either on command line or use puppet to execute them. The tool was intended 
> for single user, in case of multi-user or concurrent usage, there is no 
> concurrency control wrt configs.
> 
> One solution could be that, each server profile has their own config file 
> instead of a single config file, and you can start cloudmonkey to pick a 
> server profile with a command line flag such as -p . I'll see how I 
> may improve this, for this I would like to know how exactly you are using 
> puppet or any other automation?
> 
> Regards,
> Rohit Yadav
> 
> On May 26 2016, at 3:40 am, ilya <ilya.mailing.li...@gmail.com> wrote:
> 
> I've seen the similar behaviour.
> 
> For some reason cloudmonkey try to persist the configs each time your
> run something.
> 
> If i open cloudmonkey in multiple terminals and use different profiles
> and execute commands in mutliple terminals in parallel - i've seen
> cloudmonkey mess up the config for one of open profiles.
> 
> Specifically, the URL of cloudstack in profile1 might be changed with
> url of cloudstack in profile2.
> 
> Rohit, is there a reason why cloudmonkey tries to update the settings in
> config file each time something gets executed?
> 
> On 5/24/16 1:31 PM, Yiping Zhang wrote:
>> Hi,
>>
>> We have a few scripts that use cloudmonkey to talk to CloudStack server. The 
>> scripts are invoked by Puppet once per hour.
>>
>> However, every once a while, the /root/.cloudmonkey/config file would be 
>> over written with default settings. That is, blank apikey/secretkey, default 
>> password, default log file location etc.
>>
>> I am wondering by any chance that cloudmonkey would put a default config 
>> file in place for some reason ?
>>
>> Thanks,
>>
>> Yiping
>>
> 
> rohit.ya...@shapeblue.com 
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
> 
> 

Re: The libvirt version for production

[ilya]

EL7.2 build of linux with all KVM dependencies works well..

On 5/23/16 5:01 PM, Lv Haijiao wrote:
> hi, Community
> 
> We are trying to upgrade libvirt from 1.2.2. to a higher version in our
> production environment.
> 
> As there are so many releases,  anyone can share advice or experience for a
> stable one ?  or if we can always adopt the latest one ?
> 
> Thanks in advance !
> 

Re: cloudmonkey config file get reset to default settings

[ilya]

I've seen the similar behaviour.

For some reason cloudmonkey try to persist the configs each time your
run something.

If i open cloudmonkey in multiple terminals and use different profiles
and execute commands in mutliple terminals in parallel - i've seen
cloudmonkey mess up the config for one of open profiles.

Specifically, the URL of cloudstack in profile1 might be changed with
url of cloudstack in profile2.

Rohit, is there a reason why cloudmonkey tries to update the settings in
config file each time something gets executed?



On 5/24/16 1:31 PM, Yiping Zhang wrote:
> Hi,
> 
> We have a few scripts that use cloudmonkey to talk to CloudStack server.  The 
> scripts are invoked by Puppet once per hour.
> 
> However, every once a while, the /root/.cloudmonkey/config file would be over 
> written with default settings. That is, blank apikey/secretkey, default 
> password, default log file location etc.
> 
> I am wondering by any chance that cloudmonkey would put a default config file 
> in place for some reason ?
> 
> Thanks,
> 
> Yiping
> 

Re: Template and ISO downloaded but stuck at "installing template / installing ISO"

[ilya musayev]

Indra,

While I see there is a solution to your problem due to cloudstack changes
with web servers, there is also another tangent to this issue..

Let me know if your issue is resolved by leveraging a different web server
for image downloads.

But if not, or someone else comes across this message thread, here is a
another tangent to this problem.


Below is a summary to the problem we've seen that seem very similar

In the nutshell, here is what we've seen and what we believe cause the
issue.

CloudStack Java Agent initiates 2 https sessions when attempting to
download.

1) Session is established to web server to get the header information and
keep it open..
2) Session is established to download the content - this is usually a
longer stream operation


What we've seen happen, as session # 2 goes on downloading, somehow http
session #1 gets corrupted. Session #2 is eventually terminated as it
completes the operation, yet session #1 remains to be open.

This defies the logic in a way, because no packets are sent over the open
session between SSVM and HTTP server. I'd expect for one of two parties
sending termination packet - but that never happens. As the result - the
1st session hangs and prevents agent from proceed to the next stage of
installing the template. I've seen the socket being open for days on SSVM
and yet not a single packet comes - not even keep alive.

Patch should be out soon. Credit goes to Marcus for figuring this out.


I also have a shell script that can resolve the specific stuck template
download, but i've made it geared toward QCOW2.  I can post it if need be,
but we should probably release a proper fix..

Regards
ilya





On Wed, May 11, 2016 at 4:37 AM, Indra Pramana <in...@sg.or.id> wrote:

> Dear all,
>
> I setup a test environment using CloudStack 4.8.0 (latest from repository),
> one management server and two KVM agent hosts running Ubuntu 14.04 LTS.
> Setup a zone with basic networking because the test environment doesn't
> support VLAN and the test management server only have 1 NIC. The private
> and public network are using the same subnet. Using Ceph RBD as primary
> storage and NFS as secondary storage.
>
> Zone is created successfully, the two hosts are added successfully and
> connected to the management servers. SSVM and CPVM are created without any
> issues. However, the default
> "CentOS 5.5(64-bit) no GUI (KVM)" template is stuck at "Installing
> Template" status and Ready = "No" although it seems to be downloaded
> successfully. I also tried registering an Ubuntu ISO and it faces the same
> problem, it's downloaded successfully but then will stuck at "Installing
> ISO" status.
>
> I ran the SSVM health check and everything is OK. Also go through the
> troubleshooting steps as per below documentation but still unable to
> determine the root cause:
>
>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/SSVM,+templates,+Secondary+storage+troubleshooting
>
> I tried restarting the SSVM service, stop and start the SSVM and restarting
> the management-server service, it will cause the download of the ISO and
> template to be restarted, but it will then stuck at installing again.
>
> Logs from management server shows below lines which keeps on appearing /
> looping every several seconds:
>
> ===
> 2016-05-11 19:31:29,031 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> (Timer-6:ctx-369bfc57) (logid:0b1754a2) getCommandHostDelegation: class
> org.apache.cloudstack.storage.command.DownloadProgressCommand
> 2016-05-11 19:31:29,031 DEBUG [c.c.h.XenServerGuru] (Timer-6:ctx-369bfc57)
> (logid:0b1754a2) getCommandHostDelegation: class
> org.apache.cloudstack.storage.command.DownloadProgressCommand
> 2016-05-11 19:31:29,031 DEBUG [o.a.c.s.RemoteHostEndPoint]
> (Timer-6:ctx-369bfc57) (logid:0b1754a2) Sending command
> org.apache.cloudstack.storage.command.DownloadProgressCommand to host: 3
> 2016-05-11 19:31:29,034 DEBUG [c.c.a.t.Request] (Timer-6:ctx-369bfc57)
> (logid:0b1754a2) Seq 3-3028107799453237894: Sending  { Cmd , MgmtId:
> 181122461655966, via: 3(s-3-VM), Ver: v1, Flags: 100011,
> [{"org.apache.cloudstack.storage.command
>
> .DownloadProgressCommand":{"jobId":"cab4ada2-e52f-4c25-9c30-9ec23dc3fcd4","request":"GET_STATUS","hvm":false,"description":"CentOS
> 5.5(64-bit) no GUI
>
> (KVM)","checksum":"ed0e788280ff2912ea40f7f91ca7a249","maxDownloadSizeInBytes":536870912
>
> 00,"id":4,"resourceType":"TEMPLATE","installPath":"template/tmpl/1/4","_store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://x.x.0.242/export/cloudstack","_role":"Ima

Re: Register ISO

[ilya]

Certainly doable - but not ideal.

You would have to inspect what cloudstack does when it imports iso.
Something somewhat similar happens when you register the systemvm
template for the first time - but DB entry already exists.

I'd do following:

In your test env:

create mysql dump with single line inserts
import iso via cloudstack interface
create mysql dump with single line inserts
diff two outputs to identify whats being changed in mysql

identify what happens on secondary storage and where its being copied.

On 5/5/16 9:28 PM, Mohd Zainal Abidin Rabani wrote:
> Hi,
> 
>  
> 
> IS there anyway to quick register without using URL? Like copy-paste and
> define back at backend? 
> 
>  
> 
> Regards,
> 
>  
> 
> Mohd Zainal Abidin Rabani
> 
> Technical Support
> 
>  
> 
> ModernOne Data Solutions Sdn. Bhd.
> 
> [ Dedicated Server | VPS | Co-Location | Hosting ]
> 
> [ Managed Service | System Integration | Domain ]
> 
> W : www.nocser.net
> 
> FB: http://www.facebook.com/nocser
> 
> Twitter: @nocsertech
> 
> Skype: zabidin5417
> 
>  
> 
> 

Re: [Urgent]: corrupt DB after VM live migration with storage migration

[ilya]

never mind - on the "removed" disks - it deletes well.

On 5/4/16 9:55 PM, ilya wrote:
> I'm pretty certain cloudstack does not have purging on data disks as i
> had to write my own :)
> 
> On 5/4/16 9:51 PM, Ahmad Emneina wrote:
>> I'm not sure if the expunge interval/delay plays a part... but you might
>> want to set: storage.cleanup.enabled to false. That might prevent your
>> disks from being purged. You might also look to export those volumes, or
>> copy them to a safe location, out of band.
>>
>> On Wed, May 4, 2016 at 8:49 PM, Yiping Zhang <yzh...@marketo.com> wrote:
>>
>>> Before I try the direct DB modifications, I would first:
>>>
>>> * shutdown the VM instances
>>> * stop cloudstack-management service
>>> * do a DB backup with mysqldump
>>>
>>> What I worry the most is that the volumes on new cluster’s primary storage
>>> device are marked as “removed”, so if I shutdown the instances, the
>>> cloudstack may kick off a storage cleanup job to remove them from new
>>> cluster’s primary storage  before I can get the fixes in.
>>>
>>> Is there a way to temporarily disable storage cleanups ?
>>>
>>> Yiping
>>>
>>>
>>>
>>>
>>> On 5/4/16, 3:22 PM, "Yiping Zhang" <yzh...@marketo.com> wrote:
>>>
>>>> Hi, all:
>>>>
>>>> I am in a situation that I need some help:
>>>>
>>>> I did a live migration with storage migration required for a production
>>> VM instance from one cluster to another.  The first migration attempt
>>> failed after some time, but the second attempt succeeded. During all this
>>> time the VM instance is accessible (and it is still up and running).
>>> However, when I use my api script to query volumes, it still reports that
>>> the volume is on the old cluster’s primary storage.  If I shut down this
>>> VM,  I am afraid that it won’t start again as it would try to use
>>> non-existing volumes.
>>>>
>>>> Checking database, sure enough, the DB still has old info about these
>>> volumes:
>>>>
>>>>
>>>> mysql> select id,name from storage_pool where id=1 or id=8;
>>>>
>>>> ++--+
>>>>
>>>> | id | name |
>>>>
>>>> ++--+
>>>>
>>>> |  1 | abprod-primary1  |
>>>>
>>>> |  8 | abprod-p1c2-pri1 |
>>>>
>>>> ++--+
>>>>
>>>> 2 rows in set (0.01 sec)
>>>>
>>>>
>>>> Here the old cluster’s primary storage has id=1, and the new cluster’s
>>> primary storage has id=8.
>>>>
>>>>
>>>> Here are the entries with wrong info in volumes table:
>>>>
>>>>
>>>> mysql> select id,name, uuid, path,pool_id, removed from volumes where
>>> name='ROOT-97' or name='DATA-97';
>>>>
>>>
>>>> +-+-+--+--+-+-+
>>>>
>>>> | id  | name| uuid | path
>>>  | pool_id | removed |
>>>>
>>>
>>>> +-+-+--+--+-+-+
>>>>
>>>> | 124 | ROOT-97 | 224bf673-fda8-4ccc-9c30-fd1068aee005 |
>>> 5d1ab4ef-2629-4384-a56a-e2dc1055d032 |   1 | NULL|
>>>>
>>>> | 125 | DATA-97 | d385d635-9230-4130-8d1f-702dbcf0f22c |
>>> 6b75496d-5907-46c3-8836-5618f11dac8e |   1 | NULL|
>>>>
>>>> | 316 | ROOT-97 | 691b5c12-7ec4-408d-b66f-1ff041f149c1 | NULL
>>>  |   8 | 2016-05-03 06:10:40 |
>>>>
>>>> | 317 | ROOT-97 | 8ba29fcf-a81a-4ca0-9540-0287230f10c7 | NULL
>>>  |   8 | 2016-05-03 06:10:45 |
>>>>
>>>
>>>> +-+-+--+--+-+-+
>>>>
>>>> 4 rows in set (0.01 sec)
>>>>
>>>> On the xenserver of old cluster, the volumes do not exist:
>>>>
>>>>
>>>> [root@abmpc-hv01 ~]# xe vdi-list name-label='ROOT-97'
>>>>
>>>> [root@abmpc-

Re: [Urgent]: corrupt DB after VM live migration with storage migration

[ilya]

I'm pretty certain cloudstack does not have purging on data disks as i
had to write my own :)

On 5/4/16 9:51 PM, Ahmad Emneina wrote:
> I'm not sure if the expunge interval/delay plays a part... but you might
> want to set: storage.cleanup.enabled to false. That might prevent your
> disks from being purged. You might also look to export those volumes, or
> copy them to a safe location, out of band.
> 
> On Wed, May 4, 2016 at 8:49 PM, Yiping Zhang  wrote:
> 
>> Before I try the direct DB modifications, I would first:
>>
>> * shutdown the VM instances
>> * stop cloudstack-management service
>> * do a DB backup with mysqldump
>>
>> What I worry the most is that the volumes on new cluster’s primary storage
>> device are marked as “removed”, so if I shutdown the instances, the
>> cloudstack may kick off a storage cleanup job to remove them from new
>> cluster’s primary storage  before I can get the fixes in.
>>
>> Is there a way to temporarily disable storage cleanups ?
>>
>> Yiping
>>
>>
>>
>>
>> On 5/4/16, 3:22 PM, "Yiping Zhang"  wrote:
>>
>>> Hi, all:
>>>
>>> I am in a situation that I need some help:
>>>
>>> I did a live migration with storage migration required for a production
>> VM instance from one cluster to another.  The first migration attempt
>> failed after some time, but the second attempt succeeded. During all this
>> time the VM instance is accessible (and it is still up and running).
>> However, when I use my api script to query volumes, it still reports that
>> the volume is on the old cluster’s primary storage.  If I shut down this
>> VM,  I am afraid that it won’t start again as it would try to use
>> non-existing volumes.
>>>
>>> Checking database, sure enough, the DB still has old info about these
>> volumes:
>>>
>>>
>>> mysql> select id,name from storage_pool where id=1 or id=8;
>>>
>>> ++--+
>>>
>>> | id | name |
>>>
>>> ++--+
>>>
>>> |  1 | abprod-primary1  |
>>>
>>> |  8 | abprod-p1c2-pri1 |
>>>
>>> ++--+
>>>
>>> 2 rows in set (0.01 sec)
>>>
>>>
>>> Here the old cluster’s primary storage has id=1, and the new cluster’s
>> primary storage has id=8.
>>>
>>>
>>> Here are the entries with wrong info in volumes table:
>>>
>>>
>>> mysql> select id,name, uuid, path,pool_id, removed from volumes where
>> name='ROOT-97' or name='DATA-97';
>>>
>>
>>> +-+-+--+--+-+-+
>>>
>>> | id  | name| uuid | path
>>  | pool_id | removed |
>>>
>>
>>> +-+-+--+--+-+-+
>>>
>>> | 124 | ROOT-97 | 224bf673-fda8-4ccc-9c30-fd1068aee005 |
>> 5d1ab4ef-2629-4384-a56a-e2dc1055d032 |   1 | NULL|
>>>
>>> | 125 | DATA-97 | d385d635-9230-4130-8d1f-702dbcf0f22c |
>> 6b75496d-5907-46c3-8836-5618f11dac8e |   1 | NULL|
>>>
>>> | 316 | ROOT-97 | 691b5c12-7ec4-408d-b66f-1ff041f149c1 | NULL
>>  |   8 | 2016-05-03 06:10:40 |
>>>
>>> | 317 | ROOT-97 | 8ba29fcf-a81a-4ca0-9540-0287230f10c7 | NULL
>>  |   8 | 2016-05-03 06:10:45 |
>>>
>>
>>> +-+-+--+--+-+-+
>>>
>>> 4 rows in set (0.01 sec)
>>>
>>> On the xenserver of old cluster, the volumes do not exist:
>>>
>>>
>>> [root@abmpc-hv01 ~]# xe vdi-list name-label='ROOT-97'
>>>
>>> [root@abmpc-hv01 ~]# xe vdi-list name-label='DATA-97'
>>>
>>> [root@abmpc-hv01 ~]#
>>>
>>> But the volumes are on the new cluster’s primary storage:
>>>
>>>
>>> [root@abmpc-hv04 ~]# xe vdi-list name-label=ROOT-97
>>>
>>> uuid ( RO): a253b217-8cdc-4d4a-a111-e5b6ad48a1d5
>>>
>>>  name-label ( RW): ROOT-97
>>>
>>>name-description ( RW):
>>>
>>> sr-uuid ( RO): 6d4bea51-f253-3b43-2f2f-6d7ba3261ed3
>>>
>>>virtual-size ( RO): 34359738368
>>>
>>>sharable ( RO): false
>>>
>>>   read-only ( RO): true
>>>
>>>
>>> uuid ( RO): c46b7a61-9e82-4ea1-88ca-692cd4a9204b
>>>
>>>  name-label ( RW): ROOT-97
>>>
>>>name-description ( RW):
>>>
>>> sr-uuid ( RO): 6d4bea51-f253-3b43-2f2f-6d7ba3261ed3
>>>
>>>virtual-size ( RO): 34359738368
>>>
>>>sharable ( RO): false
>>>
>>>   read-only ( RO): false
>>>
>>>
>>> [root@abmpc-hv04 ~]# xe vdi-list name-label=DATA-97
>>>
>>> uuid ( RO): bc868e3d-b3c0-4c6a-a6fc-910bc4dd1722
>>>
>>>  name-label ( RW): DATA-97
>>>
>>>name-description ( RW):
>>>
>>> sr-uuid ( RO): 6d4bea51-f253-3b43-2f2f-6d7ba3261ed3
>>>
>>>virtual-size ( RO): 107374182400
>>>
>>>sharable ( RO): false
>>>
>>>   read-only ( RO): false

Re: [Urgent]: corrupt DB after VM live migration with storage migration

[ilya]

Yiping,

We've dealt with many corruptions in past. It was more around VMware as
it would eat up disks time to time. Or someone would move the VM out of
bound by doing storage or cluster vmotion.

The solution you described should work.

However, for extra paranoid:

step 1, full db backup
step 2, backup the root and data disks as some other file name - just in
case

Then proceed with your proposed solution.

As long as you have proper backups, you should be ok. If VM start
failed, the logs will tell you where cloudstack expects for volume to
be, you can either move the volume there or update cloudstack volumes
table and point it to correct pool_id.

Regards
ilya


On 5/4/16 8:49 PM, Yiping Zhang wrote:
> Before I try the direct DB modifications, I would first:
> 
> * shutdown the VM instances
> * stop cloudstack-management service
> * do a DB backup with mysqldump
> 
> What I worry the most is that the volumes on new cluster’s primary storage 
> device are marked as “removed”, so if I shutdown the instances, the 
> cloudstack may kick off a storage cleanup job to remove them from new 
> cluster’s primary storage  before I can get the fixes in.
> 
> Is there a way to temporarily disable storage cleanups ?
> 
> Yiping
> 
> 
> 
> 
> On 5/4/16, 3:22 PM, "Yiping Zhang" <yzh...@marketo.com> wrote:
> 
>> Hi, all:
>>
>> I am in a situation that I need some help:
>>
>> I did a live migration with storage migration required for a production VM 
>> instance from one cluster to another.  The first migration attempt failed 
>> after some time, but the second attempt succeeded. During all this time the 
>> VM instance is accessible (and it is still up and running).  However, when I 
>> use my api script to query volumes, it still reports that the volume is on 
>> the old cluster’s primary storage.  If I shut down this VM,  I am afraid 
>> that it won’t start again as it would try to use non-existing volumes.
>>
>> Checking database, sure enough, the DB still has old info about these 
>> volumes:
>>
>>
>> mysql> select id,name from storage_pool where id=1 or id=8;
>>
>> ++--+
>>
>> | id | name |
>>
>> ++--+
>>
>> |  1 | abprod-primary1  |
>>
>> |  8 | abprod-p1c2-pri1 |
>>
>> ++--+
>>
>> 2 rows in set (0.01 sec)
>>
>>
>> Here the old cluster’s primary storage has id=1, and the new cluster’s 
>> primary storage has id=8.
>>
>>
>> Here are the entries with wrong info in volumes table:
>>
>>
>> mysql> select id,name, uuid, path,pool_id, removed from volumes where 
>> name='ROOT-97' or name='DATA-97';
>>
>> +-+-+--+--+-+-+
>>
>> | id  | name| uuid | path
>>  | pool_id | removed |
>>
>> +-+-+--+--+-+-+
>>
>> | 124 | ROOT-97 | 224bf673-fda8-4ccc-9c30-fd1068aee005 | 
>> 5d1ab4ef-2629-4384-a56a-e2dc1055d032 |   1 | NULL|
>>
>> | 125 | DATA-97 | d385d635-9230-4130-8d1f-702dbcf0f22c | 
>> 6b75496d-5907-46c3-8836-5618f11dac8e |   1 | NULL|
>>
>> | 316 | ROOT-97 | 691b5c12-7ec4-408d-b66f-1ff041f149c1 | NULL
>>  |   8 | 2016-05-03 06:10:40 |
>>
>> | 317 | ROOT-97 | 8ba29fcf-a81a-4ca0-9540-0287230f10c7 | NULL
>>  |   8 | 2016-05-03 06:10:45 |
>>
>> +-+-+--+--+-+-+
>>
>> 4 rows in set (0.01 sec)
>>
>> On the xenserver of old cluster, the volumes do not exist:
>>
>>
>> [root@abmpc-hv01 ~]# xe vdi-list name-label='ROOT-97'
>>
>> [root@abmpc-hv01 ~]# xe vdi-list name-label='DATA-97'
>>
>> [root@abmpc-hv01 ~]#
>>
>> But the volumes are on the new cluster’s primary storage:
>>
>>
>> [root@abmpc-hv04 ~]# xe vdi-list name-label=ROOT-97
>>
>> uuid ( RO): a253b217-8cdc-4d4a-a111-e5b6ad48a1d5
>>
>>  name-label ( RW): ROOT-97
>>
>>name-description ( RW):
>>
>> sr-uuid ( RO): 6d4bea51-f253-3b43-2f2f-6d7ba3261ed3
>>
>>virtual-size ( RO): 34359738368
>>
>>sharable ( RO): false
>

Re: CloudStack and ScaleIO

[ilya]

I know of specific organization that is considering ScaleIO with
CloudStack and KVM.

If their POC succeeds, they will most likely work with EMC to create a
native CloudStack driver.

Based on the benchmarks, you should be able to run ScaleIO along side
KVM on the same server - assuming you have decent servers and good
network backplane to support your workload.

Regards
ilya

On 5/4/16 1:59 AM, Mindaugas Milinavičius wrote:
> Hello,
> 
> does anyone using CS with ScaleIO?
> 
> If you using:
> 
> 1. Storage and VM's on the same servers or differents?
> 2. Do you use sdc and sds on the same server or differents? SDS:SDC 1:1 or
> 1:*?
> 3. Is it any problem with KVM?
> 
> 
> 
> Pagarbiai
> Mindaugas Milinavičius
> UAB STARNITA
> Direktorius
> http://www.clustspace.com
> LT: +37068882880
> RU: +79651806396
> 
> Tomorrow's possibilities today
> <http://www.clustspace.com/>
> 
>- 1 Core, 512MB RAM, 20GB SSD, 1Gbps, Unlimited, Location: Romania, Los
>Angeles, Ashburn Washington - 11EUR
>- 1 Core, 1024MB RAM, 30GB SSD, 1Gbps, Unlimited, Location: Romania, Los
>Angeles, Ashburn Washington - 18,7EUR
>- 2 Cores, 2048MB RAM, 40GB SSD, 1Gbps, Unlimited, Location: Romania,
>Los Angeles, Ashburn Washington - 27,5EUR
>- 4 Cores, 4096MB RAM, 100GB SSD, 1Gbps, Unlimited, Location: Romania,
>Los Angeles, Ashburn Washington - 46EUR
> 

Re: I cannot start agent - Cloudstack 4.8 - Connection closed with -1 on reading size

[ilya]

Ellie,

We recently fixed a bug that prevents monopolizing the 8250 socket when
someone telnets onto 8250 without proper SSL Handshake.

Not certain if 4.8 has it merged and we released a 4.8.x patch upgrade.

Perhaps we need to release 4.8 with bugfix, please kindly post a message
on dev list.

Would you be able to rebuild cloudstack from source?

CCing Rohit as he was the developer for this fix.

On 5/3/16 4:30 PM, Elie MABO wrote:
> Hi Simon, 
> 
> I added the settings to JAVA_OPTS as you suggested, and now, ACS Manegement 
> is listened on TCP port 8250admin@mngt:~$ sudo netstat -tanp | grep 8250
> tcp0  0 0.0.0.0:82500.0.0.0:*   LISTEN
>   3143/jsvc.exec
> But, I always have the message "Connection closed by foreign host." when I 
> telnet the Management server on port 8250. So, agent on KVM hosts cannot 
> start.
> Thanks again for you help
> Elie.
>  
> 
> Le Lundi 2 mai 2016 22h07, Simon Weller  a écrit :
>  
> 
>  Ellie,
> 
> Can you try forcing ACS Management to only bind to IPV4?
> Normally this is done in the JAVA_OPTS by setting 
> -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses=true
> 
> I think on the ACS Ubuntu release, JAVA_OPTS is set in your init scripts, but 
> I'm not absolutely sure about that.
> 
> Maybe someone who runs Ubuntu can chime in on this.
> 
> - Si
> 
> 
> 
> 
> From: Elie MABO 
> Sent: Monday, May 2, 2016 8:38 PM
> To: Users; Dev
> Subject: I cannot start agent - Cloudstack 4.8 -  Connection closed with -1 
> on reading size
> 
> Hi,
> 
> I have upgraded my cloudstack from 4.4 to 4.8. After upgrading, the agent on 
> hosts cannot start. I receive the following message:
>  20:08:45,833 ERROR [cloud.agent.AgentShell] (main:null) (logid:) Unable to 
> start agent:
> com.cloud.utils.exception.CloudRuntimeException: Unable to start the 
> connection!
> at com.cloud.agent.Agent.start(Agent.java:230)
> at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399)
> at 
> com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367)
> at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)
> at com.cloud.agent.AgentShell.start(AgentShell.java:461)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:243)
> Caused by: com.cloud.utils.exception.NioConnectionException: Connection 
> closed with -1 on reading size.
> at com.cloud.utils.nio.NioConnection.start(NioConnection.java:94)
> at com.cloud.agent.Agent.start(Agent.java:228)
> ... 9 more
> Caused by: java.io.IOException: Connection closed with -1 on reading size.
> at com.cloud.utils.nio.Link.doHandshake(Link.java:513)
> at com.cloud.utils.nio.NioClient.init(NioClient.java:80)
> at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
> ... 10 more
> 2016-05-02 20:08:45,834 INFO  [cloud.agent.Agent] (AgentShutdownThread:null) 
> (logid:) Stopping the agent: Reason = sig.kill
> 
> I tried to troubleshoot, but nothing. Please, can anybody help me to solve my 
> problem ?
> 
> Additional information
> 
> 1- The output of netstat command shows that management server does not listen 
> on TCP port 8250, but on TCP6 port 8250
> admin@mgnt:~$ sudo netstat -tanp | grep 8250
> tcp6  0  0 :::8250:::*LISTEN  
> 20101/jsvc.exec
> 
> 2- The telnet command on port 8250 return the message "Connection closed by 
> foreign host."admin@mgnt:~$ telnet 192.168.1.10 8250
> Trying 192.168.1.10...
> Connected to 192.168.1.10.
> Escape character is '^]'.
> Connection closed by foreign host.
> 
> 3- Management server and hosts (2) are running Ubuntu 14.04 LTS as operating 
> system, and all are up-to-date.
> Thank you in advance.
> Elie
> 
>   
>