Re: cs 4.5.1, vm stopped when putting xen hosts in maintenance

[Rafael Weingärtner]

Got the idea of your work around now.

I have not paid much attention to the table data you posted (sorry for
that). Well, it seems you have all taken care of.

I have a suggestion to make; instead of changing data directly on the data
base, what about using CloudMonkey to update the HA flag to false on VMs?
Currently, it is the UI that is blocking such customization, but according
to the API, you should be able to do that.
https://cloudstack.apache.org/api/apidocs-4.5/root_admin/updateVirtualMachine.html

On Tue, Feb 7, 2017 at 11:25 AM, Francois Scheurer <
francois.scheu...@everyware.ch> wrote:

> Dear Rafael
>
>
> We changed it on the VM to avoid the required stop and start. Changing on
> the used Offering is not affecting the vm_instance.ha_enabled.
> We could create a new Offering without HA but the VM need to be stopped to
> let us apply it.
>
> We did not disabled the HA on these 4 VM's to solve the problem, but just
> to minimize the impact of the problem (now only i-189-1177-VM should be
> stopped intermittently) if the cleaning of op_it_work is not fixing the
> issue.
> Of course we could filter the update query with “instance id”, but if you
> look the the table output all instance_id are related to the 5 problematic
> VM's.
> The only exceptions are the first 3 rows that are related to VM's that
> does no exist anymore since July 2015 ...
> That is why I thought it may be safe to do.
>
>
> Best Regards
>
> Francois Scheurer
>
>
>
>
>
>
>
>
> On 07.02.2017 16:27, Rafael Weingärtner wrote:
>
> So, you changed the flags directly on "vm_instance". Why did not you
> change the service offering of these VMs using ACS UI or API? For me, it
> feels that the best approach would be to change the service offering of the
> affected VMs (the HA flag is first configured at the service offering). You
> could assign a service offering without HA; I mean, it would be a matter of
> creating a similar service offering (same configurations to the one they
> are using right now), but without HA.
>
> I am assuming that the change directly on “vm_instance” did not solve your
> problem, right? That is why you want to change some data at “op_it_work”. I
> did a quick look at “op_it_work”, and it seems that this table may be used
> for more things than HA. Here is a tip for complex production system, never
> run update SQLs in production environments without a very, very restrictive
> and well thought “where” clause. In my opinion this is not restrictive
> enough (update op_it_work set step="Done" where step!="Done"). I would
> advise you not to go further and change configurations directly on database
> (unless you have a dev team that is capable of reading, debugging, and work
> with ACS source code). ACS has over 500k lines of java code, and God knows
> how many lines of shell, python and other languages, software and systems
> that may be used; it is pretty hard sometimes to get a clear picture of how
> a change in the database will reverberate in the whole system without
> knowing in details the installation configurations and underlying
> structures (networking, hypervisors, storage systems and other pieces).
> However, if you want to proceed with that, I would suggest you adding some
> extra filters to that “where” clause. What about adding the “instance id”
> there?
>
> PS: BTW, do you already heard about the Cloudstack Fork from Schuberg
> Philis named Cosmic? https://github.com/MissionCriticalCloud/cosmic
> Yes, I have. They were part of the active ACS community back there. I was
> already here when they created the fork. Still, I kept with ACS.
>
>
> On Tue, Feb 7, 2017 at 3:54 AM, Francois Scheurer <
> francois.scheu...@everyware.ch> wrote:
>
>> Dear Rafael
>>
>>
>> We disabled temporarily the HA Flag on these VM's by updating the DB
>> directly: update vm_instance set ha_enabled=0 where removed is null and
>> id=1178 or id=1240 or id=1690 or id=1736;
>>
>> We kept the test VM i-189-1177-VM with HA Flag.
>>
>> Restarting the MS + cleaning the op_it_work table will hopefully fix the
>> issue. And if the issue is persisting, then only the i-189-1177-VM should
>> be affected, hence mitigating the impact.
>>
>> In the other hand if everything run fine we would thereafter re-enable
>> the HA Flag on the problematic VM's.
>>
>>
>> Do you think it is safe to run this update?: update op_it_work set
>> step="Done" where step!="Done";
>>
>>
>> Best Regards
>>
>> Francois Scheurer
>>
>>
>>
>> PS: BTW, do you already heard about the Cloudstack Fork from Schuberg
>> Philis named Cosmic? 
>> https://github.com/MissionCriticalCloud/cosmic
>>
>>
>>
>>
>>
>> On 06.02.2017 16:41, Rafael Weingärtner wrote:
>>
>> I totally understand your position of not being able to debug the ACS in
>> production ;)
>>
>>
>> There is one bit I quite did not understand. If you have disabled the HA
>> for VMs, why do you still need to change data on the data base? How did you
>> disabled the HA of

Re: VPC Site to Site VPN

[Rafael Weingärtner]

Isn't this only a matter of changing the parameter "site2site.vpn.vpnga"?
The number 4 seems to be an arbitrarily small number that was set by a
developer; maybe 4 was the best amount for the developer test/production
environment.

Have you tested changing the parameter to 20, and then creating 20
connections and checking the performance?
Maybe for 20 connections the system VM that handles these connections will
need more computing resources; but, this is also a matter of changing the
service offering for this VM type.

On Tue, Feb 7, 2017 at 10:30 AM, Gian Paolo Buono 
wrote:

> Hi,
>
> I have need to implement a vpn site to site of my domain cloudstak with
> 20 external remote gateway/network. I have notice taht the default
> maximum number of VPN connection to the VPN gateway is 4. Can I increase
> this value to 20, can the VPC  handle this traffic or which is the
> suggested alternative...Pfsense ?
>
> Thanks, bye
> Gian Paolo




-- 
Rafael Weingärtner

VPC Site to Site VPN

[Gian Paolo Buono]

Hi,

I have need to implement a vpn site to site of my domain cloudstak with  
20 external remote gateway/network. I have notice taht the default 
maximum number of VPN connection to the VPN gateway is 4. Can I increase 
this value to 20, can the VPC  handle this traffic or which is the 
suggested alternative...Pfsense ?

Thanks, bye
Gian Paolo

Re: cs 4.5.1, vm stopped when putting xen hosts in maintenance

[Rafael Weingärtner]

So, you changed the flags directly on "vm_instance". Why did not you change
the service offering of these VMs using ACS UI or API? For me, it feels
that the best approach would be to change the service offering of the
affected VMs (the HA flag is first configured at the service offering). You
could assign a service offering without HA; I mean, it would be a matter of
creating a similar service offering (same configurations to the one they
are using right now), but without HA.

I am assuming that the change directly on “vm_instance” did not solve your
problem, right? That is why you want to change some data at “op_it_work”. I
did a quick look at “op_it_work”, and it seems that this table may be used
for more things than HA. Here is a tip for complex production system, never
run update SQLs in production environments without a very, very restrictive
and well thought “where” clause. In my opinion this is not restrictive
enough (update op_it_work set step="Done" where step!="Done"). I would
advise you not to go further and change configurations directly on database
(unless you have a dev team that is capable of reading, debugging, and work
with ACS source code). ACS has over 500k lines of java code, and God knows
how many lines of shell, python and other languages, software and systems
that may be used; it is pretty hard sometimes to get a clear picture of how
a change in the database will reverberate in the whole system without
knowing in details the installation configurations and underlying
structures (networking, hypervisors, storage systems and other pieces).
However, if you want to proceed with that, I would suggest you adding some
extra filters to that “where” clause. What about adding the “instance id”
there?

PS: BTW, do you already heard about the Cloudstack Fork from Schuberg
Philis named Cosmic? https://github.com/MissionCriticalCloud/cosmic
Yes, I have. They were part of the active ACS community back there. I was
already here when they created the fork. Still, I kept with ACS.


On Tue, Feb 7, 2017 at 3:54 AM, Francois Scheurer <
francois.scheu...@everyware.ch> wrote:

> Dear Rafael
>
>
> We disabled temporarily the HA Flag on these VM's by updating the DB
> directly: update vm_instance set ha_enabled=0 where removed is null and
> id=1178 or id=1240 or id=1690 or id=1736;
>
> We kept the test VM i-189-1177-VM with HA Flag.
>
> Restarting the MS + cleaning the op_it_work table will hopefully fix the
> issue. And if the issue is persisting, then only the i-189-1177-VM should
> be affected, hence mitigating the impact.
>
> In the other hand if everything run fine we would thereafter re-enable the
> HA Flag on the problematic VM's.
>
>
> Do you think it is safe to run this update?: update op_it_work set
> step="Done" where step!="Done";
>
>
> Best Regards
>
> Francois Scheurer
>
>
>
> PS: BTW, do you already heard about the Cloudstack Fork from Schuberg
> Philis named Cosmic? https://github.com/MissionCriticalCloud/cosmic
>
>
>
>
>
> On 06.02.2017 16:41, Rafael Weingärtner wrote:
>
> I totally understand your position of not being able to debug the ACS in
> production ;)
>
>
> There is one bit I quite did not understand. If you have disabled the HA
> for VMs, why do you still need to change data on the data base? How did you
> disabled the HA of VMs?
>
> On Mon, Feb 6, 2017 at 5:04 AM, Francois Scheurer <
> francois.scheu...@everyware.ch> wrote:
>
>> Dear Rafael / Dear CS contributors
>>
>>
>> I tried the following:
>>
>> 1. stop all MS
>> 2. backup DB
>> 3. select * from op_ha_work where type="Migration" and mgmt_server_id
>> is null and taken is null;
>> 4. delete from op_ha_work where type="Migration" and mgmt_server_id
>> is null and taken is null;
>> 5. start all MS
>>
>> But actually after the MS stopped the op_ha_work table was automatically
>> cleaned, making my steps useless.
>> After the MS started the op_ha_work table was filled again and related
>> VM's were stopped several times (3-7 times), requiring high attention to
>> manually restart them on each stop.
>>
>> I later found another table that could be the cause of the issue:
>>
>> mysql> select op_it_work.*,vm_instance.instance_name,vm_instance.removed
>> from op_it_work join vm_instance where op_it_work.step!="Done" and
>> op_it_work.instance_id=vm_instance.id order by instance_id;
>> +--++---
>> -+---+---+--
>> --+---++-+--
>> -+-+---+-+
>> | id   | mgmt_server_id | created_at |
>> thread| type  | vm_type| step  |
>> updated_at | instance_id | resource_type | resource_id | instance_name |
>> removed |
>> +--++---
>> -+---+---+--
>> --+--

Re: Cloudstack 4.7 Secondary Storage problem

[merlin]

Hi,

So I was able to fix the error (Secondary storage shows 0 in dashboard). I
simply logged into the ssvm and ran the health checks, which said NFS
currently not mounted. So I flushed the iptables on the host of my
cloudstack setup( which also happens to be the NFS server and Management
server) and then retried the command "mount" in ssvm and it had mounted.
So I restarted the services and now I am able to view the Secondary
Storage 6.33GB/50GB.

However, When I try too download and ISO or upload. It fails.
When I try to download the default XS-tools.iso it gives me an error
saying "Unable to extract the ISO xs-tools.iso as it resides on host and
not on SSVM"
If I try to upload any iso,it again fails and in the status there is
nothing that is updated.

When I checked the ssvm troubleshooting steps
It read that I need to fix up my secstorage.allowed.internal.sites.
Now the problem is that I have given it the ip of my host(which also acts
as a Management server,NFS server and Mysql Database) : -10.8.19.111/32
While the range of ips I gave to Pods and guest VMs was 10.8.170.x range.
Which means that even ssvm has an IP in that range.
What should I do?



> Merlin,
>
> Can you actually ping 10.8.19.111? Have you set up the NFS shares
> according to CloudStack specification ( /export
> (rw,async,no_root_squash,no_subtree_check) )? What is the contents
> of your /etc/exports file?
>
> In short it looks like you have either a connectivity or permissions issue
> – which is unrelated to CloudStack. Until you have this sorted you
> won’t be able to spin up system VMs etc. With regards to ACLs on your
> NFS share – keep in mind you need to open this up to any IP address in
> your management IP range – since the SSVM can appear on any IP in this.
>
> Regards,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
>
> On 18/01/2017, 12:10, "mer...@students.iitmandi.ac.in"
>  wrote:
>
> It did not throw me any error this time, however, there was no
> response
> either. The cursor just kept blinking for a very long time. untill I
> did
> ctrl+c.
>
> Also, a brief observation: when I write mount in my nfs server:
>
> I see three lines at the end
> 10.8.19.111:/export/primary on
> /mnt/38454858-08bb-3e92-88c9-354465ad3715
> type nfs
> (rw,noac,actimeo=0,vers=4,addr=10.8.19.111,clientaddr=10.8.19.111)
> 10.8.19.111:/export/primary on /mnt/primary type nfs
> 
> (rw,rsize=8192,wsize=8192,timeo=14,intr,vers=3,noac,actimeo=0,addr=10.8.19.111)
> 10.8.19.111:/export/secondary on /mnt/secondary type nfs
> 
> (rw,rsize=8192,wsize=8192,timeo=14,intr,vers=3,noac,actimeo=0,addr=10.8.19.111)
>
> Which used to be just 2 when I was using cloudstack version 4.4. I
> recently installed 4.8 and I see this after clean installation.
>
> > try "mount -t nfs 10.8.19.111:/export/secondary/ /tmp/secondary"
> >
> > *Vivek Kumar*
> > Virtualization and Cloud Consultant
> >
> > [image: http://www.indiqus.com/images/logo.jpg]
> 
> > *I*ndi*Q*us Technologies Pvt Ltd
> > A-98, LGF, C.R.Park, New Delhi - 110019
> > *O* +91 11 4055 1411 | *M* +91 7503460090
> > www.indiqus.com  
> >
> > On Wed, Jan 18, 2017 at 5:23 PM, 
> wrote:
> >
> >> Hi,
> >>
> >> I tried like you said: here is the result
> >>
> >> root@s-2-VM:~# mount -t NFS 10.8.19.111:/export/secondary/
> >> /tmp/secondary/
> >> mount: unknown filesystem type 'NFS'
> >> root@s-2-VM:~# mount -t  10.8.19.111:/export/secondary/
> /tmp/secondary/
> >> nfs
> >> mount: mount point nfs does not exist
> >>
> >> I might be running it wrong. Also, i checked the ssvm for the nfs
> >> package
> >> and neither was it there, nor could i download.
> >> Please help
> >>
> >>
> >> > Hi Merlin,
> >> >
> >> > What Paul means is you need to log on to the SSVM, then do
> something
> >> like
> >> > this:
> >> >
> >> > mkdir /tmp/secondary
> >> > mount –t NFS ://
> /tmp/secondary
> >> >
> >> > This will tell you if your NFS permissions are configured
> correctly.
> >> >
> >> > Regards,
> >> > Dag Sonstebo
> >> > Cloud Architect
> >> > ShapeBlue
> >> >
> >> > On 18/01/2017, 09:29, "mer...@students.iitmandi.ac.in"
> >> >  wrote:
> >> >
> >> > Also when you say mount from ssvm. Does it mean this?
> >> > Log into ssvm and run "mount":- the reply is down below. I am
> not
> >> able
> >> > to
> >> > understand much even from the documentation,
> >> > I am sorry for the lack of full knowledge in this concern.
> >> >
> >> >
> >> > sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
> >> > proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
> >> > udev on /dev type devtmpfs
> >> > (rw,relatime,size=10240k,nr_inodes=62128,mode=755)
> >

Re: MariaDB/Galera Cluster with CloudStack Management

[Dag Sonstebo]

Hi Tyler,

CloudStack is fully cluster aware – as you add more management servers they 
distribute tasks between them automatically, so you shouldn’t have to worry 
about this.

When you set up HAproxy keep in mind the session stickiness settings - 
http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.9/reliability.html


Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue

On 07/02/2017, 00:12, "Tyler Wilson"  wrote:

Hey All,

I am in the process of setting up 3x management servers behind keepalived +
haproxy on Ubuntu 16.04 using a VIP and had a couple questions.

Is CloudStack vulnerable to this issue:

https://severalnines.com/blog/avoiding-deadlocks-galera-set-haproxy-single-node-writes-and-multi-node-reads
? Or are there any other issues/pitfalls I should expect?

Thanks for your help!



dag.sonst...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue