Re: [4.11] Management to VR connection issues

[Rene Moser]

Hi again

We found the main problem.

== cloud-postinit hang

When having many iptables rules resulting in cloud-postinit to hang for
10min unless it was killed by systemd. As a result the ssh daemon was
not started for 10 min because it is configured to be started after
cloud-postinit.

It seems the issue was already fixed by
https://github.com/apache/cloudstack/commit/ce67726c6d3db6e7db537e76da6217c5d5f4b10e

== VR still needs manual reboot

However, we still notice adapter changes after a reboot: see before
after screenshots of "ip addr" in
https://photos.app.goo.gl/9XsjOJjLqQ9SRjYV2. We still need to manually
reboot the VR to make the network actually working.

== VR has too many adapters?

Next thing we noticed there are many network adapters (NICs) for this
non-vpc router (see screenshot of the vcenter in
https://photos.app.goo.gl/9XsjOJjLqQ9SRjYV2). Adapter 4 and 5 seem
unnecessary. Any comments on that?

== VR with 256 MB RAM dows not work

Next issue we found is, that the VR must have more than 256MB RAM.
Otherwise systemd will complain the daemon can not be reloaded, because
the ram disk of /run has too less space.

Feb 23 16:24:36 r-413-VM postinit.sh[1089]: Failed to reload daemon:
Refusing to reload, not enough space available on /run/systemd.
Currently, 8.6M are free, but a safety buffer of 16.0M is enforced.
root@r-413-VM:~# df -h /run/
Filesystem  Size  Used Avail Use% Mounted on
tmpfs16M  7.2M  8.7M  46% /run

Increaing to 512MB RAM helped:

root@r-413-VM:~# df -h /run/
Filesystem  Size  Used Avail Use% Mounted on
tmpfs41M  7.8M   34M  19% /run

Unsure if this can be tuned on systemd level, didn't find a way yet.

== VR API Command timeouts

When executing command related to VR, e.g. restart network, start/stop
router the command won't reach the vcenter api, and times out. We are
unsure yet, why.

== VR minor fixes

Next we fixed 2 minor things along.

* rsyslogd config syntax issue
* IMHO we should start apache2 also after cloud-postinit

Also see https://github.com/apache/cloudstack/pull/2468

Regards
René

RE: [4.11] Management to VR connection issues

[Paul Angus]

Rene,
Have you checked the OS getting applied on vCenter?
A lot of the issues went away once I changed the OS when testing over the 
weekend.

Kind regards,

Paul Angus

paul.an...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 


-Original Message-
From: Rene Moser [mailto:m...@renemoser.net] 
Sent: 26 February 2018 10:22
To: users@cloudstack.apache.org; d...@cloudstack.apache.org
Subject: Re: [4.11] Management to VR connection issues

Hi again

We found the main problem.

== cloud-postinit hang

When having many iptables rules resulting in cloud-postinit to hang for 10min 
unless it was killed by systemd. As a result the ssh daemon was not started for 
10 min because it is configured to be started after cloud-postinit.

It seems the issue was already fixed by
https://github.com/apache/cloudstack/commit/ce67726c6d3db6e7db537e76da6217c5d5f4b10e

== VR still needs manual reboot

However, we still notice adapter changes after a reboot: see before after 
screenshots of "ip addr" in https://photos.app.goo.gl/9XsjOJjLqQ9SRjYV2. We 
still need to manually reboot the VR to make the network actually working.

== VR has too many adapters?

Next thing we noticed there are many network adapters (NICs) for this non-vpc 
router (see screenshot of the vcenter in 
https://photos.app.goo.gl/9XsjOJjLqQ9SRjYV2). Adapter 4 and 5 seem unnecessary. 
Any comments on that?

== VR with 256 MB RAM dows not work

Next issue we found is, that the VR must have more than 256MB RAM.
Otherwise systemd will complain the daemon can not be reloaded, because the ram 
disk of /run has too less space.

Feb 23 16:24:36 r-413-VM postinit.sh[1089]: Failed to reload daemon:
Refusing to reload, not enough space available on /run/systemd.
Currently, 8.6M are free, but a safety buffer of 16.0M is enforced.
root@r-413-VM:~# df -h /run/
Filesystem  Size  Used Avail Use% Mounted on
tmpfs16M  7.2M  8.7M  46% /run

Increaing to 512MB RAM helped:

root@r-413-VM:~# df -h /run/
Filesystem  Size  Used Avail Use% Mounted on
tmpfs41M  7.8M   34M  19% /run

Unsure if this can be tuned on systemd level, didn't find a way yet.

== VR API Command timeouts

When executing command related to VR, e.g. restart network, start/stop router 
the command won't reach the vcenter api, and times out. We are unsure yet, why.

== VR minor fixes

Next we fixed 2 minor things along.

* rsyslogd config syntax issue
* IMHO we should start apache2 also after cloud-postinit

Also see https://github.com/apache/cloudstack/pull/2468

Regards
René

CloudStack Meetup - Frankfurt, Wednesday 28th of February

[Swen - swen.io]

Hi @all,

just a reminder for the first German CloudStack Meetup this year in
Frankfurt. Speeches will be mainly in English, so do not hesitate to visit
this event even if you are not speaking German!

https://www.meetup.com/de-DE/german-CloudStack-user-group/events/246861772/


Mit freundlichen Grüßen / With kind regards,

Swen

Re: CloudStack Meetup - Frankfurt, Wednesday 28th of February

[Melanie Desaive]

Sehr schön,

ich habe gerade meine Bahntickets und Hotebuchung bekommen. :)

Bin also auch beim Abendprogramm dabei.

Liebe Grüße,

Melanie

Am 26.02.2018 um 11:58 schrieb Swen - swen.io:
> Hi @all,
> 
> just a reminder for the first German CloudStack Meetup this year in
> Frankfurt. Speeches will be mainly in English, so do not hesitate to visit
> this event even if you are not speaking German!
> 
> https://www.meetup.com/de-DE/german-CloudStack-user-group/events/246861772/
> 
> 
> Mit freundlichen Grüßen / With kind regards,
> 
> Swen
> 
> 

-- 
--

Heinlein Support GmbH
Linux: Akademie - Support - Hosting

http://www.heinlein-support.de
Tel: 030 / 40 50 51 - 0
Fax: 030 / 40 50 51 - 19

Zwangsangaben lt. §35a GmbHG:
HRB 93818 B / Amtsgericht Berlin-Charlottenburg,
Geschäftsführer: Peer Heinlein  -- Sitz: Berlin



signature.asc
Description: OpenPGP digital signature

Re: [4.11] Management to VR connection issues

[Rohit Yadav]

Hi Rene,


- I think on the general issue of slow iptables rules application, we need to 
fix that. Does it help to increase aggregation timeouts?


- If waiting for ssh and apache2 as part of post-init solves the issue, this 
would require a new systemvmtemplate as the systemd scripts cannot be changed 
or make effect during first boot.


- I think the additional nics always used to show up for vmware, there is a 
global setting to configure this (extra nics for vmware, probably because older 
versions did not support dynamic nic addition on vmware vrs).


- For VR timeouts, see logs and check if from management server host you're 
able to SSH into the VR using the private IP and port 3922. See the 
troubleshooting wiki: 
https://cwiki.apache.org/confluence/display/CLOUDSTACK/SSVM%2C+templates%2C+Secondary+storage+troubleshooting


- Can you share/check which processes are consuming the RAM, 256MB ram is 
usually enough for non-redundant VRs. (share output of top or check using 
htop?). Make sure to use a latest Linux version (any Debian variant such as 
Debian 8, 9 or Ubuntu 16.04+ may also work). The issue is vCenter/ESXi 6.5 for 
some reason, gives lower RAM compared to 6.0 and 5.5 and has poor support for 
legacy os. I had faced/found this issue while testing redundant VRs which take 
more RAM usually than normal VRs.


- Rohit






From: Rene Moser 
Sent: Monday, February 26, 2018 11:22:27 AM
To: users@cloudstack.apache.org; d...@cloudstack.apache.org
Subject: Re: [4.11] Management to VR connection issues

Hi again

We found the main problem.

== cloud-postinit hang

When having many iptables rules resulting in cloud-postinit to hang for
10min unless it was killed by systemd. As a result the ssh daemon was
not started for 10 min because it is configured to be started after
cloud-postinit.

It seems the issue was already fixed by
https://github.com/apache/cloudstack/commit/ce67726c6d3db6e7db537e76da6217c5d5f4b10e

== VR still needs manual reboot

However, we still notice adapter changes after a reboot: see before
after screenshots of "ip addr" in
https://photos.app.goo.gl/9XsjOJjLqQ9SRjYV2. We still need to manually
reboot the VR to make the network actually working.

== VR has too many adapters?

Next thing we noticed there are many network adapters (NICs) for this
non-vpc router (see screenshot of the vcenter in
https://photos.app.goo.gl/9XsjOJjLqQ9SRjYV2). Adapter 4 and 5 seem
unnecessary. Any comments on that?

== VR with 256 MB RAM dows not work

Next issue we found is, that the VR must have more than 256MB RAM.
Otherwise systemd will complain the daemon can not be reloaded, because
the ram disk of /run has too less space.

Feb 23 16:24:36 r-413-VM postinit.sh[1089]: Failed to reload daemon:
Refusing to reload, not enough space available on /run/systemd.
Currently, 8.6M are free, but a safety buffer of 16.0M is enforced.
root@r-413-VM:~# df -h /run/
Filesystem  Size  Used Avail Use% Mounted on
tmpfs16M  7.2M  8.7M  46% /run

Increaing to 512MB RAM helped:

root@r-413-VM:~# df -h /run/
Filesystem  Size  Used Avail Use% Mounted on
tmpfs41M  7.8M   34M  19% /run

Unsure if this can be tuned on systemd level, didn't find a way yet.

== VR API Command timeouts

When executing command related to VR, e.g. restart network, start/stop
router the command won't reach the vcenter api, and times out. We are
unsure yet, why.

== VR minor fixes

Next we fixed 2 minor things along.

* rsyslogd config syntax issue
* IMHO we should start apache2 also after cloud-postinit

Also see https://github.com/apache/cloudstack/pull/2468

Regards
René

rohit.ya...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

FW: Building saml plugin

[Pallavi Dumbre]

Hi All,

I have followed steps in given following FS for building cloudstack with saml 
support but getting errors during jetty startup
https://cwiki.apache.org/confluence/display/CLOUDSTACK/SAML+2.0+Plugin

Errors:
2018-02-26 17:25:12.656:WARN:oejw.WebAppContext:main: Failed startup of context 
o.e.j.m.p.JettyWebAppContext@21f9c6ea{/client,file:/home/user/acp/Accelerite-CloudPlatform/client/target/classes/META-INF/webapp/,STARTING}{file:/home/user/acp/Accelerite-CloudPlatform/client/target/classes/META-INF/webapp/}
java.lang.NullPointerException
at 
org.apache.cloudstack.utils.security.CertUtils.pemToPrivateKey(CertUtils.java:130)
at 
org.apache.cloudstack.saml.SAML2AuthManagerImpl.initSP(SAML2AuthManagerImpl.java:172)
at 
org.apache.cloudstack.saml.SAML2AuthManagerImpl.setup(SAML2AuthManagerImpl.java:378)
at 
org.apache.cloudstack.saml.SAML2AuthManagerImpl.start(SAML2AuthManagerImpl.java:129)


Also checked keystore database entry in cloud db and seems to be fine. Please 
let me know if anyone has idea related to same.

Thanks,
Pallavi
DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Accelerite, a Persistent Systems business. It is intended only for 
the use of the individual or entity to which it is addressed. If you are not 
the intended recipient, you are not authorized to read, retain, copy, print, 
distribute or use this message. If you have received this communication in 
error, please notify the sender and delete all copies of this message. 
Accelerite, a Persistent Systems business does not accept any liability for 
virus infected mails.

Re: System VM Issues 4.11?

[Kristian Liivak]

Rohit,

Systemvms are 4.11, i can see that in console before login :)
cat /proc/cmdline returnes some info. Please see attached pics

Management sevrer is working just fine. I dont have however autostart on in 
upgrade time.
Im using old centos7 management server. Once allready made upgrade from ACS 4.9 
to 4.10
Now from 4.10 to 4.11..

Is there maybe point to use clean 4.11 management server to resolve system vm 
issue ?
And do i need to have HVM flag cheked with 4.11 system management template 
install ?
There is nothing about it in  instructions. However i have tried both ways 
without any success.


Lugupidamisega / Regards
 
Kristian Liivak

Tegevjuht / Executive director

WaveCom As
Endla 16, 10142 Tallinn
Estonia
Tel: +3726850001
Gsm: +37256850001
E-mail: k...@wavecom.ee
Skype: kristian.liivak
http://www.wavecom.ee
http://www.facebook.com/wavecom.ee

- Original Message -
From: "rohit yadav" 
To: "users" 
Sent: Sunday, February 25, 2018 4:04:54 PM
Subject: Re: System VM Issues 4.11?

Hi Kristian,


Thanks for sharing the logs. Looks like systemvm template failed to grab the 
cmdline options from XenServer to patch itself, reconfigure the nics. The 
ifconfig does not return at least the control nic. Can you check what `cat 
/etc/cloudstack-release` returns on the systemvm (this is to test that the 
systemvm template is indeed the 4.11 one).


On the systemvm, see if cat /proc/cmdline returns anything? It also looks to me 
that the mgmt server is failing to run check for cloudstack-management start 
failures, see /etc/default/cloudstack-management and the 
cloudstack-management.service to find why the process fails, perhaps the 
classpath (or JAVA_HOME/java path) needs to be fixed?


Lastly, can you also comment if you're using a new host/server to run mgmt 
server, or you did an in-place upgrade?


- Rohit






From: Kristian Liivak 
Sent: Saturday, February 24, 2018 8:31:35 AM
To: users
Subject: Re: System VM Issues 4.11?

Hi Rohit,

I managed fix absent of hosts. It was host tags related afterall. db host_tags 
contained tags before xenserver update in past. After dropping them update 
succeeded until new systemvm deployment.
Like  others said in thread, systemvm are successfully deployd, but not network 
will be started and they will not come up.
I gathered logs , maybe you have time to check.
https://wavecom.ee/files/cloudstack.rar


Lugupidamisega / Regards

Kristian Liivak

Tegevjuht

WaveCom As
Endla 16, 10142 Tallinn
Estonia
Tel: +3726850001
Gsm: +37256850001
E-mail: k...@wavecom.ee
Skype: kristian.liivak
http://www.wavecom.ee
http://www.facebook.com/wavecom.ee


rohit.ya...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

- Original Message -
From: "rohit yadav" 
To: "users" 
Sent: Wednesday, February 21, 2018 6:25:11 PM
Subject: Re: System VM Issues 4.11?

Hi Kristian,


Do you get any results if you execute the listRouters/listHosts apis using say 
a CLI such as cloudmonkey as the root admin?


🐵 > list routers listall=true


🐵 > list hosts

Also, do you have any modifications to the UI, or webserver/middleware? I 
looked at the logs, and I could not find any exceptions or anything suspicious.

About host tags, clearing the tags and doing reconnect causes cloudstack to 
scp/copy scripts and files to xenserver host which usually happens when you 
upgrade cloudstack. It's not an issue for you as in your logs I see such 
activities after cloudstack upgrades. (otherwise, it's done using, something 
like:

# for host in $(xe host-list | grep ^uuid | awk '{print $NF}') ; do xe 
host-param-clear uuid=$host param-name=tags; done; )

- Rohit






From: Kristian Liivak 
Sent: Tuesday, February 20, 2018 10:25:42 PM
To: users
Subject: Re: System VM Issues 4.11?

Hi Rohit,

Made another shot.

Recreated all systemvm before update and get rid most of errors had before.

Like before no host nor systemvm was present in gui
However i saw correct numbers under infrastructure.
I attached some pics form gui  host and systemvm  selection.

I did destroyd old systemvms last time
New ones started without network.
New templates were present and new systemvm are created from 4.11 templates, 
checked from xen console.

My management server resides in different vmware enviroment, however it have 
same management network like all my xenservers.

I started with 4.9, where dynamic roles  were allready introduced, but i 
doublechecked.

My old enviroment was configured with different port using custom ssl, so 
therefore no cache.

Enviroment is only xenserver7.0 hosts

Uploaded new management logs and also /var/log/messages where update logs are 
present https://wavecom.ee/files/logs.rar

Maybe you can find time to check..

And i can allways rollback management where i was to have additional info

PS what you mean about : reset

Re: System VM Issues 4.11?

[Rohit Yadav]

.Hi Kristian,


Thanks for sharing. The mailing list strips attachments so we did not receive 
any of your images/attachments, kindly share them on any temporary hosting 
service.


I'm not sure what you mean by cleaning the management server. After your 
systemvm is up, does it work if you manually reboot the systemvm - i.e. does 
its nics come up after the manual reboot and you can see agents running (such 
as cpvm, ssvm?).


I think the db-upgrader changes the HVM flag, so it does not matter whether you 
had ticked the template or not before upgrading -- you can try but I think it 
should not make any difference.



- Rohit






From: Kristian Liivak 
Sent: Monday, February 26, 2018 2:46:14 PM
To: users
Subject: Re: System VM Issues 4.11?

Rohit,

Systemvms are 4.11, i can see that in console before login :)
cat /proc/cmdline returnes some info. Please see attached pics

Management sevrer is working just fine. I dont have however autostart on in 
upgrade time.
Im using old centos7 management server. Once allready made upgrade from ACS 4.9 
to 4.10
Now from 4.10 to 4.11..

Is there maybe point to use clean 4.11 management server to resolve system vm 
issue ?
And do i need to have HVM flag cheked with 4.11 system management template 
install ?
There is nothing about it in  instructions. However i have tried both ways 
without any success.


Lugupidamisega / Regards

Kristian Liivak

Tegevjuht / Executive director

WaveCom As
Endla 16, 10142 Tallinn
Estonia
Tel: +3726850001
Gsm: +37256850001
E-mail: k...@wavecom.ee
Skype: kristian.liivak
http://www.wavecom.ee
http://www.facebook.com/wavecom.ee


rohit.ya...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

- Original Message -
From: "rohit yadav" 
To: "users" 
Sent: Sunday, February 25, 2018 4:04:54 PM
Subject: Re: System VM Issues 4.11?

Hi Kristian,


Thanks for sharing the logs. Looks like systemvm template failed to grab the 
cmdline options from XenServer to patch itself, reconfigure the nics. The 
ifconfig does not return at least the control nic. Can you check what `cat 
/etc/cloudstack-release` returns on the systemvm (this is to test that the 
systemvm template is indeed the 4.11 one).


On the systemvm, see if cat /proc/cmdline returns anything? It also looks to me 
that the mgmt server is failing to run check for cloudstack-management start 
failures, see /etc/default/cloudstack-management and the 
cloudstack-management.service to find why the process fails, perhaps the 
classpath (or JAVA_HOME/java path) needs to be fixed?


Lastly, can you also comment if you're using a new host/server to run mgmt 
server, or you did an in-place upgrade?


- Rohit






From: Kristian Liivak 
Sent: Saturday, February 24, 2018 8:31:35 AM
To: users
Subject: Re: System VM Issues 4.11?

Hi Rohit,

I managed fix absent of hosts. It was host tags related afterall. db host_tags 
contained tags before xenserver update in past. After dropping them update 
succeeded until new systemvm deployment.
Like  others said in thread, systemvm are successfully deployd, but not network 
will be started and they will not come up.
I gathered logs , maybe you have time to check.
https://wavecom.ee/files/cloudstack.rar


Lugupidamisega / Regards

Kristian Liivak

Tegevjuht

WaveCom As
Endla 16, 10142 Tallinn
Estonia
Tel: +3726850001
Gsm: +37256850001
E-mail: k...@wavecom.ee
Skype: kristian.liivak
http://www.wavecom.ee
http://www.facebook.com/wavecom.ee


rohit.ya...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue



- Original Message -
From: "rohit yadav" 
To: "users" 
Sent: Wednesday, February 21, 2018 6:25:11 PM
Subject: Re: System VM Issues 4.11?

Hi Kristian,


Do you get any results if you execute the listRouters/listHosts apis using say 
a CLI such as cloudmonkey as the root admin?


🐵 > list routers listall=true


🐵 > list hosts

Also, do you have any modifications to the UI, or webserver/middleware? I 
looked at the logs, and I could not find any exceptions or anything suspicious.

About host tags, clearing the tags and doing reconnect causes cloudstack to 
scp/copy scripts and files to xenserver host which usually happens when you 
upgrade cloudstack. It's not an issue for you as in your logs I see such 
activities after cloudstack upgrades. (otherwise, it's done using, something 
like:

# for host in $(xe host-list | grep ^uuid | awk '{print $NF}') ; do xe 
host-param-clear uuid=$host param-name=tags; done; )

- Rohit






From: Kristian Liivak 
Sent: Tuesday, February 20, 2018 10:25:42 PM
To: users
Subject: Re: System VM Issues 4.11?

Hi Rohit,

Made another shot.

Recreated all systemvm before update and get rid most of errors had before.

Like b

Re: Building saml plugin

[Rohit Yadav]

Pallavi,


I think the build system works okay we've not seen any build failures on recent 
4.11 and master branch. Which branch are you using? If this is closed-source 
ACP, we may not be able to help you, however, keep exploring (such as put 
breakpoints) on the code you see in the stack trace and see what may be causing 
those failures.


- Rohit






From: Pallavi Dumbre 
Sent: Monday, February 26, 2018 2:19:31 PM
To: users@cloudstack.apache.org
Subject: FW: Building saml plugin


Hi All,

I have followed steps in given following FS for building cloudstack with saml 
support but getting errors during jetty startup
https://cwiki.apache.org/confluence/display/CLOUDSTACK/SAML+2.0+Plugin

Errors:
2018-02-26 17:25:12.656:WARN:oejw.WebAppContext:main: Failed startup of context 
o.e.j.m.p.JettyWebAppContext@21f9c6ea{/client,file:/home/user/acp/Accelerite-CloudPlatform/client/target/classes/META-INF/webapp/,STARTING}{file:/home/user/acp/Accelerite-CloudPlatform/client/target/classes/META-INF/webapp/}
java.lang.NullPointerException
at 
org.apache.cloudstack.utils.security.CertUtils.pemToPrivateKey(CertUtils.java:130)
at 
org.apache.cloudstack.saml.SAML2AuthManagerImpl.initSP(SAML2AuthManagerImpl.java:172)
at 
org.apache.cloudstack.saml.SAML2AuthManagerImpl.setup(SAML2AuthManagerImpl.java:378)
at 
org.apache.cloudstack.saml.SAML2AuthManagerImpl.start(SAML2AuthManagerImpl.java:129)


Also checked keystore database entry in cloud db and seems to be fine. Please 
let me know if anyone has idea related to same.

Thanks,
Pallavi
DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Accelerite, a Persistent Systems business. It is intended only for 
the use of the individual or entity to which it is addressed. If you are not 
the intended recipient, you are not authorized to read, retain, copy, print, 
distribute or use this message. If you have received this communication in 
error, please notify the sender and delete all copies of this message. 
Accelerite, a Persistent Systems business does not accept any liability for 
virus infected mails.

rohit.ya...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

Re: System VM Issues 4.11?

[Kristian Liivak]

Rohit,

Forgot about attachments.  Please see shot 
http://www.wavecom.ee/files/systemvm.PNG

Manual reboot to systemvm won´t help. Still no network. Is there maybe script 
to run initiate configuration provision again ?

I mean is there point to use new fresh centos 7 server with new install aws 
4.11 ? 
Currectly my managment server have been updated several times. 
But i understand systemvm configuration deplyment is done via vost anywway.

Lugupidamisega / Regards
 
Kristian Liivak

Tegevjuht

WaveCom As
Endla 16, 10142 Tallinn
Estonia
Tel: +3726850001
Gsm: +37256850001
E-mail: k...@wavecom.ee
Skype: kristian.liivak
http://www.wavecom.ee
http://www.facebook.com/wavecom.ee

- Original Message -
From: "rohit yadav" 
To: "users" 
Sent: Monday, February 26, 2018 4:03:04 PM
Subject: Re: System VM Issues 4.11?

.Hi Kristian,


Thanks for sharing. The mailing list strips attachments so we did not receive 
any of your images/attachments, kindly share them on any temporary hosting 
service.


I'm not sure what you mean by cleaning the management server. After your 
systemvm is up, does it work if you manually reboot the systemvm - i.e. does 
its nics come up after the manual reboot and you can see agents running (such 
as cpvm, ssvm?).


I think the db-upgrader changes the HVM flag, so it does not matter whether you 
had ticked the template or not before upgrading -- you can try but I think it 
should not make any difference.



- Rohit






From: Kristian Liivak 
Sent: Monday, February 26, 2018 2:46:14 PM
To: users
Subject: Re: System VM Issues 4.11?

Rohit,

Systemvms are 4.11, i can see that in console before login :)
cat /proc/cmdline returnes some info. Please see attached pics

Management sevrer is working just fine. I dont have however autostart on in 
upgrade time.
Im using old centos7 management server. Once allready made upgrade from ACS 4.9 
to 4.10
Now from 4.10 to 4.11..

Is there maybe point to use clean 4.11 management server to resolve system vm 
issue ?
And do i need to have HVM flag cheked with 4.11 system management template 
install ?
There is nothing about it in  instructions. However i have tried both ways 
without any success.


Lugupidamisega / Regards

Kristian Liivak

Tegevjuht / Executive director

WaveCom As
Endla 16, 10142 Tallinn
Estonia
Tel: +3726850001
Gsm: +37256850001
E-mail: k...@wavecom.ee
Skype: kristian.liivak
http://www.wavecom.ee
http://www.facebook.com/wavecom.ee


rohit.ya...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

- Original Message -
From: "rohit yadav" 
To: "users" 
Sent: Sunday, February 25, 2018 4:04:54 PM
Subject: Re: System VM Issues 4.11?

Hi Kristian,


Thanks for sharing the logs. Looks like systemvm template failed to grab the 
cmdline options from XenServer to patch itself, reconfigure the nics. The 
ifconfig does not return at least the control nic. Can you check what `cat 
/etc/cloudstack-release` returns on the systemvm (this is to test that the 
systemvm template is indeed the 4.11 one).


On the systemvm, see if cat /proc/cmdline returns anything? It also looks to me 
that the mgmt server is failing to run check for cloudstack-management start 
failures, see /etc/default/cloudstack-management and the 
cloudstack-management.service to find why the process fails, perhaps the 
classpath (or JAVA_HOME/java path) needs to be fixed?


Lastly, can you also comment if you're using a new host/server to run mgmt 
server, or you did an in-place upgrade?


- Rohit






From: Kristian Liivak 
Sent: Saturday, February 24, 2018 8:31:35 AM
To: users
Subject: Re: System VM Issues 4.11?

Hi Rohit,

I managed fix absent of hosts. It was host tags related afterall. db host_tags 
contained tags before xenserver update in past. After dropping them update 
succeeded until new systemvm deployment.
Like  others said in thread, systemvm are successfully deployd, but not network 
will be started and they will not come up.
I gathered logs , maybe you have time to check.
https://wavecom.ee/files/cloudstack.rar


Lugupidamisega / Regards

Kristian Liivak

Tegevjuht

WaveCom As
Endla 16, 10142 Tallinn
Estonia
Tel: +3726850001
Gsm: +37256850001
E-mail: k...@wavecom.ee
Skype: kristian.liivak
http://www.wavecom.ee
http://www.facebook.com/wavecom.ee


rohit.ya...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue



- Original Message -
From: "rohit yadav" 
To: "users" 
Sent: Wednesday, February 21, 2018 6:25:11 PM
Subject: Re: System VM Issues 4.11?

Hi Kristian,


Do you get any results if you execute the listRouters/listHosts apis using say 
a CLI such as cloudmonkey as the root admin?


🐵 > list routers listall=true


🐵 > list hosts

Also, do you have any modifications to the UI, or webserver/middleware?

Re: System VM Issues 4.11?

[Rohit Yadav]

Hi Kristian,


Which version of XenServer 7.x are you using? Please do note that XenServer 
7.3+ are not supported only, 7.0, 7.1 and 7.2 are. It looks like the expected 
cmdline argument is not sent/found in the systemvmtemplate which basically 
fails the deployment, however, I'm not sure about the exact cause and how to 
help you fix that.


- Rohit






From: Kristian Liivak 
Sent: Monday, February 26, 2018 3:20:34 PM
To: users
Subject: Re: System VM Issues 4.11?


Rohit,

Forgot about attachments.  Please see shot 
http://www.wavecom.ee/files/systemvm.PNG

Manual reboot to systemvm won´t help. Still no network. Is there maybe script 
to run initiate configuration provision again ?

I mean is there point to use new fresh centos 7 server with new install aws 
4.11 ?
Currectly my managment server have been updated several times.
But i understand systemvm configuration deplyment is done via vost anywway.

Lugupidamisega / Regards

Kristian Liivak

Tegevjuht

WaveCom As
Endla 16, 10142 Tallinn
Estonia
Tel: +3726850001
Gsm: +37256850001
E-mail: k...@wavecom.ee
Skype: kristian.liivak
http://www.wavecom.ee
http://www.facebook.com/wavecom.ee


rohit.ya...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

- Original Message -
From: "rohit yadav" 
To: "users" 
Sent: Monday, February 26, 2018 4:03:04 PM
Subject: Re: System VM Issues 4.11?

.Hi Kristian,


Thanks for sharing. The mailing list strips attachments so we did not receive 
any of your images/attachments, kindly share them on any temporary hosting 
service.


I'm not sure what you mean by cleaning the management server. After your 
systemvm is up, does it work if you manually reboot the systemvm - i.e. does 
its nics come up after the manual reboot and you can see agents running (such 
as cpvm, ssvm?).


I think the db-upgrader changes the HVM flag, so it does not matter whether you 
had ticked the template or not before upgrading -- you can try but I think it 
should not make any difference.



- Rohit






From: Kristian Liivak 
Sent: Monday, February 26, 2018 2:46:14 PM
To: users
Subject: Re: System VM Issues 4.11?

Rohit,

Systemvms are 4.11, i can see that in console before login :)
cat /proc/cmdline returnes some info. Please see attached pics

Management sevrer is working just fine. I dont have however autostart on in 
upgrade time.
Im using old centos7 management server. Once allready made upgrade from ACS 4.9 
to 4.10
Now from 4.10 to 4.11..

Is there maybe point to use clean 4.11 management server to resolve system vm 
issue ?
And do i need to have HVM flag cheked with 4.11 system management template 
install ?
There is nothing about it in  instructions. However i have tried both ways 
without any success.


Lugupidamisega / Regards

Kristian Liivak

Tegevjuht / Executive director

WaveCom As
Endla 16, 10142 Tallinn
Estonia
Tel: +3726850001
Gsm: +37256850001
E-mail: k...@wavecom.ee
Skype: kristian.liivak
http://www.wavecom.ee
http://www.facebook.com/wavecom.ee


rohit.ya...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue



- Original Message -
From: "rohit yadav" 
To: "users" 
Sent: Sunday, February 25, 2018 4:04:54 PM
Subject: Re: System VM Issues 4.11?

Hi Kristian,


Thanks for sharing the logs. Looks like systemvm template failed to grab the 
cmdline options from XenServer to patch itself, reconfigure the nics. The 
ifconfig does not return at least the control nic. Can you check what `cat 
/etc/cloudstack-release` returns on the systemvm (this is to test that the 
systemvm template is indeed the 4.11 one).


On the systemvm, see if cat /proc/cmdline returns anything? It also looks to me 
that the mgmt server is failing to run check for cloudstack-management start 
failures, see /etc/default/cloudstack-management and the 
cloudstack-management.service to find why the process fails, perhaps the 
classpath (or JAVA_HOME/java path) needs to be fixed?


Lastly, can you also comment if you're using a new host/server to run mgmt 
server, or you did an in-place upgrade?


- Rohit






From: Kristian Liivak 
Sent: Saturday, February 24, 2018 8:31:35 AM
To: users
Subject: Re: System VM Issues 4.11?

Hi Rohit,

I managed fix absent of hosts. It was host tags related afterall. db host_tags 
contained tags before xenserver update in past. After dropping them update 
succeeded until new systemvm deployment.
Like  others said in thread, systemvm are successfully deployd, but not network 
will be started and they will not come up.
I gathered logs , maybe you have time to check.
https://wavecom.ee/files/cloudstack.rar


Lugupidamisega / Regards

Kristian Liivak

Tegevjuht

WaveCom As
Endla 16, 10142 Tallinn
Estonia
Tel: +37268500

Re: System VM Issues 4.11?

[Kristian Liivak]

Hi

I have 2 pools with xenserver 7.0 (all patches installed) as AWS 4.10 not 
supporting higher version of xenserver. At least officially.
Does it make sence to upgrade on pool to 7.1 ? I  have 1 pool for testing only.


Lugupidamisega / Regards
 
Kristian Liivak

Tegevjuht / Executive director

WaveCom As
Endla 16, 10142 Tallinn
Estonia
Tel: +3726850001
Gsm: +37256850001
E-mail: k...@wavecom.ee
Skype: kristian.liivak
http://www.wavecom.ee
http://www.facebook.com/wavecom.ee

- Original Message -
From: "rohit yadav" 
To: "users" 
Sent: Monday, February 26, 2018 5:48:43 PM
Subject: Re: System VM Issues 4.11?

Hi Kristian,


Which version of XenServer 7.x are you using? Please do note that XenServer 
7.3+ are not supported only, 7.0, 7.1 and 7.2 are. It looks like the expected 
cmdline argument is not sent/found in the systemvmtemplate which basically 
fails the deployment, however, I'm not sure about the exact cause and how to 
help you fix that.


- Rohit






From: Kristian Liivak 
Sent: Monday, February 26, 2018 3:20:34 PM
To: users
Subject: Re: System VM Issues 4.11?


Rohit,

Forgot about attachments.  Please see shot 
http://www.wavecom.ee/files/systemvm.PNG

Manual reboot to systemvm won´t help. Still no network. Is there maybe script 
to run initiate configuration provision again ?

I mean is there point to use new fresh centos 7 server with new install aws 
4.11 ?
Currectly my managment server have been updated several times.
But i understand systemvm configuration deplyment is done via vost anywway.

Lugupidamisega / Regards

Kristian Liivak

Tegevjuht

WaveCom As
Endla 16, 10142 Tallinn
Estonia
Tel: +3726850001
Gsm: +37256850001
E-mail: k...@wavecom.ee
Skype: kristian.liivak
http://www.wavecom.ee
http://www.facebook.com/wavecom.ee


rohit.ya...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

- Original Message -
From: "rohit yadav" 
To: "users" 
Sent: Monday, February 26, 2018 4:03:04 PM
Subject: Re: System VM Issues 4.11?

.Hi Kristian,


Thanks for sharing. The mailing list strips attachments so we did not receive 
any of your images/attachments, kindly share them on any temporary hosting 
service.


I'm not sure what you mean by cleaning the management server. After your 
systemvm is up, does it work if you manually reboot the systemvm - i.e. does 
its nics come up after the manual reboot and you can see agents running (such 
as cpvm, ssvm?).


I think the db-upgrader changes the HVM flag, so it does not matter whether you 
had ticked the template or not before upgrading -- you can try but I think it 
should not make any difference.



- Rohit






From: Kristian Liivak 
Sent: Monday, February 26, 2018 2:46:14 PM
To: users
Subject: Re: System VM Issues 4.11?

Rohit,

Systemvms are 4.11, i can see that in console before login :)
cat /proc/cmdline returnes some info. Please see attached pics

Management sevrer is working just fine. I dont have however autostart on in 
upgrade time.
Im using old centos7 management server. Once allready made upgrade from ACS 4.9 
to 4.10
Now from 4.10 to 4.11..

Is there maybe point to use clean 4.11 management server to resolve system vm 
issue ?
And do i need to have HVM flag cheked with 4.11 system management template 
install ?
There is nothing about it in  instructions. However i have tried both ways 
without any success.


Lugupidamisega / Regards

Kristian Liivak

Tegevjuht / Executive director

WaveCom As
Endla 16, 10142 Tallinn
Estonia
Tel: +3726850001
Gsm: +37256850001
E-mail: k...@wavecom.ee
Skype: kristian.liivak
http://www.wavecom.ee
http://www.facebook.com/wavecom.ee


rohit.ya...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue



- Original Message -
From: "rohit yadav" 
To: "users" 
Sent: Sunday, February 25, 2018 4:04:54 PM
Subject: Re: System VM Issues 4.11?

Hi Kristian,


Thanks for sharing the logs. Looks like systemvm template failed to grab the 
cmdline options from XenServer to patch itself, reconfigure the nics. The 
ifconfig does not return at least the control nic. Can you check what `cat 
/etc/cloudstack-release` returns on the systemvm (this is to test that the 
systemvm template is indeed the 4.11 one).


On the systemvm, see if cat /proc/cmdline returns anything? It also looks to me 
that the mgmt server is failing to run check for cloudstack-management start 
failures, see /etc/default/cloudstack-management and the 
cloudstack-management.service to find why the process fails, perhaps the 
classpath (or JAVA_HOME/java path) needs to be fixed?


Lastly, can you also comment if you're using a new host/server to run mgmt 
server, or you did an in-place upgrade?


- Rohit






From: Kristian Liivak 
S

Re: [4.11] Management to VR connection issues

[Rene Moser]

On 02/26/2018 12:41 PM, Rohit Yadav wrote:

> - If waiting for ssh and apache2 as part of post-init solves the issue, this 
> would require a new systemvmtemplate as the systemd scripts cannot be changed 
> or make effect during first boot.

The waiting for ssh was not the issue, it was a result.

The hang of cloud-postinit caused by p.wait() when having a ton of
iptable rules was the issue. But this is addressed already. should be fine.

a systemctl list-jobs shows "no pending jobs" anymore, so the boot has
completed.

After that the VR should be accessable by SSH (3922) by managemement
right, but it is not.

Did you see  the changes after a reboot (please compare the screenshots
of the ip addr output I sent). After that reboot/network change, SSH
works...


> - I think the additional nics always used to show up for vmware, there is a 
> global setting to configure this (extra nics for vmware, probably because 
> older versions did not support dynamic nic addition on vmware vrs).

On 4.5.2, we only see 4 NICs. in 4.11 we see 5 of them. We were just
wondering if this could result in an issue. What global setting would
that be?


> - For VR timeouts, see logs and check if from management server host you're 
> able to SSH into the VR using the private IP and port 3922. See the 
> troubleshooting wiki: 
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/SSVM%2C+templates%2C+Secondary+storage+troubleshooting

Yes, after a manual reboot of the VR, we can SSH-in as I wrote. Without
a reboot of the VR, we get a "no route to host". So it seems not even an
arp ping is working.


> - Can you share/check which processes are consuming the RAM, 256MB ram is 
> usually enough for non-redundant VRs. (share output of top or check using 
> htop?). Make sure to use a latest Linux version (any Debian variant such as 
> Debian 8, 9 or Ubuntu 16.04+ may also work). The issue is vCenter/ESXi 6.5 
> for some reason, gives lower RAM compared to 6.0 and 5.5 and has poor support 
> for legacy os. I had faced/found this issue while testing redundant VRs which 
> take more RAM usually than normal VRs.

Using the shapeblue VR template (your template ;))

So the man docs says
https://manpages.debian.org/stretch/initscripts/tmpfs.5.en.html

unfortunately only a fstab entry worked for me, setting the
/etc/default/tmpfs didn't.

https://github.com/apache/cloudstack/pull/2468/commits/bd882a8f80763595a89a3b74330500e1965bfda3

Re: System VM Issues 4.11?

[Rohit Yadav]

Hi Kristian,


Thanks for sharing. I think your setup seems alright, there is no need to 
upgrade the pool to 7.1 to make it work -- it should have worked for you. 
However, you may choose to upgrade the pool to 7.1 which is lts.


If it's a test system, can you attempt to do a fresh ACS 4.10 install and try 
to re-upgrade? Perhaps something in your environment got messed up?


- Rohit






From: Kristian Liivak 
Sent: Monday, February 26, 2018 5:07:14 PM
To: users
Subject: Re: System VM Issues 4.11?

Hi

I have 2 pools with xenserver 7.0 (all patches installed) as AWS 4.10 not 
supporting higher version of xenserver. At least officially.
Does it make sence to upgrade on pool to 7.1 ? I  have 1 pool for testing only.


Lugupidamisega / Regards

Kristian Liivak

Tegevjuht / Executive director

WaveCom As
Endla 16, 10142 Tallinn
Estonia
Tel: +3726850001
Gsm: +37256850001
E-mail: k...@wavecom.ee
Skype: kristian.liivak
http://www.wavecom.ee
http://www.facebook.com/wavecom.ee


rohit.ya...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

- Original Message -
From: "rohit yadav" 
To: "users" 
Sent: Monday, February 26, 2018 5:48:43 PM
Subject: Re: System VM Issues 4.11?

Hi Kristian,


Which version of XenServer 7.x are you using? Please do note that XenServer 
7.3+ are not supported only, 7.0, 7.1 and 7.2 are. It looks like the expected 
cmdline argument is not sent/found in the systemvmtemplate which basically 
fails the deployment, however, I'm not sure about the exact cause and how to 
help you fix that.


- Rohit






From: Kristian Liivak 
Sent: Monday, February 26, 2018 3:20:34 PM
To: users
Subject: Re: System VM Issues 4.11?


Rohit,

Forgot about attachments.  Please see shot 
http://www.wavecom.ee/files/systemvm.PNG

Manual reboot to systemvm won´t help. Still no network. Is there maybe script 
to run initiate configuration provision again ?

I mean is there point to use new fresh centos 7 server with new install aws 
4.11 ?
Currectly my managment server have been updated several times.
But i understand systemvm configuration deplyment is done via vost anywway.

Lugupidamisega / Regards

Kristian Liivak

Tegevjuht

WaveCom As
Endla 16, 10142 Tallinn
Estonia
Tel: +3726850001
Gsm: +37256850001
E-mail: k...@wavecom.ee
Skype: kristian.liivak
http://www.wavecom.ee
http://www.facebook.com/wavecom.ee


rohit.ya...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue



- Original Message -
From: "rohit yadav" 
To: "users" 
Sent: Monday, February 26, 2018 4:03:04 PM
Subject: Re: System VM Issues 4.11?

.Hi Kristian,


Thanks for sharing. The mailing list strips attachments so we did not receive 
any of your images/attachments, kindly share them on any temporary hosting 
service.


I'm not sure what you mean by cleaning the management server. After your 
systemvm is up, does it work if you manually reboot the systemvm - i.e. does 
its nics come up after the manual reboot and you can see agents running (such 
as cpvm, ssvm?).


I think the db-upgrader changes the HVM flag, so it does not matter whether you 
had ticked the template or not before upgrading -- you can try but I think it 
should not make any difference.



- Rohit






From: Kristian Liivak 
Sent: Monday, February 26, 2018 2:46:14 PM
To: users
Subject: Re: System VM Issues 4.11?

Rohit,

Systemvms are 4.11, i can see that in console before login :)
cat /proc/cmdline returnes some info. Please see attached pics

Management sevrer is working just fine. I dont have however autostart on in 
upgrade time.
Im using old centos7 management server. Once allready made upgrade from ACS 4.9 
to 4.10
Now from 4.10 to 4.11..

Is there maybe point to use clean 4.11 management server to resolve system vm 
issue ?
And do i need to have HVM flag cheked with 4.11 system management template 
install ?
There is nothing about it in  instructions. However i have tried both ways 
without any success.


Lugupidamisega / Regards

Kristian Liivak

Tegevjuht / Executive director

WaveCom As
Endla 16, 10142 Tallinn
Estonia
Tel: +3726850001
Gsm: +37256850001
E-mail: k...@wavecom.ee
Skype: kristian.liivak
http://www.wavecom.ee
http://www.facebook.com/wavecom.ee


rohit.ya...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue



- Original Message -
From: "rohit yadav" 
To: "users" 
Sent: Sunday, February 25, 2018 4:04:54 PM
Subject: Re: System VM Issues 4.11?

Hi Kristian,


Thanks for sharing the logs. Looks like systemvm template failed to grab the 
cmdline options from XenServer to patch itself, reconfigure the nics. The 
ifconfig does not return at least the control nic. Can y