Re: kvm host gets wrong IP address in host= at agent.properties

[jay hs]

Thanks for the tip.
I looked, and sure enough the "host" setting in the Global Settings has the 
bizarre value.
I think I figured out where the wrong value is coming from.

I am using a script to do the install.  The script sets the network 
configuration --- but it doesn't do a systemctl restart network before running 
the cloudstack mgt configuration so it must be picking up whatever random 
IP address is set before the static address is applied.

That breaks the logjam for me.
Thanks again.

--jay



On 2021/09/28 08:35:29, Wei ZHOU  wrote: 
> Hi Jay,
> 
> Please note that `host` is the IP of the cloudstack management server, not
> the IP of the kvm host.
> 
> If it is wrong, you can change agent.properties manually and restart
> cloudstack-agent.
> 
> Please also check `host` in the cloustack global settings.
> 
> -Wei
> 
> 
> 
> 
> 
> On Tue, 28 Sept 2021 at 09:02, jay hs  wrote:
> 
> > Hi all:
> > I must be doing something wrong.
> > I am installing 4.15 from the apt repositories on ubuntu 16.04
> > deb http://download.cloudstack.org/ubuntu xenial 4.15
> >
> > Everything goes grandly, except that when cloudstack-agent starts up, it
> > puts a seemingly random IP address (from the correct subnet) in host= at
> > agent.properties.
> >
> > It should be
> > host=172.16.10.2
> > but it picks up
> > host=172.16.10.246@static
> >
> > Evidently, it passes this *.246 value on to the ssvm -- which makes it
> > very mad.
> >
> > I've redone my management server and first kvm host a second time, and it
> > got *.234 instead of *.246.
> >
> > I am doing these on a test network that is NATed off from the main
> > network.  That NAT network did have its own DHCP active.  I've switched
> > that off -- and will do another try (tomorrow).
> >
> > But does this sound like I've got a configuration missing -- or am I
> > picking up a bum copy of the cloudstack-agent from the repo?  or... ideas?
> >
> > thanks.
> > --jay
> >
> 

Re: kvm host gets wrong IP address in host= at agent.properties

[jay hs]

Daan -- 
Thanks... I got it thanks to @weizhou.
In brief, the install is scripted, and the static IP address I assign is NOT in 
play when the cloudstack mgt config routine is run.  So... I will add a 
systemctl restart network (or some such) to fix.

Thanks for your time.

On 2021/09/28 08:30:20, Daan Hoogland  wrote: 
> Jay, is this after you added the host to cloudstack or before?
> 
> On Tue, Sep 28, 2021 at 9:02 AM jay hs  wrote:
> 
> > Hi all:
> > I must be doing something wrong.
> > I am installing 4.15 from the apt repositories on ubuntu 16.04
> > deb http://download.cloudstack.org/ubuntu xenial 4.15
> >
> > Everything goes grandly, except that when cloudstack-agent starts up, it
> > puts a seemingly random IP address (from the correct subnet) in host= at
> > agent.properties.
> >
> > It should be
> > host=172.16.10.2
> > but it picks up
> > host=172.16.10.246@static
> >
> > Evidently, it passes this *.246 value on to the ssvm -- which makes it
> > very mad.
> >
> > I've redone my management server and first kvm host a second time, and it
> > got *.234 instead of *.246.
> >
> > I am doing these on a test network that is NATed off from the main
> > network.  That NAT network did have its own DHCP active.  I've switched
> > that off -- and will do another try (tomorrow).
> >
> > But does this sound like I've got a configuration missing -- or am I
> > picking up a bum copy of the cloudstack-agent from the repo?  or... ideas?
> >
> > thanks.
> > --jay
> >
> 
> 
> -- 
> Daan
> 

Virtual Router failing health chek - webserver.service

[Christopher Brown]

Hi everyone,

I am setting up a redundant VPC.
Therefore i created a custome service offering with the following Service
provided by the virtual Router:
- Portforwarding
- Network ACL
- SourceNAT
- VPN
- StaticNat

I now getting alerts, that the router couldn't pass all health checks. The
one failing at the moment is "webserver.service".
As far as i got reading the log files, the apache service on the router is
shutting down, as know valid / correct ssl - certificate is availeable.

So now i am looking for some support on how to get rid of the errors and
some information, why a webserver is running on the router?
I found some articel in the wiki about the integrated Load Balancer. But i
didn't have the service in the vpc service offering as well it wasn't
selected for the (custome) service offering for the tier-networks for the
vpc.

Hope someone can help me out!
Regards!

Re: Virtual Router failing health chek - webserver.service

[Wei ZHOU]

Hi Christopher,

There is no such service 'webserver.service' in cloudstack VR. Do you mean
apache2 ? It is set up in VR for 'Userdata' service.
Considering your custom vpc offering does not support 'Userdata',
the health check on it should not be performed. it might be a bug (please
provide more info).

You can exclude a health check (effectively on all VRs) by setting global
configuration `router.health.checks.to.exclude`.

-Wei



On Wed, 29 Sept 2021 at 09:11, Christopher Brown <
mail2christopher.br...@gmail.com> wrote:

> Hi everyone,
>
> I am setting up a redundant VPC.
> Therefore i created a custome service offering with the following Service
> provided by the virtual Router:
> - Portforwarding
> - Network ACL
> - SourceNAT
> - VPN
> - StaticNat
>
> I now getting alerts, that the router couldn't pass all health checks. The
> one failing at the moment is "webserver.service".
> As far as i got reading the log files, the apache service on the router is
> shutting down, as know valid / correct ssl - certificate is availeable.
>
> So now i am looking for some support on how to get rid of the errors and
> some information, why a webserver is running on the router?
> I found some articel in the wiki about the integrated Load Balancer. But i
> didn't have the service in the vpc service offering as well it wasn't
> selected for the (custome) service offering for the tier-networks for the
> vpc.
>
> Hope someone can help me out!
> Regards!
>

Re: Virtual Router failing health chek - webserver.service

[Wei ZHOU]

I confirm that the health check on `webserver` checks the process `apache2`
in VR.
{"id":"0","service":"webserver","processname":"apache2","serviceName":"apache2","servicePath":"/var/run/apache2/apache2.pid","pidFile":"/var/run/apache2/apache2.pid","isDefault":"true"}

Christopher, could you please file an issue on github ?
https://github.com/apache/cloudstack/issues

-Wei


On Wed, 29 Sept 2021 at 09:21, Wei ZHOU  wrote:

> Hi Christopher,
>
> There is no such service 'webserver.service' in cloudstack VR. Do you mean
> apache2 ? It is set up in VR for 'Userdata' service.
> Considering your custom vpc offering does not support 'Userdata',
> the health check on it should not be performed. it might be a bug (please
> provide more info).
>
> You can exclude a health check (effectively on all VRs) by setting global
> configuration `router.health.checks.to.exclude`.
>
> -Wei
>
>
>
> On Wed, 29 Sept 2021 at 09:11, Christopher Brown <
> mail2christopher.br...@gmail.com> wrote:
>
>> Hi everyone,
>>
>> I am setting up a redundant VPC.
>> Therefore i created a custome service offering with the following Service
>> provided by the virtual Router:
>> - Portforwarding
>> - Network ACL
>> - SourceNAT
>> - VPN
>> - StaticNat
>>
>> I now getting alerts, that the router couldn't pass all health checks. The
>> one failing at the moment is "webserver.service".
>> As far as i got reading the log files, the apache service on the router is
>> shutting down, as know valid / correct ssl - certificate is availeable.
>>
>> So now i am looking for some support on how to get rid of the errors and
>> some information, why a webserver is running on the router?
>> I found some articel in the wiki about the integrated Load Balancer. But i
>> didn't have the service in the vpc service offering as well it wasn't
>> selected for the (custome) service offering for the tier-networks for the
>> vpc.
>>
>> Hope someone can help me out!
>> Regards!
>>
>

Re: Virtual Router failing health chek - webserver.service

[vas...@gmx.de]

Hi Wei,

thanks for your effort,

I will open an issue. In the mean time, i also get several errors - even
more then of the failing healthserver check.

it seams, that in an redundant setup, there are several things broken at
the moment.
i now get additionally several errors on the master-router.
Failing healthchecks:
  - dhcp_chck.py
  - dns_chck.py (here i get some information, that die instance has no
entry of the hostname in the /etc/hosts)

Nevertheless my offering for the cloud doesn't have this services specified
as well as the tier networks don't offer dhcp or dns services. 


Am Mi., 29. Sept. 2021 um 09:29 Uhr schrieb Wei ZHOU :

> I confirm that the health check on `webserver` checks the process
> `apache2` in VR.
>
> {"id":"0","service":"webserver","processname":"apache2","serviceName":"apache2","servicePath":"/var/run/apache2/apache2.pid","pidFile":"/var/run/apache2/apache2.pid","isDefault":"true"}
>
> Christopher, could you please file an issue on github ?
> https://github.com/apache/cloudstack/issues
>
> -Wei
>
>
> On Wed, 29 Sept 2021 at 09:21, Wei ZHOU  wrote:
>
>> Hi Christopher,
>>
>> There is no such service 'webserver.service' in cloudstack VR. Do you
>> mean apache2 ? It is set up in VR for 'Userdata' service.
>> Considering your custom vpc offering does not support 'Userdata',
>> the health check on it should not be performed. it might be a bug (please
>> provide more info).
>>
>> You can exclude a health check (effectively on all VRs) by setting global
>> configuration `router.health.checks.to.exclude`.
>>
>> -Wei
>>
>>
>>
>> On Wed, 29 Sept 2021 at 09:11, Christopher Brown <
>> mail2christopher.br...@gmail.com> wrote:
>>
>>> Hi everyone,
>>>
>>> I am setting up a redundant VPC.
>>> Therefore i created a custome service offering with the following Service
>>> provided by the virtual Router:
>>> - Portforwarding
>>> - Network ACL
>>> - SourceNAT
>>> - VPN
>>> - StaticNat
>>>
>>> I now getting alerts, that the router couldn't pass all health checks.
>>> The
>>> one failing at the moment is "webserver.service".
>>> As far as i got reading the log files, the apache service on the router
>>> is
>>> shutting down, as know valid / correct ssl - certificate is availeable.
>>>
>>> So now i am looking for some support on how to get rid of the errors and
>>> some information, why a webserver is running on the router?
>>> I found some articel in the wiki about the integrated Load Balancer. But
>>> i
>>> didn't have the service in the vpc service offering as well it wasn't
>>> selected for the (custome) service offering for the tier-networks for the
>>> vpc.
>>>
>>> Hope someone can help me out!
>>> Regards!
>>>
>>

Re: Virtual Router failing health chek - webserver.service

[vas...@gmx.de]

Okay, just thought for a second.
I guess the dhcp.service will be needed for handling the additionally
needed IP adresses for the NICS of the gateway (1-"Shared" IP and 2x
individual IPs for each Master / Backup ). As these can't be specified, the
routers need to choose one randomely or via dhcp.
However is there a way to actually see / configure IP leases in the GUI?
Same question would be regarding dns services though.


Am Mi., 29. Sept. 2021 um 10:57 Uhr schrieb vas...@gmx.de :

> Hi Wei,
>
> thanks for your effort,
>
> I will open an issue. In the mean time, i also get several errors - even
> more then of the failing healthserver check.
>
> it seams, that in an redundant setup, there are several things broken at
> the moment.
> i now get additionally several errors on the master-router.
> Failing healthchecks:
>   - dhcp_chck.py
>   - dns_chck.py (here i get some information, that die instance has no
> entry of the hostname in the /etc/hosts)
>
> Nevertheless my offering for the cloud doesn't have this services
> specified as well as the tier networks don't offer dhcp or dns services.
> 
>
>
> Am Mi., 29. Sept. 2021 um 09:29 Uhr schrieb Wei ZHOU <
> ustcweiz...@gmail.com>:
>
>> I confirm that the health check on `webserver` checks the process
>> `apache2` in VR.
>>
>> {"id":"0","service":"webserver","processname":"apache2","serviceName":"apache2","servicePath":"/var/run/apache2/apache2.pid","pidFile":"/var/run/apache2/apache2.pid","isDefault":"true"}
>>
>> Christopher, could you please file an issue on github ?
>> https://github.com/apache/cloudstack/issues
>>
>> -Wei
>>
>>
>> On Wed, 29 Sept 2021 at 09:21, Wei ZHOU  wrote:
>>
>>> Hi Christopher,
>>>
>>> There is no such service 'webserver.service' in cloudstack VR. Do you
>>> mean apache2 ? It is set up in VR for 'Userdata' service.
>>> Considering your custom vpc offering does not support 'Userdata',
>>> the health check on it should not be performed. it might be a bug (please
>>> provide more info).
>>>
>>> You can exclude a health check (effectively on all VRs) by setting
>>> global configuration `router.health.checks.to.exclude`.
>>>
>>> -Wei
>>>
>>>
>>>
>>> On Wed, 29 Sept 2021 at 09:11, Christopher Brown <
>>> mail2christopher.br...@gmail.com> wrote:
>>>
 Hi everyone,

 I am setting up a redundant VPC.
 Therefore i created a custome service offering with the following
 Service
 provided by the virtual Router:
 - Portforwarding
 - Network ACL
 - SourceNAT
 - VPN
 - StaticNat

 I now getting alerts, that the router couldn't pass all health checks.
 The
 one failing at the moment is "webserver.service".
 As far as i got reading the log files, the apache service on the router
 is
 shutting down, as know valid / correct ssl - certificate is availeable.

 So now i am looking for some support on how to get rid of the errors and
 some information, why a webserver is running on the router?
 I found some articel in the wiki about the integrated Load Balancer.
 But i
 didn't have the service in the vpc service offering as well it wasn't
 selected for the (custome) service offering for the tier-networks for
 the
 vpc.

 Hope someone can help me out!
 Regards!

>>>

HA behaviour with Cloudstack / XCP-NG

[Florian Noel]

Hi everyone,

We are using Cloudstack 4.15.1 with a cluster of hypervisors XCP 8.2
We are testing High Availability.

I have read that XCP (Xenserver) is responsible to elect a new pool master if 
the pool master fails.
And Cloudstack is responsible to restart virtual machines if a hypervisor fails.

We have virtual machines with compute offering inclued HA capability and 
virtual routers on the pool master.
If we disconnect the management cable from the pool master, HA-XCP electes a 
new pool master, the old pool master restarts.
This is expected behaviour.


However, Cloudstack never launches the virtual machines and the virtual routers 
on our other nodes XCP that were on the pool master.
Is this the expected behaviour ? We would have liked Cloudstack to be able to 
launch the VMs on the other nodes.
I'm not sure Cloudstack knows the pool master have changed. Why ? How can I 
solve this ?


We have also tested disconnect storage cables from the pool master and in this 
case, Cloudstack is able to launch virtual machines and virtual routers on our 
other nodes of the cluster after few minutes.

Thanks for your help.

Best regards.

Florian


[Logo Web et 
Solutions]

[Facebook]

[Twitter]

[LinkedIn]

[Youtube]

Florian Noel

Administrateur Systèmes Et Réseaux

[https://storage.letsignit.com/icons/designer/v2/phone-1.png] 02 35 78 11 90

705 Avenue Isaac Newton

76800 Saint-Etienne-Du-Rouvray

[Payneo]

Creating a second Pod and Public IP ranges

[Brian Fitzpatrick]

Hi all,

We have been building a Cloudstack 4.15.2 environment  using Ubuntu 20.04 and 
KVM.

We have thus far only built one Pod with some clusters and hosts in clusters 
sharing nfs storage.

The existing Pod is on a physical subnet 10.250.0.0/22 (private address) but 
Public in terms of out on our network.

We have assigned some Public IP address ranges 10.250.2.xxx - 10.250.2.yyy ie. 
within this physical /22 network.

I would like to add a second Pod on a new subnet 10.250.20.0/22.

I guess, in order to support virtual routers hosted onto that new Pod, I need 
to add a new Public IP range that falls within it?

ie. 10.250.22.xxx - 10.250.22.yyy ?

There doesn't seem to be a way to link a Public IP range to a Pod? I haven't 
tested this yet but does it just link the Public range with the Pod subnet and 
gateway automatically ? associating the gateway?

So if somebody creates a network and it lives on Pod2 the Virtual router and 
network gets a public IP from that network. Or is there something I have to do 
to associate Public IP ranges to Pods?

Hope this makes sense?

Thanks

Brian

RE: Unable to read/process CSR: Command failed due to Exception: java.lang.Exception

[cristian.c]

Hi,

  Yes, it is the server, but I already started to check the BIOS ... and I 
have very interesting question, why with KVM is working?  What is OS different 
between KVM and VMware when comes to systemvm patching/inject keys? I think 
here I have the answer to may error.

Regards,
Cristian

-Original Message-
From: Daan Hoogland  
Sent: Tuesday, September 28, 2021 12:14 PM
To: users 
Subject: Re: Unable to read/process CSR: Command failed due to Exception: 
java.lang.Exception

ok, than I would like to add an idea to your "it's the server". Not saying you 
are wrong but location points to network i.e. firewall(s).
Hope you strike luck,

On Tue, Sep 28, 2021 at 10:58 AM  wrote:

> Hi,
>
> We already tested in all the ways is possible, this problem is 
> related to these specific servers, 100%.
>
>The last test we did, was to add the ESXI server to the zone we 
> already have and works, we disabled all hosts from that zone and 
> destroyed the SSVM to force a redeploy on this ESXI, of course, we 
> have ended with the same error.
>
>I want to mention that we have the exact same ESXI version and 
> hardware, for me looks like there is a BIOS/HW/DISK issue.
>
>
> FYI:
>
>"I did this in 3 different new Setup (new Management install, NFS, 
> etc), no matter if I add as first zone or second zone.
>
>" I have tested this with multiple Cloudstack versions, 4.15.0, 
> 4.15.1,
> 4.15.2 and VMware, 6.5 and with 6.7 different patches and with the 
> last patches.  I'm 100% that is related to these 2 servers, I do not 
> understand what is wrong with these servers, this is the problem.
>
> We have 2 identical servers' hardware/ESXI, the only difference is 
> the location, the not working, are in UK, and any other location we 
> have, DE, NYC, FR, CA, etc. works fine. But we tested with the exact 
> same servers which are from DE(Germany) works perfect, with any ACS or 
> VMware version (also same CIDR range/size for public and private), 
> when we test with UK, we get same error no matter what."
>
>I want to mention that we have these servers (UK, DE, FR, CA ) from 
> OVH in vRack  and the UK, are not working.  ( I manage these server 
> for more than 5 years )"
>
> Regards,
> Cristian
>
> -Original Message-
> From: Daan Hoogland 
> Sent: Tuesday, September 28, 2021 11:00 AM
> To: users 
> Subject: Re: Unable to read/process CSR: Command failed due to Exception:
> java.lang.Exception
>
> Christian, did you solve this (i just encountered your mail and see it 
> is yet unanswered) the problem is with the certificate generation, it 
> seems to try and generate from an empty string (the csr/certificate 
> signing request) Not being able to sign in to the SVM is a problem but 
> on vmware you would use a specific command as described in [1]. Did 
> you try `ssh -i /opt/xensource/bin/id_rsa --p 3922 root@privateIP 
>  OfTheHost`?
>
> [1]
>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/SSVM%2C+templat
> es%2C+Secondary+storage+troubleshooting
>
> On Thu, Sep 23, 2021 at 11:31 AM  wrote:
>
> > Hello,
> >
> >
> >
> >I'm not sure why I get this error (added new vmware zone), 
> > there are not permission issues for systemvms folder, and I do not 
> > see any error above this. If I try to login to login into SSVM/Proxy 
> > console via cloudstack management, I get this access denied.
> >
> >
> >
> >Any suggestion?
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > 021-09-23 05:22:15,012 ERROR [c.c.u.s.SshHelper] 
> > (DirectAgent-283:ctx-b1e493bd lnd-uk-001.shape.host, 
> > job-34950/job-34972,
> > cmd: SetupKeyStoreCommand) (logid:f3cd2252) Failed to authentication 
> > SSH user root on host 10.15.0.160
> >
> > 2021-09-23 05:22:15,012 INFO  [c.c.h.v.u.VmwareHelper] 
> > (DirectAgent-283:ctx-b1e493bd lnd-uk-001.shape.host, 
> > job-34950/job-34972,
> > cmd: SetupKeyStoreCommand) (logid:f3cd2252) [ignored]failed to get 
> > message for exception: Failed to authentication SSH user root on 
> > host
> > 10.15.0.160
> >
> > 2021-09-23 05:22:15,012 ERROR [c.c.h.v.r.VmwareResource] 
> > (DirectAgent-283:ctx-b1e493bd lnd-uk-001.shape.host, 
> > job-34950/job-34972,
> > cmd: SetupKeyStoreCommand) (logid:f3cd2252) Command failed due to
> > Exception:
> > java.lang.Exception
> >
> > Message: Failed to authentication SSH user root on host 10.15.0.160
> >
> >
> >
> > 2021-09-23 05:22:15,012 DEBUG [c.c.h.v.r.VmwareResource] 
> > (DirectAgent-283:ctx-b1e493bd lnd-uk-001.shape.host, 
> > job-34950/job-34972,
> > cmd: SetupKeyStoreCommand) (logid:f3cd2252) keystore-setup execution
> > result:
> > false
> >
> > 2021-09-23 05:22:15,014 DEBUG [c.c.a.m.DirectAgentAttache]
> > (DirectAgent-283:ctx-b1e493bd) (logid:f3cd2252) Seq
> 40-1394708509601300577:
> > Response Received:
> >
> > 2021-09-23 05:22:15,016 DEBUG [c.c.a.t.Request]
> > (DirectAgent-283:ctx-b1e493bd) (logid:f3cd2252) Seq
> 40-1394708509601300577:
> > Processing:  { Ans: , MgmtId: 345049356158, via:
> > 40(lnd-uk-001.

Re: [VOTE] Release Apache CloudStack CloudMonkey 6.2.0

[Abhishek Kumar]

+1

Tested build for different operations on a ACS 4.16-snapshot env. Everything 
worked as expected.

Regards,
Abhishek

From: Pearl Dsilva 
Sent: 24 September 2021 13:14
To: d...@cloudstack.apache.org 
Cc: users@cloudstack.apache.org 
Subject: [VOTE] Release Apache CloudStack CloudMonkey 6.2.0

Hi All,

I've created a v6.2.0 release of CloudMonkey, with the following
artifacts up for a vote:

Git Branch and commit SHA:
https://github.com/apache/cloudstack-cloudmonkey/commits/8aae61e20c6789133c1d97d49e58f354ba7428c3

Commit:
8aae61e20c6789133c1d97d49e58f354ba7428c3

GitHub pre-release (for RC1 testing, contains changelog,
artifacts/binaries to test, checksums/usage details):
https://github.com/apache/cloudstack-cloudmonkey/releases/tag/6.2.0

Source release (checksums and signatures are available at the same location):
https://dist.apache.org/repos/dist/dev/cloudstack/cloudmonkey-6.2.0/

PGP release keys (signed using 986611B4A5B7090D0145B230E7DB9FC18F16C6AE)
https://dist.apache.org/repos/dist/release/cloudstack/KEYS

The vote will be open until October 1st, 2021.

For sanity in tallying the vote, can PMC members please be sure to
indicate "(binding)" with their vote?
[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove (and the reason why)

Regards,
Pearl Dsilva