Looks like some error occurred while generating the keystore. Can you
check if you see any .jks and crt/key files at /etc/cloudstack/agent/
directory?
Also share output of:
netstat -nl | grep 16509 # if you get any listening libvirtd, then
your libvirtd is NOT secured
netstat -nl | grep 16514 # if you get this, then libvirtd is secured
ls -lahi /etc/cloudstack/agent
Did you upgrade to the latest LTS minor 4.11.2.0 release? If not
please do that, we've some bugs around CA certificates fixed.
- Rohit
rohit.ya...@shapeblue.com
www.shapeblue.com
@shapeblue
------------------------------------------------------------------------
*From:* Ugo Vasi <ugo.v...@procne.it>
*Sent:* Thursday, January 31, 2019 4:56:28 PM
*To:* users@cloudstack.apache.org; Rohit Yadav
*Subject:* Re: secure hosts communications
Update:
by rebooting the host system, the libvirt is restarted and the ACS-agent
has been reconnected to management.
The host remains in "unsecure" mode....
If I set to false "ca.plugin.root.auth.strictness" can I migrate the VM?
Il 31/01/19 11:50, Ugo Vasi ha scritto:
> Hi Rohit,
> I tryed renew certificate but it failed!
> Now libvirt does not restart and agent is disconnected:
>
> agent log:
> 2019-01-31 11:17:07,530 INFO
> [resource.wrapper.LibvirtPostCertificateRenewalCommandWrapper]
> (Certificate Renewal Timer:null) (logid:fe1554cc) Restarting libvirt
> after certificate provisioning/renewal
> 2019-01-31 11:17:07,567 INFO [cloud.agent.Agent]
> (AgentShutdownThread:null) (logid:) Stopping the agent: Reason =
sig.kill
> 2019-01-31 11:17:07,568 WARN [cloud.agent.Agent] (Certificate Renewal
> Timer:null) (logid:fe1554cc) Failed to execute post certificate
> renewal command:
> java.lang.IllegalStateException: Shutdown in progress
> at
>
java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82)
> at java.lang.Runtime.removeShutdownHook(Runtime.java:239)
> at
>
com.cloud.agent.Agent$PostCertificateRenewalTask.runInContext(Agent.java:1157)
> at
>
org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30)
> at
>
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
>
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
>
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
>
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
>
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
>
org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:32)
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505)
> 2019-01-31 11:17:09,797 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Agent started
> 2019-01-31 11:17:09,800 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Implementation Version is 4.11.2.0
> 2019-01-31 11:17:09,802 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) agent.properties found at
/etc/cloudstack/agent/agent.properties
> 2019-01-31 11:17:09,815 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Defaulting to using properties file for storage
> 2019-01-31 11:17:09,816 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Defaulting to the constant time backoff algorithm
> 2019-01-31 11:17:09,828 INFO [cloud.utils.LogUtils] (main:null)
> (logid:) log4j configuration found at
> /etc/cloudstack/agent/log4j-cloud.xml
> 2019-01-31 11:17:09,850 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Using default Java settings for IPv6 preference for agent
> connection
> 2019-01-31 11:17:09,998 INFO [cloud.agent.Agent] (main:null) (logid:)
> id is 5
> 2019-01-31 11:17:10,030 INFO [kvm.resource.LibvirtConnection]
> (main:null) (logid:) No existing libvirtd connection found. Opening a
> new one
> 2019-01-31 11:17:10,175 ERROR [cloud.agent.AgentShell] (main:null)
> (logid:) Unable to start agent:
> com.cloud.utils.exception.CloudRuntimeException: Failed to connect
> socket to '/var/run/libvirt/libvirt-sock': No such file or directory
> at
>
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.configure(LibvirtComputingResource.java:914)
> at com.cloud.agent.Agent.<init>(Agent.java:190)
> at
com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:453)
> at
> com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422)
> at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406)
> at com.cloud.agent.AgentShell.start(AgentShell.java:512)
> at com.cloud.agent.AgentShell.main(AgentShell.java:547)
> (logs repeat)
>
> syslog:
>
>
> Jan 31 11:17:07 cshp214 sh[5065]: INFO
> [resource.wrapper.LibvirtPostCertificateRenewalCommandWrapper]
> (Certificate Renewal Timer:) (logid:fe1554cc) Restarting libvirt after
> certificate provisioning/renewal
> Jan 31 11:17:07 cshp214 systemd[1]: Stopping CloudStack Agent...
> Jan 31 11:17:07 cshp214 sh[5065]: INFO [cloud.agent.Agent]
> (AgentShutdownThread:) (logid:) Stopping the agent: Reason = sig.kill
> Jan 31 11:17:07 cshp214 sh[5065]: WARN [cloud.agent.Agent]
> (Certificate Renewal Timer:) (logid:fe1554cc) Failed to execute post
> certificate renewal command:
> Jan 31 11:17:07 cshp214 sh[5065]: java.lang.IllegalStateException:
> Shutdown in progress
> Jan 31 11:17:07 cshp214 sh[5065]: #011at
>
java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82)
> Jan 31 11:17:07 cshp214 sh[5065]: #011at
> java.lang.Runtime.removeShutdownHook(Runtime.java:239)
> Jan 31 11:17:07 cshp214 sh[5065]: #011at
>
com.cloud.agent.Agent$PostCertificateRenewalTask.runInContext(Agent.java:1157)
> Jan 31 11:17:07 cshp214 sh[5065]: #011at
>
org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30)
> Jan 31 11:17:07 cshp214 sh[5065]: #011at
>
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> Jan 31 11:17:07 cshp214 sh[5065]: #011at
>
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> Jan 31 11:17:07 cshp214 sh[5065]: #011at
>
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> Jan 31 11:17:07 cshp214 sh[5065]: #011at
>
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> Jan 31 11:17:07 cshp214 sh[5065]: #011at
>
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> Jan 31 11:17:07 cshp214 sh[5065]: #011at
>
org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:32)
> Jan 31 11:17:07 cshp214 sh[5065]: #011at
> java.util.TimerThread.mainLoop(Timer.java:555)
> Jan 31 11:17:07 cshp214 sh[5065]: #011at
> java.util.TimerThread.run(Timer.java:505)
> Jan 31 11:17:08 cshp214 libvirtd[4700]: End of file while reading
> data: Input/output error
> Jan 31 11:17:08 cshp214 libvirtd[4700]: End of file while reading
> data: Input/output error
> Jan 31 11:17:08 cshp214 systemd[1]: Stopped CloudStack Agent.
> Jan 31 11:17:08 cshp214 systemd[1]: Stopping Virtualization daemon...
> Jan 31 11:17:08 cshp214 systemd[1]: Stopped Virtualization daemon.
> Jan 31 11:17:08 cshp214 systemd[1]: Starting Virtualization daemon...
> Jan 31 11:17:08 cshp214 systemd[1]: Started Virtualization daemon.
> Jan 31 11:17:08 cshp214 systemd[1]: Started CloudStack Agent.
> Jan 31 11:17:09 cshp214 sh[25387]: log4j:WARN No appenders could be
> found for logger (com.cloud.agent.AgentShell).
> Jan 31 11:17:09 cshp214 sh[25387]: log4j:WARN Please initialize the
> log4j system properly.
> Jan 31 11:17:09 cshp214 sh[25387]: log4j:WARN See
> http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
> Jan 31 11:17:09 cshp214 sh[25387]: INFO [cloud.agent.AgentShell]
> (main:) (logid:) Agent started
> Jan 31 11:17:09 cshp214 sh[25387]: INFO [cloud.agent.AgentShell]
> (main:) (logid:) Implementation Version is 4.11.2.0
> Jan 31 11:17:09 cshp214 sh[25387]: INFO [cloud.agent.AgentShell]
> (main:) (logid:) agent.properties found at
> /etc/cloudstack/agent/agent.properties
> Jan 31 11:17:09 cshp214 sh[25387]: INFO [cloud.agent.AgentShell]
> (main:) (logid:) Defaulting to using properties file for storage
> Jan 31 11:17:09 cshp214 sh[25387]: INFO [cloud.agent.AgentShell]
> (main:) (logid:) Defaulting to the constant time backoff algorithm
> Jan 31 11:17:09 cshp214 sh[25387]: INFO [cloud.utils.LogUtils]
> (main:) (logid:) log4j configuration found at
> /etc/cloudstack/agent/log4j-cloud.xml
> Jan 31 11:17:09 cshp214 sh[25387]: INFO [cloud.agent.AgentShell]
> (main:) (logid:) Using default Java settings for IPv6 preference for
> agent connection
> Jan 31 11:17:09 cshp214 sh[25387]: INFO [cloud.agent.Agent] (main:)
> (logid:) id is 5
> Jan 31 11:17:10 cshp214 sh[25387]: INFO
> [kvm.resource.LibvirtConnection] (main:) (logid:) No existing libvirtd
> connection found. Opening a new one
> --
> Jan 31 11:17:16 cshp214 snmpd[2460]: error on subcontainer 'ia_addr'
> insert (-1)
> Jan 31 11:17:16 cshp214 snmpd[2460]: message repeated 3 times: [ error
> on subcontainer 'ia_addr' insert (-1)]
> Jan 31 11:17:20 cshp214 systemd[1]: cloudstack-agent.service: Service
> hold-off time over, scheduling restart.
> Jan 31 11:17:20 cshp214 systemd[1]: Stopped CloudStack Agent.
> Jan 31 11:17:20 cshp214 systemd[1]: Started CloudStack Agent.
> Jan 31 11:17:20 cshp214 sh[25457]: log4j:WARN No appenders could be
> found for logger (com.cloud.agent.AgentShell).
> Jan 31 11:17:20 cshp214 sh[25457]: log4j:WARN Please initialize the
> log4j system properly.
> Jan 31 11:17:20 cshp214 sh[25457]: log4j:WARN See
> http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
> Jan 31 11:17:21 cshp214 sh[25457]: INFO [cloud.agent.AgentShell]
> (main:) (logid:) Agent started
> Jan 31 11:17:21 cshp214 sh[25457]: INFO [cloud.agent.AgentShell]
> (main:) (logid:) Implementation Version is 4.11.2.0
> Jan 31 11:17:21 cshp214 sh[25457]: INFO [cloud.agent.AgentShell]
> (main:) (logid:) agent.properties found at
> /etc/cloudstack/agent/agent.properties
> Jan 31 11:17:21 cshp214 sh[25457]: INFO [cloud.agent.AgentShell]
> (main:) (logid:) Defaulting to using properties file for storage
> Jan 31 11:17:21 cshp214 sh[25457]: INFO [cloud.agent.AgentShell]
> (main:) (logid:) Defaulting to the constant time backoff algorithm
> Jan 31 11:17:21 cshp214 sh[25457]: INFO [cloud.utils.LogUtils]
> (main:) (logid:) log4j configuration found at
> /etc/cloudstack/agent/log4j-cloud.xml
> Jan 31 11:17:21 cshp214 sh[25457]: INFO [cloud.agent.AgentShell]
> (main:) (logid:) Using default Java settings for IPv6 preference for
> agent connection
> Jan 31 11:17:21 cshp214 sh[25457]: INFO [cloud.agent.Agent] (main:)
> (logid:) id is 5
> Jan 31 11:17:21 cshp214 sh[25457]: INFO
> [kvm.resource.LibvirtConnection] (main:) (logid:) No existing libvirtd
> connection found. Opening a new one
> Jan 31 11:17:21 cshp214 sh[25457]: libvirt: XML-RPC error : Failed to
> connect socket to '/var/run/libvirt/libvirt-sock': No such file or
> directory
> Jan 31 11:17:21 cshp214 sh[25457]: ERROR [cloud.agent.AgentShell]
> (main:) (logid:) Unable to start agent:
> Jan 31 11:17:21 cshp214 sh[25457]:
> com.cloud.utils.exception.CloudRuntimeException: Failed to connect
> socket to '/var/run/libvirt/libvirt-sock': No such file or directory
> Jan 31 11:17:21 cshp214 sh[25457]: #011at
>
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.configure(LibvirtComputingResource.java:914)
> Jan 31 11:17:21 cshp214 sh[25457]: #011at
> com.cloud.agent.Agent.<init>(Agent.java:190)
> Jan 31 11:17:21 cshp214 sh[25457]: #011at
> com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:453)
> Jan 31 11:17:21 cshp214 sh[25457]: #011at
> com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422)
> Jan 31 11:17:21 cshp214 sh[25457]: #011at
> com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406)
> Jan 31 11:17:21 cshp214 sh[25457]: #011at
> com.cloud.agent.AgentShell.start(AgentShell.java:512)
> Jan 31 11:17:21 cshp214 sh[25457]: #011at
> com.cloud.agent.AgentShell.main(AgentShell.java:547)
> Jan 31 11:17:21 cshp214 sh[25457]: Unable to start agent: Failed to
> connect socket to '/var/run/libvirt/libvirt-sock': No such file or
> directory
> Jan 31 11:17:21 cshp214 systemd[1]: cloudstack-agent.service: Main
> process exited, code=exited, status=67/n/a
> Jan 31 11:17:21 cshp214 systemd[1]: cloudstack-agent.service: Unit
> entered failed state.
> Jan 31 11:17:21 cshp214 systemd[1]: cloudstack-agent.service: Failed
> with result 'exit-code'.
> Jan 31 11:17:21 cshp214 dnsmasq[4000]: read /etc/hosts - 13 addresses
> Jan 31 11:17:21 cshp214 dnsmasq[4000]: read
> /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
> Jan 31 11:17:21 cshp214 dnsmasq-dhcp[4000]: read
> /var/lib/libvirt/dnsmasq/default.hostsfile
> Jan 31 11:17:22 cshp214 snmpd[2460]: Connection from UDP:
> [127.0.0.1]:37699->[127.0.0.1]:161
> Jan 31 11:17:24 cshp214 snmpd[2460]: message repeated 2 times: [
> Connection from UDP: [127.0.0.1]:37699->[127.0.0.1]:161]
> Jan 31 11:17:24 cshp214 libvirtd[25368]: libvirt version: 1.3.1,
> package: 1ubuntu10.24 (Marc Deslauriers <marc.deslauri...@ubuntu.com>
> Wed, 23 May 2018 13:29:29 -0400)
> Jan 31 11:17:24 cshp214 libvirtd[25368]: hostname: cshp214
> Jan 31 11:17:24 cshp214 libvirtd[25368]: Configured security driver
> "none" disables default policy to create confined guests
> Jan 31 11:17:25 cshp214 libvirtd[25368]: unsupported configuration:
> Security driver apparmor not enabled
>
>
> Can anyone help me?
>
> Il 30/01/19 13:37, Rohit Yadav ha scritto:
>>
>> Hi Ugo,
>>
>>
>> This will be a one-time procedure, and the KVM host and the VMs do
>> not need a reboot but the provisionCertificate API will restart the
>> libvirtd process (just check if that can have any side effects for
>> your VMs/distro, on most modern distros restarting libvirtd does not
>> have any side-effects on existing running VMs).
>>
>>
>> - Rohit
>>
>>
>>
>> rohit.ya...@shapeblue.com
>> www.shapeblue.com <http://www.shapeblue.com>
>> @shapeblue
>>
>>
------------------------------------------------------------------------
>> *From:* Ugo Vasi <ugo.v...@procne.it>
>> *Sent:* Wednesday, January 30, 2019 4:47:09 PM
>> *To:* users@cloudstack.apache.org; Rohit Yadav
>> *Subject:* Re: secure hosts communications
>> Hi Rohit,
>> I have a 4.11.2.0 ACS infrastructure (Ubuntu 16.04 with KVM hypervisor)
>> I see that all the hosts are in unsecure state from the UI and so the
>> live migration don't works (we had trubles with mgmt server).
>>
>> I read in the documentation that launching the provisionCertificate API
>> (by pressing the appropriate button in the UI) the certificates will be
>> renewed/regenerated for already connected agents/hosts.
>>
>> I do not understand if provisioning should be done manually on each
host
>> or if the procedure should be done only once.
>>
>> Do this procedure reboot the host or the instances that it contains?
>>
>>
>> Thanks
>>
>>
>>
>> Il 27/11/18 09:49, Rohit Yadav ha scritto:
>> > Hi Richard,
>> >
>> >
>> > Please read:
>>
http://docs.cloudstack.apache.org/en/4.11.2.0/adminguide/hosts.html#security
>> >
>> >
>> > 4.11.2 is out, please consider using it instead of 4.11.1 as it has
>> several bugfixes etc.
>> >
>> > In short, with all of your KVM hosts up and connected to mgmt
>> server, first change the auth strictness global setting to true, then
>> using API secure the hosts using the provisionCertificate API. In the
>> UI, go to your hosts that don't show up as secure and click on the
>> key button (a new button) to secure the host which calls the
>> provisionCertificate API as well.
>> >
>> >
>> > - Rohit
>> >
>> > <https://cloudstack.apache.org>
>> >
>> >
>> >
>> > ________________________________
>> > From: Richard Persaud <richard.pers...@macys.com>
>> > Sent: Monday, November 26, 2018 8:19:56 PM
>> > To: users@cloudstack.apache.org
>> > Subject: RE: secure hosts communications
>> >
>> > Thank you, Rohit.
>> >
>> > I am using 4.11.1 with a full KVM environment. They are showing
>> unsecure with strictness set to true.
>> >
>> > What configuration needs to be adjusted to have the KVM hosts show
>> secure?
>> >
>> > Regards,
>> >
>> > Richard Persaud
>> >
>> > From: Rohit Yadav <rohit.ya...@shapeblue.com>
>> > Sent: Saturday, November 24, 2018 2:02 PM
>> > To: users@cloudstack.apache.org
>> > Subject: Re: secure hosts communications
>> >
>> > ⚠ EXT MSG:
>> >
>> > Richard,
>> >
>> >
>> > Starting 4.11, agent and management servers will use an in-built CA
>> framework to secured hosts. Only in case of KVM hosts you may see an
>> insecure state, otherwise all KVM hosts (agents) and SSVM/CPVM agents
>> will by default in Up state will be secured. There is an auth
>> strictness setting that should be true.
>> >
>> >
>> >
>> > - Rohit
>> >
>> > <https://cloudstack.apache.org>
>> >
>> >
>> >
>> > ________________________________
>> > From: Richard Persaud
>> <richard.pers...@macys.com<mailto:richard.pers...@macys.com>>
>> > Sent: Saturday, November 24, 2018 4:21:24 AM
>> > To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>
>> > Subject: secure hosts communications
>> >
>> > Hello,
>> >
>> > Is there straight-forward to enable secure communications between
>> the management and the hosts?
>> >
>> > I have looked at many documentations but am still unable to get the
>> hosts to show a "secure" state.
>> >
>> > Regards,
>> >
>> > Richard Persaud
>> >
>> >
>> > rohit.ya...@shapeblue.com<mailto:rohit.ya...@shapeblue.com>
>> >
>>
www.shapeblue.com<https://isolate.menlosecurity.com/0/eJyrViotylGyUsooKSmw0tcvLy_XK85ILEhNyilN1UvOz1XSUSrKV7Iy1FEqyUwBqjM0MFaqBQDf4BCe>
>> > Amadeus House, Floral Street, London WC2E 9DPUK
>> > @shapeblue
>> >
>> >
>> >
>> >
>> > * This is an EXTERNAL EMAIL. Stop and think before clicking a link
>> or opening attachments.
>> >
>> > rohit.ya...@shapeblue.com
>> > www.shapeblue.com <http://www.shapeblue.com>
<http://www.shapeblue.com>
>> > Amadeus House, Floral Street, London WC2E 9DPUK
>> > @shapeblue
>> >
>> >
>> >
>> >
>> >
>> >
>>
>>
>> --
>>
>> *Ugo Vasi* / System Administrator
>> ugo.v...@procne.it <mailto:ugo.v...@procne.it>
>>
>>
>>
>>
>> *Procne S.r.l.*
>> +39 0432 486 523
>> via Cotonificio, 45
>> 33010 Tavagnacco (UD)
>> www.procne.it <http://www.procne.it> <http://www.procne.it>
<http://www.procne.it/>
>>
>>
>> Le informazioni contenute nella presente comunicazione ed i relativi
>> allegati possono essere riservate e sono, comunque, destinate
>> esclusivamente alle persone od alla Società sopraindicati. La
>> diffusione, distribuzione e/o copiatura del documento trasmesso da
parte
>> di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
>> dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
>> "Codice in materia di protezione dei dati personali". Se avete ricevuto
>> questo messaggio per errore, vi preghiamo di distruggerlo e di
informare
>> immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
>> i...@procne.it <mailto:i...@procne.it>.
>>
>
>
--
*Ugo Vasi* / System Administrator
ugo.v...@procne.it <mailto:ugo.v...@procne.it>
*Procne S.r.l.*
+39 0432 486 523
via Cotonificio, 45
33010 Tavagnacco (UD)
www.procne.it <http://www.procne.it> <http://www.procne.it/>
Le informazioni contenute nella presente comunicazione ed i relativi
allegati possono essere riservate e sono, comunque, destinate
esclusivamente alle persone od alla Società sopraindicati. La
diffusione, distribuzione e/o copiatura del documento trasmesso da parte
di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
"Codice in materia di protezione dei dati personali". Se avete ricevuto
questo messaggio per errore, vi preghiamo di distruggerlo e di informare
immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
i...@procne.it <mailto:i...@procne.it>.
