Re: microvm support

[Antoine Boucher]

Mostly used for virtual network appliances and web services, we used 
Firecracker on OpenNebula.  That is a feature we used and liked, however, we 
could not appreciate their storage paradigm but then again the networking was 
too basic for our needs.  

Antoine Boucher
antoi...@haltondc.com
[o] +1-226-505-9734
www.haltondc.com

“Data security made simple and affordable”





Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.


> On Mar 3, 2022, at 10:41 AM, Nux  wrote:
> 
> Antoine,
> 
> Out of curiosity, what would be your use/business case for such microVM 
> feature?
> 
> Perhaps I'm misinformed, but had a quick look at it and it's just a "slim" 
> KVM. It seems like it would make sense to use something like this if you have 
> significant "serverless" type of workloads, or indeed sell such thing 
> publicly.
> I can see why Amazon wants Firecracker, it goes with their narrative (and 
> "hype" may I add) for "serverless".
> 
> PS: Opennebula is pretty nice, love the "contextualisation", very powerful; 
> but yeah, quirky, like everything else.
> 
> ---
> Nux!
> www.nux.ro
> 
> On 2022-03-03 15:30, Antoine Boucher wrote:
>> microVM support is definitely a feature I’m missing that we were
>> accustom to from OpenNebula, fortunately CloudStack makes up for many
>> OpenNebula weirdness…
>> -Antoine
>>> On Mar 3, 2022, at 10:04 AM, Wei ZHOU  wrote:
>>> Hi Ricardo,
>>> CloudStack does not support customized qemu machine type for vms or
>>> templates (All VMs use `pc` or `q35`) .
>>> The feature is currently on our idea board.
>>> -Wei
>>> On Thu, 3 Mar 2022 at 13:00, Ricardo Pertuz 
>>> wrote:
 Hi,
 Is there a way to deploy microvm (firecracker approach) using qemu on
 Cloudstack?
 As explained in this link
 https://qemu.readthedocs.io/en/latest/system/i386/microvm.html
 Regards,

Re: microvm support

[Nux]

Antoine,

Out of curiosity, what would be your use/business case for such microVM 
feature?


Perhaps I'm misinformed, but had a quick look at it and it's just a 
"slim" KVM. It seems like it would make sense to use something like this 
if you have significant "serverless" type of workloads, or indeed sell 
such thing publicly.
I can see why Amazon wants Firecracker, it goes with their narrative 
(and "hype" may I add) for "serverless".


PS: Opennebula is pretty nice, love the "contextualisation", very 
powerful; but yeah, quirky, like everything else.


---
Nux!
www.nux.ro

On 2022-03-03 15:30, Antoine Boucher wrote:

microVM support is definitely a feature I’m missing that we were
accustom to from OpenNebula, fortunately CloudStack makes up for many
OpenNebula weirdness…

-Antoine



On Mar 3, 2022, at 10:04 AM, Wei ZHOU  wrote:

Hi Ricardo,

CloudStack does not support customized qemu machine type for vms or
templates (All VMs use `pc` or `q35`) .
The feature is currently on our idea board.

-Wei

On Thu, 3 Mar 2022 at 13:00, Ricardo Pertuz 
wrote:


Hi,

Is there a way to deploy microvm (firecracker approach) using qemu on
Cloudstack?
As explained in this link
https://qemu.readthedocs.io/en/latest/system/i386/microvm.html

Regards,

Expected 1 answers while executing SetMonitorServiceCommand but received 3

[Antoine Boucher]

Hello,

Is the following issue significant?

From one of my kvm host logs:
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-2:null) 
(logid:69ceff92) Expected 1 answers while executing SetMonitorServiceCommand 
but received 3
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) 
(logid:5b95c453) Fetching health check result for 169.254.4.222 and executing 
fresh checks: false

I was looking at issue: vr: fix python exception when configure VRs #4489 


I'm using ACS 4.16 with KVM qemu-ev on CentOS 7.9

Regards,
Antoine

Re: microvm support

[Antoine Boucher]

microVM support is definitely a feature I’m missing that we were accustom to 
from OpenNebula, fortunately CloudStack makes up for many OpenNebula weirdness…

-Antoine


> On Mar 3, 2022, at 10:04 AM, Wei ZHOU  wrote:
> 
> Hi Ricardo,
> 
> CloudStack does not support customized qemu machine type for vms or
> templates (All VMs use `pc` or `q35`) .
> The feature is currently on our idea board.
> 
> -Wei
> 
> On Thu, 3 Mar 2022 at 13:00, Ricardo Pertuz 
> wrote:
> 
>> Hi,
>> 
>> Is there a way to deploy microvm (firecracker approach) using qemu on
>> Cloudstack?
>> As explained in this link
>> https://qemu.readthedocs.io/en/latest/system/i386/microvm.html
>> 
>> Regards,
>> 

Re: microvm support

[Wei ZHOU]

Hi Ricardo,

CloudStack does not support customized qemu machine type for vms or
templates (All VMs use `pc` or `q35`) .
The feature is currently on our idea board.

-Wei

On Thu, 3 Mar 2022 at 13:00, Ricardo Pertuz 
wrote:

> Hi,
>
> Is there a way to deploy microvm (firecracker approach) using qemu on
> Cloudstack?
> As explained in this link
> https://qemu.readthedocs.io/en/latest/system/i386/microvm.html
>
> Regards,
>

Re: microvm support

[Ricardo Pertuz]

Cool Antoine, also evaluation Kata Container, not sure if I can integrate with 
CS

Regards

From: Antoine Boucher 
Date: Thursday, 3 March 2022, 9:20 AM
To: users , Ricardo Pertuz 

Subject: Re: microvm support

Hi Ricardo,

I’m also interested in this topic as opposed to lxc

Regards,
Antoine




On Mar 3, 2022, at 6:59 AM, Ricardo Pertuz 
mailto:ricardo.per...@kuasar.co>> wrote:

Hi,

Is there a way to deploy microvm (firecracker approach) using qemu on 
Cloudstack?
As explained in this link
https://qemu.readthedocs.io/en/latest/system/i386/microvm.html

Regards,

Re: microvm support

[Antoine Boucher]

Hi Ricardo,

I’m also interested in this topic as opposed to lxc

Regards,
Antoine



> On Mar 3, 2022, at 6:59 AM, Ricardo Pertuz  wrote:
> 
> Hi,
> 
> Is there a way to deploy microvm (firecracker approach) using qemu on 
> Cloudstack?
> As explained in this link
> https://qemu.readthedocs.io/en/latest/system/i386/microvm.html
> 
> Regards,

KVM Host Disconnects from Management server

[Antoine Boucher]

I have 2 zones hosted from the same management server,  one larger local to 
management server zone and one smaller “remote” zone using an IPSec tunnel.  

All is well except that recently one of the “remote” KVM host keeps loosing 
connectivity with the management server.   It used to happened every few days 
24 but now it happens every few hours.  

Other than rebooting, I’m only able to recover by killing the Java process and 
restarting the CloudStack-Agent. 

The tunnel is monitored and used by many other processes without any 
interruptions.

Has anyone experienced anything similar before?


Here are my host logs, the management server is 10.101.2.40 and the kvm host is 
10.91.1.11:

2022-03-03 08:21:09,477 INFO  [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-3:null) (logid:7ff2c4c1) Trying to fetch storage pool 
cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt
2022-03-03 08:21:09,482 INFO  [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-3:null) (logid:7ff2c4c1) Asking libvirt to refresh 
storage pool cec8c1db-c9a0-42cc-96d3-5c06369d115c
2022-03-03 08:22:03,802 WARN  [resource.virtualnetwork.VirtualRoutingResource] 
(agentRequest-Handler-1:null) (logid:4fc5e215) Expected 1 answers while 
executing SetMonitorServiceCommand but received 3
2022-03-03 08:22:07,740 INFO  [resource.virtualnetwork.VirtualRoutingResource] 
(agentRequest-Handler-5:null) (logid:ba7d1810) Fetching health check result for 
169.254.4.222 and executing fresh checks: false
2022-03-03 08:22:13,881 INFO  [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-3:null) (logid:f0ed50d5) Trying to fetch storage pool 
cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt
2022-03-03 08:22:13,886 INFO  [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-3:null) (logid:f0ed50d5) Asking libvirt to refresh 
storage pool cec8c1db-c9a0-42cc-96d3-5c06369d115c
2022-03-03 08:22:17,641 INFO  [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-2:null) (logid:54076348) Trying to fetch storage pool 
cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt
2022-03-03 08:22:17,651 INFO  [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-2:null) (logid:54076348) Trying to fetch storage pool 
cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt
2022-03-03 08:22:17,659 INFO  [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-2:null) (logid:54076348) Trying to fetch storage pool 
cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt
2022-03-03 08:26:45,714 INFO  [cloud.agent.Agent] (Agent-Handler-4:null) 
(logid:3959c447) Lost connection to host: 10.101.2.40. Attempting reconnection 
while we still have 0 commands in progress.
2022-03-03 08:26:45,715 INFO  [utils.nio.NioClient] (Agent-Handler-4:null) 
(logid:3959c447) NioClient connection closed
2022-03-03 08:26:45,716 INFO  [cloud.agent.Agent] (Agent-Handler-4:null) 
(logid:3959c447) Reconnecting to host:10.101.2.40
2022-03-03 08:26:45,716 INFO  [utils.nio.NioClient] (Agent-Handler-4:null) 
(logid:3959c447) Connecting to 10.101.2.40:8250
2022-03-03 08:26:45,733 INFO  [utils.nio.Link] (Agent-Handler-4:null) 
(logid:3959c447) Conf file found: /etc/cloudstack/agent/agent.properties
2022-03-03 08:27:15,762 WARN  [utils.nio.Link] (Agent-Handler-4:null) 
(logid:3959c447) This SSL engine was forced to close inbound due to end of 
stream.
javax.net.ssl.SSLException: closing inbound before receiving peer's close_notify
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
at 
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:339)
at 
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:295)
at 
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:286)
at 
java.base/sun.security.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:733)
at com.cloud.utils.nio.Link.doHandshakeUnwrap(Link.java:490)
at com.cloud.utils.nio.Link.doHandshake(Link.java:618)
at com.cloud.utils.nio.NioClient.init(NioClient.java:64)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95)
at com.cloud.agent.Agent.reconnect(Agent.java:536)
at com.cloud.agent.Agent$ServerHandler.doTask(Agent.java:1117)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-03-03 08:27:15,768 INFO  [utils.nio.NioClient] (Agent-Handler-4:null) 
(logid:3959c447) SSL: Handshake done
2022-03-03 08:27:15,768 INFO  [utils.nio.NioClient] (Agent-Handler-4:null) 
(logid:3959c447) Connected to 

[RESULT][VOTE] Apache CloudStack 4.16.1.0 (RC2)

[Suresh Anaparti]

Hi All,

The vote for Apache CloudStack 4.16.1.0 *passes* with
7 PMC + 4 non-PMC votes.

+1 (PMC / binding)
7 persons (Boris, Nux, Rohit, Nicolas, Wei, Daan, Andrija)

+1 (non binding)
4 persons (Vladi, Harikrishna, Suresh, Slavka)

0
none

-1
none

Thank you everyone for participating.

I will now prepare the release announcement to go out after 24-48 hours to give 
the mirrors time to catch up.

 
Regards,
Suresh

On 03/03/22, 3:56 PM, "Andrija Panic"  wrote:

+1 (binding)

Tested upgrading 4.13.1 with multiple hypervisors, automatic-systemVM
template registration; basic VM/network lifecycle, Advanced Zone.

On Wed, 2 Mar 2022 at 16:48, Slavka Peleva 
wrote:

> +1 Tested on KVM (CentOS7) with NFS/Ceph/StorPool a few storage operations
> with a Basic Zone.
>
> Best regards,
> Slavka
>
> On Wed, Mar 2, 2022 at 4:02 PM Suresh Anaparti <
> suresh.anapa...@shapeblue.com> wrote:
>
> > +1
> >
> > Adding my vote as well. Manually tested various VM, Volume and Network
> > operations in Advanced Zone, on fresh and upgrade(from 4.15.2)
> environments
> > (MS: CentOS 7 + Hypervisor: CentOS 7 KVM). Also, verified the automated
> > smoke tests results in environments with MS on CentOS 7 and Hypervisors 
-
> > CentOS 7 KVM, XenServer 7.1, and VMware 6.5u2. No issues noticed.
> >
> >
> > Regards,
> > Suresh
> >
> > On 02/03/22, 6:03 PM, "Daan Hoogland"  wrote:
> >
> > +1
> >
> >
> >
> >
> 
 

> On Wed, Mar 2, 2022 at 1:29 PM Wei ZHOU  wrote:
> >
> > > +1 (binding)
> > >
> > > tested vm lifecycle on vmware70u3.
> > >
> > > -Wei
> > >
> > > On Fri, 25 Feb 2022 at 16:09, Suresh Anaparti <
> > > suresh.anapa...@shapeblue.com>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I have created a 4.16.1.0 release (RC2), with the following
> > artifacts up
> > > > for testing and a vote:
> > > >
> > > > Git Branch and Commit SHA:
> > > >
> https://github.com/apache/cloudstack/tree/4.16.1.0-RC20220225T1901
> > > > Commit: cad9332082a1f85eedc30cf547ae28224be170c2
> > > >
> > > > Source release (checksums and signatures are available at the
> same
> > > > location):
> > > > https://dist.apache.org/repos/dist/dev/cloudstack/4.16.1.0/
> > > >
> > > > PGP release keys (signed using
> > D6E0581ECF8A2FBE3FF6B3C9D7CEAE3A9E71D0AA):
> > > > https://dist.apache.org/repos/dist/release/cloudstack/KEYS
> > > >
> > > > The vote will be open until 2nd March 2022.
> > > >
> > > > For sanity in tallying the vote, can PMC members please be sure
> to
> > > > indicate "(binding)" with their vote?
> > > >
> > > > [ ] +1  approve
> > > > [ ] +0  no opinion
> > > > [ ] -1  disapprove (and reason why)
> > > >
> > > > For users convenience, the packages from this release candidate
> > (RC2) and
> > > > 4.16.1 systemvm templates are available here:
> > > > https://download.cloudstack.org/testing/4.16.1.0-RC2/
> > > > https://download.cloudstack.org/systemvm/4.16/
> > > >
> > > > Documentation is not published yet, but the following may be
> > referenced
> > > > for upgrade related tests:
> > > > (there's a new 4.16.1 systemvm template to be registered prior 
to
> > > upgrade)
> > > >
> > > >
> > >
> >
> 
https://github.com/apache/cloudstack-documentation/tree/4.16/source/upgrading/upgrade
> > > >
> > > > NOTES on the issues fixed in this RC2 release:
> > > >
> > > > (these do *NOT* require a full retest if you were testing RC1
> > already -
> > > > just if you were affected by these issues):
> > > > - https://github.com/apache/cloudstack/issues/6017 (regression
> in
> > > > register template form
> > > > when select/unselect check boxes using space in keyboard)
> > > > - https://github.com/apache/cloudstack/issues/6026 (affects
> > volumes on
> > > > managed storages when
> > > > stopping or migrating a VM)
> > > > - https://github.com/apache/cloudstack/issues/6038 (regression
> in
> > SSVM
> > > > scaling down behavior,
> > > > new config 'secstorage.vm.auto.scale.down' added to control
> > scaling down)
> > > >
> > > >
> > > > Regards,
> > > > Suresh
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
> >
> > --
> > Daan
> >
> >
>


-- 

Andrija Panić

microvm support

[Ricardo Pertuz]

Hi,

Is there a way to deploy microvm (firecracker approach) using qemu on 
Cloudstack?
As explained in this link
https://qemu.readthedocs.io/en/latest/system/i386/microvm.html

Regards,

Re: Corruption on SSVM

[Nux]

Yes, indeed.

I'd be taking a deep look at the underlying storage platform, 
something's afoot.


On 2022-03-03 08:55, Jeremy Hansen wrote:

I assume I would have to destroy the vm and allow it to launch a new 
one?  Will that allocate a new disk image to get rid of whatever 
corruption I'm seeing?


On Tuesday, Mar 01, 2022 at 6:56 PM, Jeremy Hansen 
 wrote:


I'm seeing what looks like some file system corruption or block device 
corruption on the secondary storage system vm:


Rebooting seems to fix it temporarily but the issues pop up within 24 
hours.  What's the best way to fix this?


-jeremy

Re: Corruption on SSVM

[Jeremy Hansen]

I assume I would have to destroy the vm and allow it to launch a new one? Will 
that allocate a new disk image to get rid of whatever corruption I’m seeing?

> On Tuesday, Mar 01, 2022 at 6:56 PM, Jeremy Hansen  (mailto:jer...@skidrow.la.INVALID)> wrote:
> I’m seeing what looks like some file system corruption or block device 
> corruption on the secondary storage system vm:
>
> Rebooting seems to fix it temporarily but the issues pop up within 24 hours. 
> What’s the best way to fix this?
>
> -jeremy
>
>
>
>


signature.asc
Description: PGP signature