Re: Is there any issue with Ceph RBD as Primary Storage to achieve HA-Enabled VM?

[Simon Weller]

Hi Nazmul,

Currently, RBD backed VMs will auto-recover as long as the host doesn't fail. 
If the underlying host fails, intervention will be required to take the host 
out of production to force the VMs to be restarted on another host.
There is a PR currently under review that will add full HA with heartbeat for 
RBD and KVM - https://github.com/apache/cloudstack/pull/5862

-Si

From: Nazmul Parvej 
Sent: Tuesday, June 28, 2022 6:05 AM
To: users@cloudstack.apache.org 
Cc: Product Development | BEXIMCO IT 
Subject: Is there any issue with Ceph RBD as Primary Storage to achieve 
HA-Enabled VM?

EXTERNAL EMAIL: This message originated outside of ENA. Use caution when 
clicking links, opening attachments, or complying with requests. Click the 
"Phish Alert Report" button above the email, or contact MIS, regarding any 
suspicious message.



Hello Team,

Is there any issue with Ceph RBD as Primary Storage to achieve HA-Enabled
VM?

I am using KVM Hypervisors and my CloudStack version is 4.17

Yours sincerely,


Nazmul Parvej
Deputy Manager, Product Development
IT Division

Bangladesh Export Import Company Ltd.

Level-9, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh

Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14193, Fax:  +880 2 
95757

Cell: +8801787680841, Email: nazmul.par...@bol-online.com, Web:
www.bol-online.com

Re: Asymmetric traffic issues?

[Vivek Kumar]

Hey Fuller,

What hypervisor are you using ? I know you have checked all bandwidth limit on 
templates and global settings, but it’s worth to check the QoS on the 
hypervisor level, because at the end it’s the hypervisor which manages all.  
And from where are you trying to check the network throughout,  between client 
and server ? 


Vivek Kumar
Sr. Manager - Cloud & DevOps
TechOps | Indiqus Technologies

+ 91 7503460090 
vivek.ku...@indiqus.com 
www.indiqus.com 



> On 28-Jun-2022, at 1:58 AM, S.Fuller  wrote:
> 
> Environment:
> 
> Two physical hosts
> - Cloudstack 4.11.3
> - Verified that there are no bandwidth limits in place on any of the
> templates or in global settings.
> 
> Two isolated networks ("Client" and  "Server")
> - Each has a vrouter with a public and private address
> - One Windows 2016 VM on each network (running the latest virtio drivers)
> - each node running latest version of Iperf3 to test throughput
> 
> Testing/Observation:
> 
> If the Client VM and the vrouter for the isolated Client network are on the
> same physical host, we see symmetrical throughput in the 2 Gbps range,
> whether we run iperf in regular mode or in reverse mode (iperf -R).
> 
> If the Client VM and the vrouter for the isolated Client network are on
> different physical hosts, we are seeing 25% of the throughput running iperf
> in regular mode vs running it in reverse mode.
> 
> Has anyone encountered this issue before? If we change the Client VM to
> Linux (either CentOS 7 or Ubuntu) OR we use the E1000 driver, we see
> symmetrical throughput in our tests, no matter where the vrouter is in
> relation to the Client VM.
> 
> -- 
> Steve Fuller
> steveful...@gmail.com


-- 
This message is intended only for the use of the individual or entity to 
which it is addressed and may contain confidential and/or privileged 
information. If you are not the intended recipient, please delete the 
original message and any copy of it from your computer system. You are 
hereby notified that any dissemination, distribution or copying of this 
communication is strictly prohibited unless proper authorization has been 
obtained for such action. If you have received this communication in error, 
please notify the sender immediately. Although IndiQus attempts to sweep 
e-mail and attachments for viruses, it does not guarantee that both are 
virus-free and accepts no liability for any damage sustained as a result of 
viruses.

Re: Volume download link on HTTPS

[Vivek Kumar]

Thanks Wei..! We have done exact the same thing but console is working on HTTPs 
but volume download is still on the HTTP..! I will re-check all the settings 
and will let you know..! 



Vivek Kumar
Sr. Manager - Cloud & DevOps
TechOps | Indiqus Technologies

+ 91 7503460090 
vivek.ku...@indiqus.com 
www.indiqus.com 



> On 27-Jun-2022, at 6:40 PM, Wei ZHOU  wrote:
> 
> Hi Vivek,
> 
> Please refer to
> https://www.shapeblue.com/securing-cloudstack-4-11-with-https-tls/ 
> 
> You need to change some global settings and restart mgtserver/ssvm.
> 
> -Wei
> 
> On Mon, 27 Jun 2022 at 14:44, Vivek Kumar  >
> wrote:
> 
>> Hello Folks,
>> 
>> 
>> We have option to download the volumes, when we click on the download
>> volume it will give you a http link. Can we somehow generate the HTTPs link
>> rather then the HTTP link. We have offloaded the SSL certificates in the
>> system VMs. We have not offloaded any certificate to our upfront load
>> balancer or ACS managements VMs.
>> 
>> 
>> 
>> 
>> Vivek Kumar
>> Sr. Manager - Cloud & DevOps
>> TechOps | Indiqus Technologies
>> 
>> + 91 7503460090 >
>> vivek.ku...@indiqus.com  
>> >
>> www.indiqus.com  > >
>> 
>> 
>> 
>> 
>> --
>> This message is intended only for the use of the individual or entity to
>> which it is addressed and may contain confidential and/or privileged
>> information. If you are not the intended recipient, please delete the
>> original message and any copy of it from your computer system. You are
>> hereby notified that any dissemination, distribution or copying of this
>> communication is strictly prohibited unless proper authorization has been
>> obtained for such action. If you have received this communication in
>> error,
>> please notify the sender immediately. Although IndiQus attempts to sweep
>> e-mail and attachments for viruses, it does not guarantee that both are
>> virus-free and accepts no liability for any damage sustained as a result
>> of
>> viruses.


-- 
This message is intended only for the use of the individual or entity to 
which it is addressed and may contain confidential and/or privileged 
information. If you are not the intended recipient, please delete the 
original message and any copy of it from your computer system. You are 
hereby notified that any dissemination, distribution or copying of this 
communication is strictly prohibited unless proper authorization has been 
obtained for such action. If you have received this communication in error, 
please notify the sender immediately. Although IndiQus attempts to sweep 
e-mail and attachments for viruses, it does not guarantee that both are 
virus-free and accepts no liability for any damage sustained as a result of 
viruses.

Is there any issue with Ceph RBD as Primary Storage to achieve HA-Enabled VM?

[Nazmul Parvej]

Hello Team,

Is there any issue with Ceph RBD as Primary Storage to achieve HA-Enabled
VM?

I am using KVM Hypervisors and my CloudStack version is 4.17

Yours sincerely,


Nazmul Parvej
Deputy Manager, Product Development
IT Division

Bangladesh Export Import Company Ltd.

Level-9, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh

Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14193, Fax:  +880 2 
95757

Cell: +8801787680841, Email: nazmul.par...@bol-online.com, Web:
www.bol-online.com

Re: Using S3 Storage for Secondary storage

[Vladimir Dombrovski]

Hello Antoine,

We've recently performed a thorough test of using S3 as a secondary
storage on our QA platform using the 4.17.0 release. We've tried with
2 S3 Providers (Ceph, OpenIO), without any success in both cases. We
will post one (or more) issues in the future concerning this subject,
in the meantime here's what ve found:

- The current S3 implementation doesn't support any advanced
parameters (such as region), which prevents it from working properly
with S3 providers that require the V4 signature. The culprit being a
deprecated use of the AWS S3 SDK for Java.
- The following API call:
https://cloudstack.apache.org/api/apidocs-4.17/apis/updateCloudToUseObjectStore.html,
does work, however as there is no validation on any of the provided
parameters, you are likely to get an NPE if you happen to miss one of
them. Also, this call is not reversible, even in the case of an error.
What it does under the hood is that it converts your current secondary
storage to a NFS staging store, and adds an Image store of type S3,
which will then initiate the download of all images present in your
secondary (making them unavailable until downloaded).
- Finally, the current implementation uses an asynchronous upload with
a "ProgressListener" that discards any errors (aka exceptions) thrown
to it. In practice you will see "Download error" on your templates,
without any errors in the logs (empty error message). So basically it
will either work fine if your S3 implementation works with it, or
fails without any possibility for you to know why.

Please note, this is not a critique of the feature (which I'm sure was
designed and tested against AWS S3 specifically, not S3 compatible
APIs), this is simply what we've found about the state of the current
implementation.

Regards,

Vladimir


On Mon, 27 Jun 2022 at 16:56, Levin Ng  wrote:
>
> Hi Antoine,
>
> I’m looking for same question too. I’ve tested few s3fs implementation and 
> only found rclone mount and juicejfs is working properly. However it require 
> a huge set of buffering area to store intermediate images.
>
> Regards,
> Levin
>
> From: Antoine Boucher 
> Date: Monday, 27 June 2022 at 21:02
> To: users 
> Subject: Using S3 Storage for Secondary storage
> Hello,
>
> We are consolidating our backup storage to S3 using MinIO. It appears that 
> migration from NFS-based secondary storage to S3 is none trivial task since 
> both can not coexist for the transition period. Has anyone done the 
> transition? We can’t lose the existing data from the current Secondary NFS 
> storage data.
>
> Alternatively, would an s3fs mounted on an NFS share work adequately for 
> CloudStack‘s requirement?
>
> Regards,
>
> Antoine

-- 
*CONFIDENTIALITY AND DISCLAIMER NOTICE: *
This email is intended only for 
the person to whom it is addressed and/or otherwise authorized personnel. 
The information contained herein and attached is confidential. If you are 
not the intended recipient, please be advised that viewing this message and 
any attachments, as well as copying, forwarding, printing, and 
disseminating any information related to this email is prohibited, and that 
you should not take any action based on the content of this email and/or 
its attachments. If you received this message in error, please contact the 
sender and destroy all copies of this email and any attachment. Please note 
that the views and opinions expressed herein are solely those of the author 
and do not necessarily reflect those of the company. While antivirus 
protection tools have been employed, you should check this email and 
attachments for the presence of viruses. No warranties or assurances are 
made in relation to the safety and content of this email and attachments. 
The Company accepts no liability for any damage caused by any virus 
transmitted by or contained in this email and attachments. No liability is 
accepted for any consequences arising from this email.


*AVIS DE 
CONFIDENTIALITÉ ET DE NON RESPONSABILITE* : 
Ce courriel, ainsi que toute 
pièce jointe, est confidentiel et peut être protégé par le secret 
professionnel. Si vous n’en êtes pas le destinataire visé, veuillez en 
aviser l’expéditeur immédiatement et le supprimer. Vous ne devez pas le 
copier, ni l’utiliser à quelque fin que ce soit, ni divulguer son contenu à 
qui que ce soit. BSO se réserve le droit de contrôler toute transmission 
qui passe par son réseau. Veuillez noter que les opinions exprimées dans 
cet e-mail sont uniquement celles de l'auteur et ne reflètent pas 
nécessairement celles de la société. Bien que des outils de protection 
antivirus aient été utilisés, vous devez vérifier cet e-mail et les pièces 
jointes pour toute présence de virus. Aucune garantie ou assurance n'est 
donnée concernant la sécurité et le contenu de cet e-mail et de ses pièces 
jointes. La Société décline toute responsabilité pour tout dommage causé 
par tout virus transmis par ou contenu dans cet e-mail et ses pièces 
jointes. Aucune responsabilit