Rohit,
Your reply LGTM, a few more lines from me:
- initially the export is NFS, that's how users/VMs will consume it as,
just clarifying, I know we want to keep this somewhat agnostic
- while there is no agent running there, as you noted, most of the stuff
can be configured either via userdata, udev rules or a combination of
both
- in terms of monitoring we could enable snmpd on the appliance and/or a
prometheus system exporter
On 2024-07-05 08:01, Rohit Yadav wrote:
Proposed design doc largely LGTM.
I've some additional suggestions and feedback to make requirements &
the first phase implementation more clearer and simpler:
*
+1 on implementing it in a hypervisor and storage agnostic manner.
*
Let's have the FS VMs owned by the caller (account or project), not
like a system-owned appliance. It would then be just like CKS in that
sense. This is because there is nothing special about the feature as in
users can't do it, it's really for (a) users who want the benefit of
shared storage but don't want to setup themselves and (b) orchestrate
such a feature via API/SDKs/automation. Advanced users may not prefer
to use it who want too many customisation and complexities.
*
To keep the first phase simple, let's drop adding support for
metrics/usage of FS VM and any other lifecycle that would need an agent
or need for the management servers to SSH/manage the FS VM at all.
Then, the scope can be limited to:
*
Orchestrate the initial FS VM setup that can be easily done via
user-data (config drive or VR depending on the network, cloud-init can
orchestrate NFS exports), the FS VM's nfs service can also listen on
all nics/IPs. This would make adding the FS capability to work out of
the box if somebody want to attach the FS to other networks later (than
the one it was initially created on).
*
Keep it simple: as there is no agent or mgmt server access needed or
required; any change to the FS properties or lifecycle could be done by
a FS VM reboot or recreation, as the FS VM is stateless and a separate
data disk holds the file share storage. For such operations, the UI can
clearly mention a warning or note that such an operation would cause
downtime due to reboot/recreate lifecycle operation of the FS VM.
*
Suggestions for the Lifecycle operations:
*
(*list & update API are given, should support pagination, listing by
name/keyword, by network, account/domain/project etc)
*
Create FS: initial user-data base FS VM setup (during initial setup,
disk can be check/formatted + mounted with fstab rules)
*
Recreate/restart FS: destroy & recreate FS VM, attach data disk before
starting the VM (setup can check and initialise disk if needed; and
grow/expand filesystem if underlying volume was resized).
*
Attach/detach FS (to/from network): simply CloudStack nic/network
attach/detach (worth checking if cloud-init or something in the
systemvm template automatically takes care of nic setup in the FS VM)
*
Expand FS size: this could be simply UI-based proxy to resizing data
disk, but resizing would cause recreating (or rebooting) or the FS VM,
for it to grow the FS (due to lack of agent or SSH access, this may be
acceptable in the first phase)
*
Delete FS: deleting FS with/without expunging the data disk; and for
users to recover a non-expunged FS (similar to VMs)
*
FSM states: FS should have states that correspond to the FS VM running
and state of the underlying data disk
*
Misc: Ensure FS VM is HA enabled, worth also either assuming some
default compute offering or allow caller to specify compute offering
for the FS VM.
*
Network support: support all networks except L2 or networks which don't
have userdata & dhcp capabilities
*
Hypervisor & Storage support: agnostic
*FS = File Shares (suggested name)
Regards.
________________________________
From: Alex Mattioli <alex.matti...@shapeblue.com>
Sent: Wednesday, June 19, 2024 15:13
To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>
Cc: us...@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: RE: [Proposal] Storage Filesystem as a First Class Feature
+1 on that, keeping it hypervisor agnostic is key.
-----Original Message-----
From: Nux <n...@li.nux.ro>
Sent: Wednesday, June 19, 2024 10:14 AM
To: dev@cloudstack.apache.org
Cc: us...@cloudstack.apache.org
Subject: Re: [Proposal] Storage Filesystem as a First Class Feature
Thanks Piotr,
This is the second time virtio-fs has been mentioned and just
researched it a bit, it looks like something really nice to have in
Cloudstack, definitely something to look at in the future.
Nice as it is though, it has a big drawback, it's KVM-only, so for now
we'll stick to "old school" tech that can be used in an agnostic
matter.
You are more than welcome to share thoughts on the other details
presented, perhaps pros/cons on filesystems and other gotchas you may
have encountered yourself.
On 2024-06-19 07:04, Piotr Pisz wrote:
Hi,
We considered a similar problem in our company.
Shared storage is needed between VMs running on different networks.
NFS/CephFS is ok as long as the VM can see the source.
The best solution would be to use https://virtio-fs.gitlab.io/ Any FS
would be used on the host side (e.g. NFS or CephFS) and exported to
the VM natively (the network problem disappears).
But you should start by introducing an appropriate mechanism on the CS
side (similar in operation to Manila Share from Openstack).
So, the initiative itself is very good.
Overall, CloudStack has been heading in the right direction lately :-)
Best regards,
Piotr
-----Original Message-----
From: Nux <n...@li.nux.ro>
Sent: Wednesday, June 19, 2024 12:59 AM
To: dev@cloudstack.apache.org; Users <us...@cloudstack.apache.org>
Subject: Re: [Proposal] Storage Filesystem as a First Class Feature
Hi, I'd like to draw the attention to some of the more operational
aspects of this feature, mainly the storage appliance internals and
UI.
So long story short, I've discussed with Abhisar and others and we'll
be deploying a VM based on the Cloudstack Debian systemvm template
which will export NFS v3/4 for user VMs to consume.
Below are some of the more finer details, please have a read if you
are interested in this feature and feel free to comment and make
suggestions.
1 - The appliance will only have a single export, that export will be
a single disk (data volume). Keep it simple.
2 - GPT partition table and a single partition, filesystem probably
XFS and/or customisable - something stock Debian supports, simple and
boring stuff.
3 - NFS export should be simple, we can standardise on a path name eg
/nfs or /fileshare and it will be identical on all appliances.
4 - Starting specs: 2 cores, 4 GB RAM - should be OK for a small NFS
server, the appliance can be upgraded to bigger offerings.
5 - Disk offering should be flagged accordingly, the disk offering
will have a flag/checkbox for "storage appliance" use.
6 - This appliance will not be a system VM, it will be a "blackbox",
but the approach will be similar here to CKS.
7 - Security model: by default we export to * (all hosts) into a
single network - for isolated networks - in SG zones we need to play
with security groups & a global setting for dumb shared networks
(without SG) because of security implications - requires further
discussion.
8 - We export with default, best practices NFS options - anything
against no_root_squash?
9 - Explore exporting the file share via multiple protocols - sftp,
tftp, smb, nfs, http(s)? - The issue here is authentication becomes a
problem, also user permissions will get messy and possibly conflict
with no_root_squash, in fact might require an all_squash and
everything mapped to a single user that will be then also used for all
those other services.
Also
logging will become necessary. Thoughts?
10 - UI details, but this will probably show up in the Storage section
somehow.
11 - Display free/used space, create alerts for full disk etc for this
appliance.
12 - Formatting and setting up to be done by an internal agent,
specifics are sent via the kernel cmd line of the VM, similar to how
we configure system VMs.
What do you folks think of these points and have I missed anything
crucial?
On 2024-06-04 05:04, Abhisar Sinha wrote:
Hi,
I would like to propose supporting storage filesystem as a
first-class feature in Cloudstack.
The File Share can be associated with one or more guest networks or
vpc tiers and can be used by any VM on the network in a shared
manner. It is designed to be resizable and highly available. This
feature can later be used as integration endpoints with the CSI
driver, go-sdk, Terraform, Ansible and others.
The draft functional spec is here :
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Storage+Filesys
tem+as
+a+First+Class+Feature
Looking forward to your comments and suggestions.
Thanks,
Abhisar
