[ceph-users] Re: Setting S3 bucket policies with multi-tenants

[Kai Stian Olstad]

On 12.04.2024 20:54, Wesley Dillingham wrote:
Did you actually get this working? I am trying to replicate your steps 
but

am not being successful doing this with multi-tenant.


This is what we are using, the second statement is so that bucket owner 
will have access to the object that the user is uploading.


s3-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
{
  "Effect": "Allow",
  "Principal": {
"AWS": [
  "arn:aws:iamuser/"
]
  },
  "Action": [
"s3:ListBucket",
"s3:GetObject",
"s3:PutObject"
  ],
  "Resource": [
"arn:aws:s3:::/*",
"arn:aws:s3:::"
  ]
},
{
  "Sid": "owner_full_access",
  "Effect": "Allow",
  "Principal": {
"AWS": [
  "arn:aws:iamuser/"
]
  },
  "Action": "s3:*",
  "Resource": "arn:aws:s3:::*"
}
  ]
}

And then run
s3cmd setpolicy s3-policy.json s3://


--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Performance of volume size, not a block size

[Mitsumasa KONDO]

Hi Menguy-san,

Thank you for your reply. Users who use large IO with tiny volumes are a
nuisance to cloud providers.

I confirmed my ceph cluster with 40 SSDs. Each OSD on 1TB SSD has about 50
placement groups in my cluster. Therefore, each PG has approximately 20GB
of space.
If we create a small 8GB volume, I had a feeling it wouldn't be distributed
well, but it will be distributed well.

Regards,
--
Mitsumasa KONDO

2024年4月15日(月) 15:29 Etienne Menguy :

> Hi,
>
> Volume size doesn't affect performance, cloud providers apply a limit to
> ensure they can deliver expected performances to all their customers.
>
> Étienne
> --
> *From:* Mitsumasa KONDO 
> *Sent:* Monday, 15 April 2024 06:06
> *To:* ceph-users@ceph.io 
> *Subject:* [ceph-users] Performance of volume size, not a block size
>
> [Some people who received this message don't often get email from
> kondo.mitsum...@gmail.com. Learn why this is important at
> https://aka.ms/LearnAboutSenderIdentification ]
>
> Hi,
>
> In AWS EBS gp3, AWS says that small volume size cannot achieve best
> performance. I think it's a feature or tendency of general
> distributed storages including Ceph. Is that right in Ceph block storage? I
> read many docs on ceph community. I never heard of Ceph storage.
>
>
> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.aws.amazon.com%2Febs%2Flatest%2Fuserguide%2Fgeneral-purpose.html&data=05%7C02%7Cetienne.menguy%40ubisoft.com%7C3076825a4d2a4897074208dc5d017852%7Ce01bd386fa514210a2a429e5ab6f7ab1%7C0%7C0%7C638487508098942744%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=wOQKqG41uccTbyNHDIps62ojcTFBZYlyxxp3TzccsJI%3D&reserved=0
> 
>
> Regard,
> --
> Mitsumasa KONDO
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Migrating from S3 to Ceph RGW (Cloud Sync Module)

[Ondřej Kukla]

Hello,

If you have a quite large amount of data you can maybe try the Chorus from 
CLYSO.

https://docs.clyso.com/blog/2024/01/24/opensourcing-chorus-project
Opensourcing Chorus project | Clyso GmbH
docs.clyso.com
https://github.com/clyso/chorus
clyso/chorus: s3 multi provider data lifecycle management
github.com

It's using rclone for the copying, but it has some tricks that can be handy 
with large amounts of data.

Ondrej

> On 12. 4. 2024, at 4:00, Vladimir Sigunov  wrote:
> 
> Hello,
> I used to use rclone for data synchronization between 2 ceph clusters and for 
> a directional sync from AWS to Ceph.
> In general, rclone is a really good and reliable pice of software, but could 
> be slow with large amount of syncing objects. Large  - 10^6+ objects.
> As a disclaimer  - my experience is 3 years old. Very likely rclone was 
> improved since that time, and it definitely should be considered at least as 
> POC.
> You can offload rclone's sync operations by skipping hashes and other costly 
> comparisons if this is an appropriate approach in your project.
> Good luck!
> Sincerely,
> Vladimir.
> 
> Get Outlook for Android
> 
> From: Casey Bodley 
> Sent: Thursday, April 11, 2024 5:29:30 PM
> To: James McClune 
> Cc: ceph-users@ceph.io 
> Subject: [ceph-users] Re: Migrating from S3 to Ceph RGW (Cloud Sync Module)
> 
> unfortunately, this cloud sync module only exports data from ceph to a
> remote s3 endpoint, not the other way around:
> 
> "This module syncs zone data to a remote cloud service. The sync is
> unidirectional; data is not synced back from the remote zone."
> 
> i believe that rclone supports copying from one s3 endpoint to
> another. does anyone have experience with that?
> 
> On Thu, Apr 11, 2024 at 4:45 PM James McClune  wrote:
>> 
>> Hello Ceph User Community,
>> 
>> I currently have a large Amazon S3 environment with terabytes of data
>> spread over dozens of buckets. I'm looking to migrate from Amazon S3 to an
>> on-site Ceph cluster using the RGW. I'm trying to figure out the
>> most efficient way to achieve this. Looking through the documentation, I
>> found articles related to the cloud sync module, released in Mimic (
>> https://docs.ceph.com/en/latest/radosgw/cloud-sync-module/). I also watched
>> a video on the cloud sync module as well. It *sounds* like this is the
>> functionality I'm looking for.
>> 
>> Given I'm moving away from Amazon S3, I'm really just looking for a one-way
>> replication between the buckets (i.e. Provide an Amazon S3 access
>> key/secret which is associated to the buckets and the same for the Ceph
>> environment, so object data can be replicated one-to-one, without creating
>> ad-hoc tooling). Once the data is replicated from S3 to Ceph, I plan on
>> modifying my boto connection objects to use the new Ceph environment. Is
>> what I'm describing feasible with the cloud sync module? Just looking for
>> some affirmation, given I'm not well versed in Ceph's RGW, especially
>> around multi-site configurations.
>> 
>> Thanks,
>> Jimmy
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
>> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Performance of volume size, not a block size

[Anthony D'Atri]

If you're using SATA/SAS SSDs I would aim for 150-200 PGs per OSD as shown by 
`ceph osd df`.
If NVMe, 200-300 unless you're starved for RAM.


> On Apr 15, 2024, at 07:07, Mitsumasa KONDO  wrote:
> 
> Hi Menguy-san,
> 
> Thank you for your reply. Users who use large IO with tiny volumes are a
> nuisance to cloud providers.
> 
> I confirmed my ceph cluster with 40 SSDs. Each OSD on 1TB SSD has about 50
> placement groups in my cluster. Therefore, each PG has approximately 20GB
> of space.
> If we create a small 8GB volume, I had a feeling it wouldn't be distributed
> well, but it will be distributed well.
> 
> Regards,
> --
> Mitsumasa KONDO
> 
> 2024年4月15日(月) 15:29 Etienne Menguy :
> 
>> Hi,
>> 
>> Volume size doesn't affect performance, cloud providers apply a limit to
>> ensure they can deliver expected performances to all their customers.
>> 
>> Étienne
>> --
>> *From:* Mitsumasa KONDO 
>> *Sent:* Monday, 15 April 2024 06:06
>> *To:* ceph-users@ceph.io 
>> *Subject:* [ceph-users] Performance of volume size, not a block size
>> 
>> [Some people who received this message don't often get email from
>> kondo.mitsum...@gmail.com. Learn why this is important at
>> https://aka.ms/LearnAboutSenderIdentification ]
>> 
>> Hi,
>> 
>> In AWS EBS gp3, AWS says that small volume size cannot achieve best
>> performance. I think it's a feature or tendency of general
>> distributed storages including Ceph. Is that right in Ceph block storage? I
>> read many docs on ceph community. I never heard of Ceph storage.
>> 
>> 
>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.aws.amazon.com%2Febs%2Flatest%2Fuserguide%2Fgeneral-purpose.html&data=05%7C02%7Cetienne.menguy%40ubisoft.com%7C3076825a4d2a4897074208dc5d017852%7Ce01bd386fa514210a2a429e5ab6f7ab1%7C0%7C0%7C638487508098942744%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=wOQKqG41uccTbyNHDIps62ojcTFBZYlyxxp3TzccsJI%3D&reserved=0
>> 
>> 
>> Regard,
>> --
>> Mitsumasa KONDO
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
>> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] cephFS on CentOS7

[Dario Graña]

Hello everyone!

We deployed a platform with Ceph Quincy and now we need to give access to
some old nodes with CentOS7 until 30/07/2024. I found two approaches, the
first one, deploying Ganesha NFS and bringing access through the NFS
protocol. The second one is to use an older cephfs client, specifically the
Octopus client.
I would like to know if there is a third option and which the community
would recommend.
Thanks in advance.

Regards!
-- 
Dario Graña
PIC (Port d'Informació Científica)
Campus UAB, Edificio D
E-08193 Bellaterra, Barcelona
http://www.pic.es
Avis - Aviso - Legal Notice: http://legal.ifae.es
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Migrating from S3 to Ceph RGW (Cloud Sync Module)

[Mark Nelson]

At Clyso we've been building a tool that can migrate S3 data around 
called Chorus.  Normally I wouldn't promote it here, but it's open 
source and sounds like it might be useful in this case.  I don't work on 
it myself, but thought I'd mention it:


https://github.com/clyso/chorus

One problem with rclone is that it doesn't support copying ACLs AFAIK. 
There was an incomplete PR for that a while back that never got merged. 
We did some work on that as well, though I'm not sure if it's been up 
streamed yet.


Mark

On 4/11/24 16:29, Casey Bodley wrote:

unfortunately, this cloud sync module only exports data from ceph to a
remote s3 endpoint, not the other way around:

"This module syncs zone data to a remote cloud service. The sync is
unidirectional; data is not synced back from the remote zone."

i believe that rclone supports copying from one s3 endpoint to
another. does anyone have experience with that?

On Thu, Apr 11, 2024 at 4:45 PM James McClune  wrote:


Hello Ceph User Community,

I currently have a large Amazon S3 environment with terabytes of data
spread over dozens of buckets. I'm looking to migrate from Amazon S3 to an
on-site Ceph cluster using the RGW. I'm trying to figure out the
most efficient way to achieve this. Looking through the documentation, I
found articles related to the cloud sync module, released in Mimic (
https://docs.ceph.com/en/latest/radosgw/cloud-sync-module/). I also watched
a video on the cloud sync module as well. It *sounds* like this is the
functionality I'm looking for.

Given I'm moving away from Amazon S3, I'm really just looking for a one-way
replication between the buckets (i.e. Provide an Amazon S3 access
key/secret which is associated to the buckets and the same for the Ceph
environment, so object data can be replicated one-to-one, without creating
ad-hoc tooling). Once the data is replicated from S3 to Ceph, I plan on
modifying my boto connection objects to use the new Ceph environment. Is
what I'm describing feasible with the cloud sync module? Just looking for
some affirmation, given I'm not well versed in Ceph's RGW, especially
around multi-site configurations.

Thanks,
Jimmy
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Setting up Hashicorp Vault for Encryption with Ceph

[Michael Worsham]

Is there a how-to document available on how to setup Hashicorp's Vault for 
Ceph, preferably in a HA state?

Due to some encryption needs, we need to set up LUKS, OSD encryption AND Ceph 
bucket encryption as well. Yes, we know there will be a performance hit, but 
the encrypt-everything is a hard requirement for our business needs since we 
have government and healthcare-related contracts.

-- Michael

This message and its attachments are from Data Dimensions and are intended only 
for the use of the individual or entity to which it is addressed, and may 
contain information that is privileged, confidential, and exempt from 
disclosure under applicable law. If the reader of this message is not the 
intended recipient, or the employee or agent responsible for delivering the 
message to the intended recipient, you are hereby notified that any 
dissemination, distribution, or copying of this communication is strictly 
prohibited. If you have received this communication in error, please notify the 
sender immediately and permanently delete the original email and destroy any 
copies or printouts of this email as well as any attachments.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.3 QE validation status

[Laura Flores]

smoke approved.

Infrastructure:
1. https://tracker.ceph.com/issues/64727 - suites/dbench.sh: Socket
exception: No route to host (113)

On Sun, Apr 14, 2024 at 2:08 PM Adam King  wrote:

> orch approved
>
> On Fri, Apr 12, 2024 at 2:38 PM Yuri Weinstein 
> wrote:
>
>> Details of this release are summarized here:
>>
>> https://tracker.ceph.com/issues/65393#note-1
>> Release Notes - TBD
>> LRC upgrade - TBD
>>
>> Seeking approvals/reviews for:
>>
>> smoke - infra issues, still trying, Laura PTL
>>
>> rados - Radek, Laura approved? Travis?  Nizamudeen?
>>
>> rgw - Casey approved?
>> fs - Venky approved?
>> orch - Adam King approved?
>>
>> krbd - Ilya approved
>> powercycle - seems fs related, Venky, Brad PTL
>>
>> ceph-volume - will require
>>
>> https://github.com/ceph/ceph/pull/56857/commits/63fe3921638f1fb7fc065907a9e1a64700f8a600
>> Guillaume is fixing it.
>>
>> TIA
>> ___
>> Dev mailing list -- d...@ceph.io
>> To unsubscribe send an email to dev-le...@ceph.io
>>
>> ___
> Dev mailing list -- d...@ceph.io
> To unsubscribe send an email to dev-le...@ceph.io
>


-- 

Laura Flores

She/Her/Hers

Software Engineer, Ceph Storage 

Chicago, IL

lflo...@ibm.com | lflo...@redhat.com 
M: +17087388804
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.3 QE validation status

[Yuri Weinstein]

Still waiting for approvals:

rados - Radek, Laura approved? Travis?  Nizamudeen?

ceph-volume issue was fixed by https://github.com/ceph/ceph/pull/56857

We plan not to upgrade the LRC to 18.2.3 as we are very close to the
first squid RC and will be using it for this purpose.
Please speak up if this may present any issues.

Thx

On Fri, Apr 12, 2024 at 11:37 AM Yuri Weinstein  wrote:
>
> Details of this release are summarized here:
>
> https://tracker.ceph.com/issues/65393#note-1
> Release Notes - TBD
> LRC upgrade - TBD
>
> Seeking approvals/reviews for:
>
> smoke - infra issues, still trying, Laura PTL
>
> rados - Radek, Laura approved? Travis?  Nizamudeen?
>
> rgw - Casey approved?
> fs - Venky approved?
> orch - Adam King approved?
>
> krbd - Ilya approved
> powercycle - seems fs related, Venky, Brad PTL
>
> ceph-volume - will require
> https://github.com/ceph/ceph/pull/56857/commits/63fe3921638f1fb7fc065907a9e1a64700f8a600
> Guillaume is fixing it.
>
> TIA
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Performance of volume size, not a block size

[Mitsumasa KONDO]

Hi Anthony-san,

Thank you for your advice. I confirm my settings of my ceph cluster.
Autoscaler mode is on, so I had thought it's the best PGs. But the
autoscaler feature doesn't affect OSD's PGs. It's just for PG_NUM in
storage pools. Is that right?

Regards,
--
Mitsumasa KONDO


2024年4月15日(月) 22:58 Anthony D'Atri :

> If you're using SATA/SAS SSDs I would aim for 150-200 PGs per OSD as shown
> by `ceph osd df`.
> If NVMe, 200-300 unless you're starved for RAM.
>
>
> > On Apr 15, 2024, at 07:07, Mitsumasa KONDO 
> wrote:
> >
> > Hi Menguy-san,
> >
> > Thank you for your reply. Users who use large IO with tiny volumes are a
> > nuisance to cloud providers.
> >
> > I confirmed my ceph cluster with 40 SSDs. Each OSD on 1TB SSD has about
> 50
> > placement groups in my cluster. Therefore, each PG has approximately 20GB
> > of space.
> > If we create a small 8GB volume, I had a feeling it wouldn't be
> distributed
> > well, but it will be distributed well.
> >
> > Regards,
> > --
> > Mitsumasa KONDO
> >
> > 2024年4月15日(月) 15:29 Etienne Menguy :
> >
> >> Hi,
> >>
> >> Volume size doesn't affect performance, cloud providers apply a limit to
> >> ensure they can deliver expected performances to all their customers.
> >>
> >> Étienne
> >> --
> >> *From:* Mitsumasa KONDO 
> >> *Sent:* Monday, 15 April 2024 06:06
> >> *To:* ceph-users@ceph.io 
> >> *Subject:* [ceph-users] Performance of volume size, not a block size
> >>
> >> [Some people who received this message don't often get email from
> >> kondo.mitsum...@gmail.com. Learn why this is important at
> >> https://aka.ms/LearnAboutSenderIdentification ]
> >>
> >> Hi,
> >>
> >> In AWS EBS gp3, AWS says that small volume size cannot achieve best
> >> performance. I think it's a feature or tendency of general
> >> distributed storages including Ceph. Is that right in Ceph block
> storage? I
> >> read many docs on ceph community. I never heard of Ceph storage.
> >>
> >>
> >>
> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.aws.amazon.com%2Febs%2Flatest%2Fuserguide%2Fgeneral-purpose.html&data=05%7C02%7Cetienne.menguy%40ubisoft.com%7C3076825a4d2a4897074208dc5d017852%7Ce01bd386fa514210a2a429e5ab6f7ab1%7C0%7C0%7C638487508098942744%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=wOQKqG41uccTbyNHDIps62ojcTFBZYlyxxp3TzccsJI%3D&reserved=0
> >> 
> >>
> >> Regard,
> >> --
> >> Mitsumasa KONDO
> >> ___
> >> ceph-users mailing list -- ceph-users@ceph.io
> >> To unsubscribe send an email to ceph-users-le...@ceph.io
> >>
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Performance of volume size, not a block size

[Janne Johansson]

Den mån 15 apr. 2024 kl 13:09 skrev Mitsumasa KONDO :
> Hi Menguy-san,
>
> Thank you for your reply. Users who use large IO with tiny volumes are a
> nuisance to cloud providers.
>
> I confirmed my ceph cluster with 40 SSDs. Each OSD on 1TB SSD has about 50
> placement groups in my cluster. Therefore, each PG has approximately 20GB
> of space.
> If we create a small 8GB volume, I had a feeling it wouldn't be distributed
> well, but it will be distributed well.

RBD images get split into 2 or 4M pieces when stored in ceph, so an 8G
RBD image will be split into 2048-or-4096 separate pieces that end up
"randomly" on the PGs the pool is based on, which means that if you
read or write the whole RBD image from start to end, you are going to
spread the load to all OSDs.

I think it works something like this, you ask librbd for an 8G image
named "myimage", and underneath it makes myimage.0, myimage.1, 2,3,4
and so on. The PG placement will depend on the object name, which of
course differs for all the pieces, and hence they end up on different
PGs, thereby spreading the load. If ceph did not do this, then you
could never make an RBD image that was larger than "smallest free
space on any of the pools OSDs" but also, it would mean that the RBD
client would be talking to the same single OSD for everything, and
that would not be a good way to use a clusters resources evenly.

-- 
May the most significant bit of your life be positive.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io