Re: [ceph-users] cephfs and erasure coding

[Konstantin Shalygin]

My use case - from past ages /mail is block device for kvm vm. Now I 
need more space for messages, but I don't want use 3x raw space for 
replicas.


What is your reccomendations? Create an RBD image on an erasure coded 
pools when a replicated pool tier set a cache tier?


Thanks.

On 03/30/2017 07:09 AM, Christian Balzer wrote:

If you read that whole thread, you will have noticed my reply and
questions, Sami's question and the total lack of responses by Daniel.
  
W/o further data I'd be reluctant to call that a success, at least in a

general sense.

A native dovecot-ceph object interface will certainly help performance,
but of course still be somewhat limited by the network nature of things
and also a total black box compared to maildir on a FS.

Lastly, do you feel comfortable to put all your mail eggs into one
(software) storage basket?
At a million+ users I most certainly don't.


--
Best regards,
Konstantin Shalygin

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs and erasure coding

[Christian Balzer]

Hello,

On Wed, 29 Mar 2017 21:09:23 +0700 Konstantin Shalygin wrote:

> Thanks for notice. On dovecot mail list reported 
> https://dovecot.org/pipermail/dovecot/2016-August/105210.html about 
> success usage CephFS for 30-40k of users, with replica, not EC.
>
If you read that whole thread, you will have noticed my reply and
questions, Sami's question and the total lack of responses by Daniel.
 
W/o further data I'd be reluctant to call that a success, at least in a
general sense.

A native dovecot-ceph object interface will certainly help performance,
but of course still be somewhat limited by the network nature of things
and also a total black box compared to maildir on a FS. 

Lastly, do you feel comfortable to put all your mail eggs into one
(software) storage basket?
At a million+ users I most certainly don't.

Christian

> On 03/29/2017 08:19 PM, Wido den Hollander wrote:
> > I wouldn't use CephFS for so many small files. Dovecot will do a lot of 
> > locking, opening en closing those small files which is not very efficient.
> >
> > http://tracker.ceph.com/issues/12430
> >
> > That is in development right now. NO code yet out there, but should be 
> > there later this year.  
> 


-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Global OnLine Japan/Rakuten Communications
http://www.gol.com/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs and erasure coding

[Konstantin Shalygin]

Thanks for notice. On dovecot mail list reported 
https://dovecot.org/pipermail/dovecot/2016-August/105210.html about 
success usage CephFS for 30-40k of users, with replica, not EC.


On 03/29/2017 08:19 PM, Wido den Hollander wrote:

I wouldn't use CephFS for so many small files. Dovecot will do a lot of 
locking, opening en closing those small files which is not very efficient.

http://tracker.ceph.com/issues/12430

That is in development right now. NO code yet out there, but should be there 
later this year.


--
Best regards,
Konstantin Shalygin

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs and erasure coding

[Wido den Hollander]

> Op 29 maart 2017 om 8:54 schreef Konstantin Shalygin :
> 
> 
> Hello.
> 
> How your tests? I'm looking for CephFS with EC for save space on 
> replicas for many small files (dovecot mailboxes).

I wouldn't use CephFS for so many small files. Dovecot will do a lot of 
locking, opening en closing those small files which is not very efficient.

http://tracker.ceph.com/issues/12430

That is in development right now. NO code yet out there, but should be there 
later this year.

Wudo

> 
> > Thanks everyone for the input. We are online in our test environment and 
> > are running user workflows to make sure everything is running as expected.
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] cephfs and erasure coding

[Konstantin Shalygin]

Hello.

How your tests? I'm looking for CephFS with EC for save space on 
replicas for many small files (dovecot mailboxes).



Thanks everyone for the input. We are online in our test environment and are 
running user workflows to make sure everything is running as expected.


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs and erasure coding

[Rhian Resnick]

Thanks everyone for the input. We are online in our test environment and are 
running user workflows to make sure everything is running as expected.



Rhian

From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Rhian 
Resnick
Sent: Thursday, March 9, 2017 8:31 AM
To: Maxime Guyot 
Cc: ceph-us...@ceph.com
Subject: Re: [ceph-users] cephfs and erasure coding

Thanks for the confirmations of what is possible.

We plan on creating a new file system, rsync and delete the old one.

Rhian

On Mar 9, 2017 2:27 AM, Maxime Guyot 
mailto:maxime.gu...@elits.com>> wrote:

Hi,



>“The answer as to how to move an existing cephfs pool from replication to 
>erasure coding (and vice versa) is to create the new pool and rsync your data 
>between them.”

Shouldn’t it be possible to just do the “ceph osd tier add  ecpool cachepool && 
ceph osd tier cache-mode cachepool writeback” and let Ceph redirect the 
requests (CephFS or other) to the cache pool?



Cheers,

Maxime



From: ceph-users 
mailto:ceph-users-boun...@lists.ceph.com>> 
on behalf of David Turner 
mailto:david.tur...@storagecraft.com>>
Date: Wednesday 8 March 2017 22:27
To: Rhian Resnick mailto:rresn...@fau.edu>>, 
"ceph-us...@ceph.com<mailto:ceph-us...@ceph.com>" 
mailto:ceph-us...@ceph.com>>
Subject: Re: [ceph-users] cephfs and erasure coding



I use CephFS on erasure coding at home using a cache tier.  It works fine for 
my use case, but we know nothing about your use case to know if it will work 
well for you.

The answer as to how to move an existing cephfs pool from replication to 
erasure coding (and vice versa) is to create the new pool and rsync your data 
between them.



[cid:image001.jpg@01D298AE.DE1475E0]<https://storagecraft.com>


David Turner | Cloud Operations Engineer | StorageCraft Technology 
Corporation<https://storagecraft.com>
380 Data Drive Suite 300 | Draper | Utah | 84020
Office: 801.871.2760 | Mobile: 385.224.2943




If you are not the intended recipient of this message or received it 
erroneously, please notify the sender and delete it, together with any 
attachments, and be advised that any dissemination or copying of this message 
is prohibited.





From: ceph-users [ceph-users-boun...@lists.ceph.com] on behalf of Rhian Resnick 
[rresn...@fau.edu]
Sent: Wednesday, March 08, 2017 12:54 PM
To: ceph-us...@ceph.com<mailto:ceph-us...@ceph.com>
Subject: [ceph-users] cephfs and erasure coding

Two questions on Cephfs and erasure coding that Google couldn't answer.





1) How well does cephfs work with erasure coding?



2) How would you move an existing cephfs pool that uses replication to erasure 
coding?



Rhian Resnick

Assistant Director Middleware and HPC

Office of Information Technology



Florida Atlantic University

777 Glades Road, CM22, Rm 173B

Boca Raton, FL 33431

Phone 561.297.2647

Fax 561.297.0222

 [mage] <https://hpc.fau.edu/wp-content/uploads/2015/01/image.jpg>

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs and erasure coding

[Rhian Resnick]

Thanks for the confirmations of what is possible.

We plan on creating a new file system, rsync and delete the old one.

Rhian

On Mar 9, 2017 2:27 AM, Maxime Guyot  wrote:

Hi,



>“The answer as to how to move an existing cephfs pool from replication to 
>erasure coding (and vice versa) is to create the new pool and rsync your data 
>between them.”

Shouldn’t it be possible to just do the “ceph osd tier add  ecpool cachepool && 
ceph osd tier cache-mode cachepool writeback” and let Ceph redirect the 
requests (CephFS or other) to the cache pool?



Cheers,

Maxime



From: ceph-users  on behalf of David Turner 

Date: Wednesday 8 March 2017 22:27
To: Rhian Resnick , "ceph-us...@ceph.com" 

Subject: Re: [ceph-users] cephfs and erasure coding



I use CephFS on erasure coding at home using a cache tier.  It works fine for 
my use case, but we know nothing about your use case to know if it will work 
well for you.

The answer as to how to move an existing cephfs pool from replication to 
erasure coding (and vice versa) is to create the new pool and rsync your data 
between them.



[cid:image001.jpg@01D298AE.DE1475E0]<https://storagecraft.com>


David Turner | Cloud Operations Engineer | StorageCraft Technology 
Corporation<https://storagecraft.com>
380 Data Drive Suite 300 | Draper | Utah | 84020
Office: 801.871.2760 | Mobile: 385.224.2943




If you are not the intended recipient of this message or received it 
erroneously, please notify the sender and delete it, together with any 
attachments, and be advised that any dissemination or copying of this message 
is prohibited.





From: ceph-users [ceph-users-boun...@lists.ceph.com] on behalf of Rhian Resnick 
[rresn...@fau.edu]
Sent: Wednesday, March 08, 2017 12:54 PM
To: ceph-us...@ceph.com
Subject: [ceph-users] cephfs and erasure coding

Two questions on Cephfs and erasure coding that Google couldn't answer.





1) How well does cephfs work with erasure coding?



2) How would you move an existing cephfs pool that uses replication to erasure 
coding?



Rhian Resnick

Assistant Director Middleware and HPC

Office of Information Technology



Florida Atlantic University

777 Glades Road, CM22, Rm 173B

Boca Raton, FL 33431

Phone 561.297.2647

Fax 561.297.0222

 [mage] <https://hpc.fau.edu/wp-content/uploads/2015/01/image.jpg>

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs and erasure coding

[Maxime Guyot]

Hi,

>“The answer as to how to move an existing cephfs pool from replication to 
>erasure coding (and vice versa) is to create the new pool and rsync your data 
>between them.”
Shouldn’t it be possible to just do the “ceph osd tier add  ecpool cachepool && 
ceph osd tier cache-mode cachepool writeback” and let Ceph redirect the 
requests (CephFS or other) to the cache pool?

Cheers,
Maxime

From: ceph-users  on behalf of David Turner 

Date: Wednesday 8 March 2017 22:27
To: Rhian Resnick , "ceph-us...@ceph.com" 

Subject: Re: [ceph-users] cephfs and erasure coding

I use CephFS on erasure coding at home using a cache tier.  It works fine for 
my use case, but we know nothing about your use case to know if it will work 
well for you.

The answer as to how to move an existing cephfs pool from replication to 
erasure coding (and vice versa) is to create the new pool and rsync your data 
between them.

[cid:image001.jpg@01D298AE.DE1475E0]<https://storagecraft.com>

David Turner | Cloud Operations Engineer | StorageCraft Technology 
Corporation<https://storagecraft.com>
380 Data Drive Suite 300 | Draper | Utah | 84020
Office: 801.871.2760 | Mobile: 385.224.2943


If you are not the intended recipient of this message or received it 
erroneously, please notify the sender and delete it, together with any 
attachments, and be advised that any dissemination or copying of this message 
is prohibited.



From: ceph-users [ceph-users-boun...@lists.ceph.com] on behalf of Rhian Resnick 
[rresn...@fau.edu]
Sent: Wednesday, March 08, 2017 12:54 PM
To: ceph-us...@ceph.com
Subject: [ceph-users] cephfs and erasure coding

Two questions on Cephfs and erasure coding that Google couldn't answer.





1) How well does cephfs work with erasure coding?



2) How would you move an existing cephfs pool that uses replication to erasure 
coding?



Rhian Resnick

Assistant Director Middleware and HPC

Office of Information Technology



Florida Atlantic University

777 Glades Road, CM22, Rm 173B

Boca Raton, FL 33431

Phone 561.297.2647

Fax 561.297.0222

 [mage] <https://hpc.fau.edu/wp-content/uploads/2015/01/image.jpg>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs and erasure coding

[David Turner]

I use CephFS on erasure coding at home using a cache tier.  It works fine for 
my use case, but we know nothing about your use case to know if it will work 
well for you.

The answer as to how to move an existing cephfs pool from replication to 
erasure coding (and vice versa) is to create the new pool and rsync your data 
between them.



[cid:image75705d.JPG@8b6af7ae.4c8fbdce]<https://storagecraft.com>   David 
Turner | Cloud Operations Engineer | StorageCraft Technology 
Corporation<https://storagecraft.com>
380 Data Drive Suite 300 | Draper | Utah | 84020
Office: 801.871.2760 | Mobile: 385.224.2943



If you are not the intended recipient of this message or received it 
erroneously, please notify the sender and delete it, together with any 
attachments, and be advised that any dissemination or copying of this message 
is prohibited.




From: ceph-users [ceph-users-boun...@lists.ceph.com] on behalf of Rhian Resnick 
[rresn...@fau.edu]
Sent: Wednesday, March 08, 2017 12:54 PM
To: ceph-us...@ceph.com
Subject: [ceph-users] cephfs and erasure coding


Two questions on Cephfs and erasure coding that Google couldn't answer.



1) How well does cephfs work with erasure coding?


2) How would you move an existing cephfs pool that uses replication to erasure 
coding?


Rhian Resnick

Assistant Director Middleware and HPC

Office of Information Technology


Florida Atlantic University

777 Glades Road, CM22, Rm 173B

Boca Raton, FL 33431

Phone 561.297.2647

Fax 561.297.0222

 [image] <https://hpc.fau.edu/wp-content/uploads/2015/01/image.jpg>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs and erasure coding

[John Spray]

On Wed, Mar 8, 2017 at 7:54 PM, Rhian Resnick  wrote:

> Two questions on Cephfs and erasure coding that Google couldn't answer.
>
>
>
> 1) How well does cephfs work with erasure coding?
>

In the current released versions, you cannot use erasure coded pools with
CephFS, unless there is a replicated cache tier in between.  This isn't
generally advisable because cache tiers bring their own complexity.

The reason for all this is that currently, erasure coded pools only support
a subset of operations.  Notably they do not support overwriting objects
(i.e. modifying in place, the way we would in a filesystem when someone
writes to an existing file).

There is work underway to remove that limitation.  In the current master
(development) code it is already possible to use erasure coded pools
directly as cephfs data pools when a special setting is used to enable
overwrites in EC pools.

John

>
> 2) How would you move an existing cephfs pool that uses replication to
> erasure coding?
>




>
> Rhian Resnick
>
> Assistant Director Middleware and HPC
>
> Office of Information Technology
>
>
> Florida Atlantic University
>
> 777 Glades Road, CM22, Rm 173B
>
> Boca Raton, FL 33431
>
> Phone 561.297.2647 <(561)%20297-2647>
>
> Fax 561.297.0222 <(561)%20297-0222>
>
>  [image: image] 
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] cephfs and erasure coding

[Rhian Resnick]

Two questions on Cephfs and erasure coding that Google couldn't answer.



1) How well does cephfs work with erasure coding?


2) How would you move an existing cephfs pool that uses replication to erasure 
coding?


Rhian Resnick

Assistant Director Middleware and HPC

Office of Information Technology


Florida Atlantic University

777 Glades Road, CM22, Rm 173B

Boca Raton, FL 33431

Phone 561.297.2647

Fax 561.297.0222

 [image] 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com