Re: [ceph-users] journal size suggestions

[Andrey Korolyov]

On Wed, Jul 10, 2013 at 3:28 AM, Gandalf Corvotempesta
 wrote:
> Thank you for the response.
> You are talking of median expected writes, but should I consider the single
> disk write speed or the network speed? A single disk is 100MB/s so
> 100*30=3000MB of journal for each osd? Or should I consider the network
> speed that is 1.25GB/s?
> Why 30 seconds? default flush frequency is 5 seconds.
> What do you mean with fine tuning spinning storage media? On which tuning
> are you referring to?
>
Since journal is created on per-osd basis, you should calculate it
with only disk speed in mind. As I remember no one referred directly
to flush interval when recommending referring to tens of seconds on
such calculation, neither do I - it`s just a safe road anyway to have
some capacity over this value. By fine tuning I meant such things as
readahead values, number of internal XFS partitions, size of XFS
chunks, hardware controller cache policy(if you have some) and so on -
being honest, filesystem tuning is not affecting performance so much
on general workload types, but may affect greatly on some specific
things like digits in the benchmark :) .

> Il giorno 09/lug/2013 23:45, "Andrey Korolyov"  ha scritto:
>
>> On Wed, Jul 10, 2013 at 1:16 AM, Gandalf Corvotempesta
>>  wrote:
>> > Hi,
>> > i'm planning a new cluster on a 10GbE network.
>> > Each storage node will have a maximum of 12 SATA disks and 2 SSD as
>> > journals.
>> >
>> > What do you suggest as journal size for each OSD? 5GB is enough?
>> > Should I just consider SATA writing speed when calculating journal
>> > size or also network speed?
>>
>> Hello,
>>
>> As many recommendations suggests before, you may set journal size
>> proportional to amount of median (or peak, if expected) writes
>> multiplied, say, by thirty seconds - that`s the safe area and you
>> should not able to suffer because of journal size following this
>> calculation. Twelve SATA disks in theory may have enough output to
>> thrash 10G network but you`ll face lack of IOPS times before almost
>> for sure, and OSD daemons are not working very close to the physical
>> limits speaking of transferring data from/to disk, so fine tuning of
>> spinning storage media still is primary target to play with in such
>> configuration.
>>
>> > ___
>> > ceph-users mailing list
>> > ceph-users@lists.ceph.com
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] journal size suggestions

[Gandalf Corvotempesta]

Thank you for the response.
You are talking of median expected writes, but should I consider the single
disk write speed or the network speed? A single disk is 100MB/s so
100*30=3000MB of journal for each osd? Or should I consider the network
speed that is 1.25GB/s?
Why 30 seconds? default flush frequency is 5 seconds.
What do you mean with fine tuning spinning storage media? On which tuning
are you referring to?
Il giorno 09/lug/2013 23:45, "Andrey Korolyov"  ha scritto:

> On Wed, Jul 10, 2013 at 1:16 AM, Gandalf Corvotempesta
>  wrote:
> > Hi,
> > i'm planning a new cluster on a 10GbE network.
> > Each storage node will have a maximum of 12 SATA disks and 2 SSD as
> journals.
> >
> > What do you suggest as journal size for each OSD? 5GB is enough?
> > Should I just consider SATA writing speed when calculating journal
> > size or also network speed?
>
> Hello,
>
> As many recommendations suggests before, you may set journal size
> proportional to amount of median (or peak, if expected) writes
> multiplied, say, by thirty seconds - that`s the safe area and you
> should not able to suffer because of journal size following this
> calculation. Twelve SATA disks in theory may have enough output to
> thrash 10G network but you`ll face lack of IOPS times before almost
> for sure, and OSD daemons are not working very close to the physical
> limits speaking of transferring data from/to disk, so fine tuning of
> spinning storage media still is primary target to play with in such
> configuration.
>
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] journal size suggestions

[Andrey Korolyov]

On Wed, Jul 10, 2013 at 1:16 AM, Gandalf Corvotempesta
 wrote:
> Hi,
> i'm planning a new cluster on a 10GbE network.
> Each storage node will have a maximum of 12 SATA disks and 2 SSD as journals.
>
> What do you suggest as journal size for each OSD? 5GB is enough?
> Should I just consider SATA writing speed when calculating journal
> size or also network speed?

Hello,

As many recommendations suggests before, you may set journal size
proportional to amount of median (or peak, if expected) writes
multiplied, say, by thirty seconds - that`s the safe area and you
should not able to suffer because of journal size following this
calculation. Twelve SATA disks in theory may have enough output to
thrash 10G network but you`ll face lack of IOPS times before almost
for sure, and OSD daemons are not working very close to the physical
limits speaking of transferring data from/to disk, so fine tuning of
spinning storage media still is primary target to play with in such
configuration.

> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Journal size

[Sebastien Han]

I just edited the post, I opted for the 'noout' option at the end. It's the one that makes the most sense to me.Sébastien HanCloud Engineer"Always give 100%. Unless you're giving blood."PHONE : +33 (0)1 49 70 99 72 – MOBILE : +33 (0)6 52 84 44 70EMAIL : sebastien@enovance.com – SKYPE : han.sbastienADDRESS : 10, rue de la Victoire – 75009 ParisWEB : www.enovance.com – TWITTER : @enovanceOn Mar 27, 2013, at 5:32 PM, Sebastien Han  wrote:Yes I will :-), thank you for pointing out this to me.Sébastien HanCloud Engineer"Always give 100%. Unless you're giving blood."PHONE : +33 (0)1 49 70 99 72 – MOBILE : +33 (0)6 52 84 44 70EMAIL : sebastien@enovance.com – SKYPE : han.sbastienADDRESS : 10, rue de la Victoire – 75009 ParisWEB : www.enovance.com – TWITTER : @enovanceOn Mar 27, 2013, at 5:10 PM, John Nielsen  wrote:Indeed, thanks! Sébastien, care to update your helpful blog post?On Mar 27, 2013, at 10:04 AM, Gregory Farnum  wrote:This should actually be "down", not "out". Out will trigger a data migration. :)-GregSoftware Engineer #42 @ http://inktank.com | http://ceph.comOn Wed, Mar 27, 2013 at 8:36 AM, John Nielsen  wrote:Sébastien Han actually covers this on his blog:http://www.sebastien-han.fr/blog/2012/08/17/ceph-storage-node-maintenance/His use case is moving from an embedded journal to one on an SSD but the steps can be used to move/recreate the journal for any reason:Mark the OSD "out"Stop the OSD daemonRun "ceph-osd -i N --flush-journal", replacing N with the ID of the OSDGet your new journal location ready if it isn'tUpdate ceph.conf with the new journal settingsRun "ceph-osd -i N --mkjournal", again replacing N with the ID of the OSDStart the OSD daemonMark the OSD "in"I used the procedure above to migrate 16 OSD's on 4 hosts from file-based journals to partition-based with no downtime or data loss. (Just be aware of your crush rules and only have one OSD, host, etc. down at a time.)JNOn Mar 27, 2013, at 7:03 AM, Aleksey Samarin  wrote:Well, after few tests, i can't increase size of journal. I think size of journal set only when osd create. So for change journal size u need to delete osd and create with new journal.2013/3/26 Chen, Xiaoxi Are you using a partition as journal?From: ceph-users-boun...@lists.ceph.com [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Aleksey SamarinSent: 2013年3月26日 20:45To: ceph-us...@ceph.comSubject: [ceph-users] Journal sizeHello everyone!I have question about journal. Ceph cluster is already working with 1gb journal per osd. I wanna to increase size of journal, is this possible?If yes, how i can do this?All the best, Alex!___ceph-users mailing listceph-users@lists.ceph.comhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com___ceph-users mailing listceph-users@lists.ceph.comhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com___ceph-users mailing listceph-users@lists.ceph.comhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Journal size

[Sebastien Han]

Yes I will :-), thank you for pointing out this to me.Sébastien HanCloud Engineer"Always give 100%. Unless you're giving blood."PHONE : +33 (0)1 49 70 99 72 – MOBILE : +33 (0)6 52 84 44 70EMAIL : sebastien@enovance.com – SKYPE : han.sbastienADDRESS : 10, rue de la Victoire – 75009 ParisWEB : www.enovance.com – TWITTER : @enovanceOn Mar 27, 2013, at 5:10 PM, John Nielsen  wrote:Indeed, thanks! Sébastien, care to update your helpful blog post?On Mar 27, 2013, at 10:04 AM, Gregory Farnum  wrote:This should actually be "down", not "out". Out will trigger a data migration. :)-GregSoftware Engineer #42 @ http://inktank.com | http://ceph.comOn Wed, Mar 27, 2013 at 8:36 AM, John Nielsen  wrote:Sébastien Han actually covers this on his blog:http://www.sebastien-han.fr/blog/2012/08/17/ceph-storage-node-maintenance/His use case is moving from an embedded journal to one on an SSD but the steps can be used to move/recreate the journal for any reason:Mark the OSD "out"Stop the OSD daemonRun "ceph-osd -i N --flush-journal", replacing N with the ID of the OSDGet your new journal location ready if it isn'tUpdate ceph.conf with the new journal settingsRun "ceph-osd -i N --mkjournal", again replacing N with the ID of the OSDStart the OSD daemonMark the OSD "in"I used the procedure above to migrate 16 OSD's on 4 hosts from file-based journals to partition-based with no downtime or data loss. (Just be aware of your crush rules and only have one OSD, host, etc. down at a time.)JNOn Mar 27, 2013, at 7:03 AM, Aleksey Samarin  wrote:Well, after few tests, i can't increase size of journal. I think size of journal set only when osd create. So for change journal size u need to delete osd and create with new journal.2013/3/26 Chen, Xiaoxi Are you using a partition as journal?From: ceph-users-boun...@lists.ceph.com [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Aleksey SamarinSent: 2013年3月26日 20:45To: ceph-us...@ceph.comSubject: [ceph-users] Journal sizeHello everyone!I have question about journal. Ceph cluster is already working with 1gb journal per osd. I wanna to increase size of journal, is this possible?If yes, how i can do this?All the best, Alex!___ceph-users mailing listceph-users@lists.ceph.comhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com___ceph-users mailing listceph-users@lists.ceph.comhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Journal size

[John Nielsen]

Indeed, thanks! Sébastien, care to update your helpful blog post?

On Mar 27, 2013, at 10:04 AM, Gregory Farnum  wrote:

> This should actually be "down", not "out". Out will trigger a data migration. 
> :)
> -Greg
> 
> Software Engineer #42 @ http://inktank.com | http://ceph.com
> 
> 
> On Wed, Mar 27, 2013 at 8:36 AM, John Nielsen  wrote:
>> Sébastien Han actually covers this on his blog:
>> 
>> http://www.sebastien-han.fr/blog/2012/08/17/ceph-storage-node-maintenance/
>> 
>> His use case is moving from an embedded journal to one on an SSD but the 
>> steps can be used to move/recreate the journal for any reason:
>> Mark the OSD "out"
>> Stop the OSD daemon
>> Run "ceph-osd -i N --flush-journal", replacing N with the ID of the OSD
>> Get your new journal location ready if it isn't
>> Update ceph.conf with the new journal settings
>> Run "ceph-osd -i N --mkjournal", again replacing N with the ID of the OSD
>> Start the OSD daemon
>> Mark the OSD "in"
>> 
>> I used the procedure above to migrate 16 OSD's on 4 hosts from file-based 
>> journals to partition-based with no downtime or data loss. (Just be aware of 
>> your crush rules and only have one OSD, host, etc. down at a time.)
>> 
>> JN
>> 
>> On Mar 27, 2013, at 7:03 AM, Aleksey Samarin  wrote:
>> 
>>> Well, after few tests, i can't increase size of journal. I think size of 
>>> journal set only when osd create. So for change journal size u need to 
>>> delete osd and create with new journal.
>>> 
>>> 
>>> 2013/3/26 Chen, Xiaoxi 
>>> Are you using a partition as journal?
>>> 
>>> 
>>> 
>>> From: ceph-users-boun...@lists.ceph.com 
>>> [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Aleksey Samarin
>>> Sent: 2013年3月26日 20:45
>>> To: ceph-us...@ceph.com
>>> Subject: [ceph-users] Journal size
>>> 
>>> 
>>> 
>>> Hello everyone!
>>> 
>>> I have question about journal. Ceph cluster is already working with 1gb 
>>> journal per osd. I wanna to increase size of journal, is this possible?
>>> If yes, how i can do this?
>>> 
>>> All the best, Alex!
>>> 
>>> 
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> 
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Journal size

[Gregory Farnum]

This should actually be "down", not "out". Out will trigger a data migration. :)
-Greg

Software Engineer #42 @ http://inktank.com | http://ceph.com


On Wed, Mar 27, 2013 at 8:36 AM, John Nielsen  wrote:
> Sébastien Han actually covers this on his blog:
>
> http://www.sebastien-han.fr/blog/2012/08/17/ceph-storage-node-maintenance/
>
> His use case is moving from an embedded journal to one on an SSD but the 
> steps can be used to move/recreate the journal for any reason:
> Mark the OSD "out"
> Stop the OSD daemon
> Run "ceph-osd -i N --flush-journal", replacing N with the ID of the OSD
> Get your new journal location ready if it isn't
> Update ceph.conf with the new journal settings
> Run "ceph-osd -i N --mkjournal", again replacing N with the ID of the OSD
> Start the OSD daemon
> Mark the OSD "in"
>
> I used the procedure above to migrate 16 OSD's on 4 hosts from file-based 
> journals to partition-based with no downtime or data loss. (Just be aware of 
> your crush rules and only have one OSD, host, etc. down at a time.)
>
> JN
>
> On Mar 27, 2013, at 7:03 AM, Aleksey Samarin  wrote:
>
>> Well, after few tests, i can't increase size of journal. I think size of 
>> journal set only when osd create. So for change journal size u need to 
>> delete osd and create with new journal.
>>
>>
>> 2013/3/26 Chen, Xiaoxi 
>> Are you using a partition as journal?
>>
>>
>>
>> From: ceph-users-boun...@lists.ceph.com 
>> [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Aleksey Samarin
>> Sent: 2013年3月26日 20:45
>> To: ceph-us...@ceph.com
>> Subject: [ceph-users] Journal size
>>
>>
>>
>> Hello everyone!
>>
>> I have question about journal. Ceph cluster is already working with 1gb 
>> journal per osd. I wanna to increase size of journal, is this possible?
>> If yes, how i can do this?
>>
>> All the best, Alex!
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Journal size

[Aleksey Samarin]

Yeah!
Thanks all for your replies!

2013/3/27 Sage Weil 

> On Wed, 27 Mar 2013, Aleksey Samarin wrote:
> > Well, after few tests, i can't increase size of journal. I think size of
> > journal set only when osd create. So for change journal size u need to
> > delete osd and create with new journal.
>
> ceph-osd --flush-journal
> (delete journal file)
> ceph-osd --mkjournal
>
>
>
> >
> >
> > 2013/3/26 Chen, Xiaoxi 
> >
> >   Are you using a partition as journal?
> >
> >
> >
> >   From: ceph-users-boun...@lists.ceph.com
> >   [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Aleksey
> >   Samarin
> >   Sent: 2013?3?26? 20:45
> >   To: ceph-us...@ceph.com
> >   Subject: [ceph-users] Journal size
> >
> >
> >
> >   Hello everyone!
> >
> >   I have question about journal. Ceph cluster is already working
> >   with 1gb journal per osd. I wanna to increase size of journal,
> >   is this possible?
> >   If yes, how i can do this?
> >
> >   All the best, Alex!
> >
> >
> >
> >
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Journal size

[Sage Weil]

On Wed, 27 Mar 2013, Aleksey Samarin wrote:
> Well, after few tests, i can't increase size of journal. I think size of
> journal set only when osd create. So for change journal size u need to
> delete osd and create with new journal.

ceph-osd --flush-journal
(delete journal file)
ceph-osd --mkjournal



> 
> 
> 2013/3/26 Chen, Xiaoxi 
> 
>   Are you using a partition as journal?
> 
>    
> 
>   From: ceph-users-boun...@lists.ceph.com
>   [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Aleksey
>   Samarin
>   Sent: 2013?3?26? 20:45
>   To: ceph-us...@ceph.com
>   Subject: [ceph-users] Journal size
> 
>    
> 
>   Hello everyone!
> 
>   I have question about journal. Ceph cluster is already working
>   with 1gb journal per osd. I wanna to increase size of journal,
>   is this possible?
>   If yes, how i can do this?
> 
>   All the best, Alex!
> 
> 
> 
> ___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Journal size

[John Nielsen]

Sébastien Han actually covers this on his blog:

http://www.sebastien-han.fr/blog/2012/08/17/ceph-storage-node-maintenance/

His use case is moving from an embedded journal to one on an SSD but the steps 
can be used to move/recreate the journal for any reason:
Mark the OSD "out"
Stop the OSD daemon
Run "ceph-osd -i N --flush-journal", replacing N with the ID of the OSD
Get your new journal location ready if it isn't
Update ceph.conf with the new journal settings
Run "ceph-osd -i N --mkjournal", again replacing N with the ID of the OSD
Start the OSD daemon
Mark the OSD "in"

I used the procedure above to migrate 16 OSD's on 4 hosts from file-based 
journals to partition-based with no downtime or data loss. (Just be aware of 
your crush rules and only have one OSD, host, etc. down at a time.)

JN

On Mar 27, 2013, at 7:03 AM, Aleksey Samarin  wrote:

> Well, after few tests, i can't increase size of journal. I think size of 
> journal set only when osd create. So for change journal size u need to delete 
> osd and create with new journal.
> 
> 
> 2013/3/26 Chen, Xiaoxi 
> Are you using a partition as journal?
> 
>  
> 
> From: ceph-users-boun...@lists.ceph.com 
> [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Aleksey Samarin
> Sent: 2013年3月26日 20:45
> To: ceph-us...@ceph.com
> Subject: [ceph-users] Journal size
> 
>  
> 
> Hello everyone!
> 
> I have question about journal. Ceph cluster is already working with 1gb 
> journal per osd. I wanna to increase size of journal, is this possible?
> If yes, how i can do this?
> 
> All the best, Alex!
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Journal size

[Aleksey Samarin]

Well, after few tests, i can't increase size of journal. I think size of
journal set only when osd create. So for change journal size u need to
delete osd and create with new journal.


2013/3/26 Chen, Xiaoxi 

>  Are you using a partition as journal? 
>
> ** **
>
> *From:* ceph-users-boun...@lists.ceph.com [mailto:
> ceph-users-boun...@lists.ceph.com] *On Behalf Of *Aleksey Samarin
> *Sent:* 2013年3月26日 20:45
> *To:* ceph-us...@ceph.com
> *Subject:* [ceph-users] Journal size
>
> ** **
>
> Hello everyone!
>
> I have question about journal. Ceph cluster is already working with 1gb
> journal per osd. I wanna to increase size of journal, is this possible?
> If yes, how i can do this?
>
> All the best, Alex!
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Journal size

[Chen, Xiaoxi]

Are you using a partition as journal?

From: ceph-users-boun...@lists.ceph.com 
[mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Aleksey Samarin
Sent: 2013年3月26日 20:45
To: ceph-us...@ceph.com
Subject: [ceph-users] Journal size

Hello everyone!

I have question about journal. Ceph cluster is already working with 1gb journal 
per osd. I wanna to increase size of journal, is this possible?
If yes, how i can do this?

All the best, Alex!
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com