[ceph-users] is there a Cephfs path length limit

[Götz Reinicke]

Hi,

I was asked, if Cephfs has a path length limit, and if, how long might a path 
get?

We evaluate some software wich might generate very long pathnames (I’m not yet 
told how long it could be).

Thanks for feedback . /Götz

smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Resizing a cache tier rbd

[Götz Reinicke]

Thanks I did resized using the base tier pool and the RBD is back online 
without a problem.

Regarding the depreciation, our initial design and setup was well designed with 
cache tiering for the expected workload. Today (3 - 4 releases later) with 
cephfs and our real world experiences, we would start different, if we start 
today.

For some workload a cache tier would still be amazing. e.g. old file archives 
for RO workload from time to time on a cheep EC pool 12TB SATA drives… Other 
vendors showed us some designs with nice features, even putting really old 
„never“ used data to LTO transparently. (The LTO is the end of the tier.)

But there are so many different workloads, environments, requirements, budgets 
etc, so there is never the one-fits-all.

From my POV ceph offers all kind of features for a lot of different workloads 
and is fast improving, but it is not for everybody the best choice. Check, 
evaluate and talk to a consultant.

Or ask the list :)

BTW thanks to the community for the feedback, help and suggestions. Regards . 
Götz



> Am 27.03.2019 um 00:18 schrieb Jason Dillaman :
> 
> When using cache pools (which are essentially deprecated functionality
> BTW), you should always reference the base tier pool. The fact that a
> cache tier sits in front of a slower, base tier is transparently
> handled.
> 
> On Tue, Mar 26, 2019 at 5:41 PM Götz Reinicke
>  wrote:
>> 
>> Hi,
>> 
>> I have a rbd in a cache tier setup which I need to extend.  The question is, 
>> do I resize it trough the cache pool or directly on the slow/storage pool? 
>> Or dosen t that matter at all?
>> 
>> 
>> Thanks for feedback and regards . Götz



smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Resizing a cache tier rbd

[Götz Reinicke]

Hi,

I have a rbd in a cache tier setup which I need to extend.  The question is, do 
I resize it trough the cache pool or directly on the slow/storage pool? Or 
dosen t that matter at all?


Thanks for feedback and regards . Götz



smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How to change/anable/activate a different osd_memory_target value

[Götz Reinicke]

> Am 20.02.2019 um 09:26 schrieb Konstantin Shalygin :
> 
> 
>> we run into some OSD node freezes with out of memory and eating all swap 
>> too. Till we get more physical RAM I’d like to reduce the osd_memory_target, 
>> but can’t find where and how to enable it.
>> 
>> We have 24 bluestore Disks in 64 GB centos nodes with Luminous v12.2.11
> Just set value for `osd_memory_target` in your ceph.conf and restart your 
> OSD's (`systemctl restart ceph-osd.target` to restart all your osd daemons on 
> this host).


That simple :) Thanks, worked without impact. The cluster is stable now.

Regards . Götz

smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] How to change/anable/activate a different osd_memory_target value

[Götz Reinicke]

Hi,

we run into some OSD node freezes with out of memory and eating all swap too. 
Till we get more physical RAM I’d like to reduce the osd_memory_target, but 
can’t find where and how to enable it.

We have 24 bluestore Disks in 64 GB centos nodes with Luminous v12.2.11

Thanks for hints an suggestions . Regards . Götz



smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Update / upgrade cluster with MDS from 12.2.7 to 12.2.11

[Götz Reinicke]

> Am 12.02.2019 um 00:03 schrieb Patrick Donnelly :
> 
> On Mon, Feb 11, 2019 at 12:10 PM Götz Reinicke
>  wrote:
>> as 12.2.11 is out for some days and no panic mails showed up on the list I 
>> was planing to update too.
>> 
>> I know there are recommended orders in which to update/upgrade the cluster 
>> but I don’t know how rpm packages are handling restarting services after a 
>> yum update. E.g. when MDS and MONs are on the same server.
> 
> This should be fine. The MDS only uses a new executable file if you
> explicitly restart it via systemd (or, the MDS fails and systemd
> restarts it).
> 
> More info: when the MDS respawns in normal circumstances, it passes
> the /proc/self/exe file to execve. An intended side-effect is that the
> MDS will continue using the same executable file across execs.
> 
>> And regarding an MDS Cluster I like to ask, if the upgrading instructions 
>> regarding only running one MDS during upgrading also applies for an update?
>> 
>> http://docs.ceph.com/docs/mimic/cephfs/upgrading/
> 
> If you upgrade an MDS, it may update the compatibility bits in the
> Monitor's MDSMap. Other MDSs will abort when they see this change. The
> upgrade process intended to help you avoid seeing those errors so you
> don't inadvertently think something went wrong.
> 
> If you don't mind seeing those errors and you're using 1 active MDS,
> then don't worry about it.

Thanks for your feedback and clarification!

I have one active MDS and one standby, bot on the same version. So I might see 
some errors during upgrade, but don’t have to stop the standby MDS?!

Or to be save should stop the standby?

Thanks if you can comment on that. Regards . Götz

smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Update / upgrade cluster with MDS from 12.2.7 to 12.2.11

[Götz Reinicke]

Hi,

as 12.2.11 is out for some days and no panic mails showed up on the list I was 
planing to update too.

I know there are recommended orders in which to update/upgrade the cluster but 
I don’t know how rpm packages are handling restarting services after a yum 
update. E.g. when MDS and MONs are on the same server.

And regarding an MDS Cluster I like to ask, if the upgrading instructions 
regarding only running one MDS during upgrading also applies for an update?

http://docs.ceph.com/docs/mimic/cephfs/upgrading/

For me an update is e.g. from a point release, like  12.2.7 to 12.2.11, and an 
upgrade might be from 12.2. to 12.3.

Thanks for feedback . Regards . Götz




smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] One host with 24 OSDs is offline - best way to get it back online

[Götz Reinicke]

Dear all,

thanks for your feedback and Fäll try to take any suggestion in consideration.

I’v rebooted node in question and oll 24 OSDs came online without any 
complaining.

But wat makes me wonder is: During the downtime the Object got rebalanced and 
placed on the remaining nodes.

With the failed node online, only a couple of hundreds objects where misplaced, 
out of about 35 million.

The question for me is: What happens to the objects on the OSDs that went down 
after the OSDs got back online?

Thanks for feedback 


> Am 27.01.2019 um 04:17 schrieb Christian Balzer :
> 
> 
> Hello,
> 
> this is where (depending on your topology) something like:
> ---
> mon_osd_down_out_subtree_limit = host
> ---
> can come in very handy.
> 
> Provided you have correct monitoring, alerting and operations, recovering
> a down node can often be restored long before any recovery would be
> finished and you also avoid the data movement back and forth.
> And if you see that recovering the node will take a long time, just
> manually set things out for the time being.
> 
> Christian
> 
> On Sun, 27 Jan 2019 00:02:54 +0100 Götz Reinicke wrote:
> 
>> Dear Chris,
>> 
>> Thanks for your feedback. The node/OSDs in question are part of an erasure 
>> coded pool and during the weekend the workload should be close to none.
>> 
>> But anyway, I could get a look on the console and on the server; the power 
>> is up, but I cant use any console, the Loginprompt is shown, but no key is 
>> accepted.
>> 
>> I’ll have to reboot the server and check what he is complaining about 
>> tomorrow morning ASAP I can access the server again.
>> 
>>  Fingers crossed and regards. Götz
>> 
>> 
>> 
>>> Am 26.01.2019 um 23:41 schrieb Chris :
>>> 
>>> It sort of depends on your workload/use case.  Recovery operations can be 
>>> computationally expensive.  If your load is light because its the weekend 
>>> you should be able to turn that host back on  as soon as you resolve 
>>> whatever the issue is with minimal impact.  You can also increase the 
>>> priority of the recovery operation to make it go faster if you feel you can 
>>> spare additional IO and it won't affect clients.
>>> 
>>> We do this in our cluster regularly and have yet to see an issue (given 
>>> that we take care to do it during periods of lower client io)
>>> 
>>> On January 26, 2019 17:16:38 Götz Reinicke  
>>> wrote:
>>> 
>>>> Hi,
>>>> 
>>>> one host out of 10 is down for yet unknown reasons. I guess a power 
>>>> failure. I could not yet see the server.
>>>> 
>>>> The Cluster is recovering and remapping fine, but still has some objects 
>>>> to process.
>>>> 
>>>> My question: May I just switch the server back on and in best case, the 24 
>>>> OSDs get back online and recovering will do the job without problems.
>>>> 
>>>> Or what might be a good way to handle that host? Should I first wait till 
>>>> the recover is finished?
>>>> 
>>>> Thanks for feedback and suggestions - Happy Saturday Night  :) . Regards . 
>>>> Götz  
>> 
> 
> 
> -- 
> Christian BalzerNetwork/Systems Engineer
> ch...@gol.com <mailto:ch...@gol.com>  Rakuten Communications

  <http://www.filmakademie.de/>   
Götz Reinicke 
IT-Koordinator
IT-OfficeNet
+49 7141 969 82420 
goetz.reini...@filmakademie.de <mailto:goetz.reini...@filmakademie.de>
Filmakademie Baden-Württemberg GmbH 
Akademiehof 10
71638 Ludwigsburg 
http://www.filmakademie.de <http://www.filmakademie.de/>
   <https://twitter.com/filmakademiebw>
<https://de-de.facebook.com/filmakademiebw> 
<https://www.youtube.com/user/TheFilmakademie>  
<https://vimeo.com/filmakademiebw> 
Eintragung Amtsgericht Stuttgart HRB 205016
Vorsitzende des Aufsichtsrates:
Petra Olschowski
Staatssekretärin im Ministerium für Wissenschaft,
Forschung und Kunst Baden-Württemberg
Geschäftsführer:
Prof. Thomas Schadt

Datenschutzerklärung 
<https://www.filmakademie.de/de/meta/datenschutz/datenschutzerklaerung/> | 
Transparenzinformation 
<https://www.filmakademie.de/de/meta/datenschutz/transparenzinformationen/>
Data privacy statement 
<https://www.filmakademie.de/en/meta/data-privacy/data-privacy-statement/> | 
Transparency information 
<https://www.filmakademie.de/en/meta/data-privacy/transparency-information/>



smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] One host with 24 OSDs is offline - best way to get it back online

[Götz Reinicke]

Dear Chris,

Thanks for your feedback. The node/OSDs in question are part of an erasure 
coded pool and during the weekend the workload should be close to none.

But anyway, I could get a look on the console and on the server; the power is 
up, but I cant use any console, the Loginprompt is shown, but no key is 
accepted.

I’ll have to reboot the server and check what he is complaining about tomorrow 
morning ASAP I can access the server again.

Fingers crossed and regards. Götz



> Am 26.01.2019 um 23:41 schrieb Chris :
> 
> It sort of depends on your workload/use case.  Recovery operations can be 
> computationally expensive.  If your load is light because its the weekend you 
> should be able to turn that host back on  as soon as you resolve whatever the 
> issue is with minimal impact.  You can also increase the priority of the 
> recovery operation to make it go faster if you feel you can spare additional 
> IO and it won't affect clients.
> 
> We do this in our cluster regularly and have yet to see an issue (given that 
> we take care to do it during periods of lower client io)
> 
> On January 26, 2019 17:16:38 Götz Reinicke  
> wrote:
> 
>> Hi,
>> 
>> one host out of 10 is down for yet unknown reasons. I guess a power failure. 
>> I could not yet see the server.
>> 
>> The Cluster is recovering and remapping fine, but still has some objects to 
>> process.
>> 
>> My question: May I just switch the server back on and in best case, the 24 
>> OSDs get back online and recovering will do the job without problems.
>> 
>> Or what might be a good way to handle that host? Should I first wait till 
>> the recover is finished?
>> 
>> Thanks for feedback and suggestions - Happy Saturday Night  :) . Regards . 
>> Götz



smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] One host with 24 OSDs is offline - best way to get it back online

[Götz Reinicke]

Hi,

one host out of 10 is down for yet unknown reasons. I guess a power failure. I 
could not yet see the server.

The Cluster is recovering and remapping fine, but still has some objects to 
process.

My question: May I just switch the server back on and in best case, the 24 OSDs 
get back online and recovering will do the job without problems.

Or what might be a good way to handle that host? Should I first wait till the 
recover is finished?

Thanks for feedback and suggestions - Happy Saturday Night  :) . 
Regards . Götz

smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Rezising an online mounted ext4 on a rbd - failed

[Götz Reinicke]

> Am 26.01.2019 um 14:16 schrieb Kevin Olbrich :
> 
> Am Sa., 26. Jan. 2019 um 13:43 Uhr schrieb Götz Reinicke
> :
>> 
>> Hi,
>> 
>> I have a fileserver which mounted a 4TB rbd, which is ext4 formatted.
>> 
>> I grow that rbd and ext4 starting with an 2TB rbd that way:
>> 
>> rbd resize testpool/disk01--size 4194304
>> 
>> resize2fs /dev/rbd0
>> 
>> Today I wanted to extend that ext4 to 8 TB and did:
>> 
>> rbd resize testpool/disk01--size 8388608
>> 
>> resize2fs /dev/rbd0
>> 
>> => which gives an error: The filesystem is already 1073741824 blocks. 
>> Nothing to do.
>> 
>> 
>>I bet I missed something very simple. Any hint? Thanks and regards . 
>> Götz
> 
> Try "partprobe" to read device metrics again.

Did not change anything and did not give any output/log messages. 

/Götz




smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Rezising an online mounted ext4 on a rbd - failed

[Götz Reinicke]

Hi,

I have a fileserver which mounted a 4TB rbd, which is ext4 formatted.

I grow that rbd and ext4 starting with an 2TB rbd that way:

rbd resize testpool/disk01--size 4194304

resize2fs /dev/rbd0

Today I wanted to extend that ext4 to 8 TB and did:

rbd resize testpool/disk01--size 8388608

resize2fs /dev/rbd0

=> which gives an error: The filesystem is already 1073741824 blocks. Nothing 
to do.


I bet I missed something very simple. Any hint? Thanks and regards . 
Götz



smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Suggestions/experiences with mixed disk sizes and models from 4TB - 14TB

[Götz Reinicke]

Dear Ceph users,

I’d like to get some feedback for the following thought:

Currently I run some 24*4TB bluestore OSD nodes. The main focus is on storage 
space over IOPS.

We use erasure code and cephfs, and things look good right now.

The „but“ is, I do need more disk space and don’t have so much more rack space 
available, so I was thinking of adding some 8TB or even 12TB OSDs and/or 
exchange over time 4TB OSDs with bigger disks.

My question is: How are your experiences with the current >=8TB SATA disks are 
some very bad models out there which I should avoid?

The current OSD nodes are connected by 4*10Gb bonds, so for 
replication/recovery speed is a 24 Chassis with bigger disks useful, or should 
I go with smaller chassis? Or dose the chassi sice does not matter at all that 
much in my setup.

I know, EC is quit computing intense, so may be bigger disks hav also there an 
impact?

Lot’s of questions, may be you can help answering some.

Best regards and Thanks a lot for feedback . Götz




smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] limited disk slots - should I ran OS on SD card ?

[Götz Reinicke]

Hi,

> Am 15.08.2018 um 15:11 schrieb Steven Vacaroaia :
> 
> Thank you all 
> 
> Since all concerns were about reliability I am assuming  performance impact 
> of having OS running on SD card is minimal / negligible 

some time ago we had a some Cisco Blades booting VMware esxi from SD cards and 
hat no issue for month …till after an update the blade was rebooted and the SD 
failed …and then an other on an other server … From my POV at that time the 
„server" SDs where not close as reliable as SSDs or rotating disks. My 
experiences from some years ago.

> 
> In other words, an OSD server is not writing/reading from Linux OS partitions 
> too much ( especially with logs at minimum )
> so its performance is not dependent on what type of disk  OS resides  on 

Regarding performance: What kind of SDs are supported? You can get some "SDXCTM 
| UHS-II | U3 | Class 10 | V90“ which can handle up to 260 MBytes/sec; like 
„Angelbird Matchpack EVA1“ ok they are Panasonic 4K Camera certified (and we 
use them currently to record 4K video)

https://www.angelbird.com/prod/match-pack-for-panasonic-eva1-1836/

My2cents . Götz




smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] OT: Bad Sector Count - suggestions and experiences?

[Götz Reinicke]

Hi,

I apologize for the OT, but I hope some ceph users with bigger installations 
have a lot more experiences than the users reporting theire home problem (NAS 
with 2 disks … ) I saw a lot googling that topic.

Luckily we had not as much hard disk failures as some coworkers, but now with 
more and more disks in use, the topic of Bad Sector Counts gets more importent.

How do you react to Bad Sector Counts from SMART? At what point do you start to 
replace the disk? Any thresholds in percent or absolute numbers?

Currently I got the message from two 8 TB disk; one has 5 the other 24 Bad 
Sectors reported.

Thanks for feedback and suggestions. Regards . Götz

smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Upgrading ceph and mapped rbds

[Götz Reinicke]

> Am 03.04.2018 um 13:31 schrieb Konstantin Shalygin :
> 
>> and true the VMs have to be shut down/server rebooted
> 
> 
> Is not necessary. Just migrate VM.

Hi,

The VMs are XenServer VMs with virtual Disk saved at the NFS Server which has 
the RBD mounted … So there is nor migration from my POV as there is no second 
storage to migrate to ...

Regards . Götz

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Upgrading ceph and mapped rbds

[Götz Reinicke]

Hi Robert,

> Am 29.03.2018 um 10:27 schrieb Robert Sander :
> 
> On 28.03.2018 11:36, Götz Reinicke wrote:
> 
>> My question is: How to proceed with the serves which map the rbds?
> 
> Do you intend to upgrade the kernels on these RBD clients acting as NFS
> servers?
> 
> If so you have to plan a reboot anyway. If not, nothing changes.

Not in that step, but soon; and true the VMs have to be shut down/server 
rebooted, I have that in mind.

Do I understand you correctly, that I don’t have to update the NFS servers ceph 
installation? So the OSD/MONs can run 12.2 and a client can still be on 10.2? 
(I remember a flag might be used to set the cluster compatibility)

Thanks for feedback and regards . Götz

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Upgrading ceph and mapped rbds

[Götz Reinicke]

Hi, I bet I did read it somewhere already, but can’t remember where….

Our ceph 10.2. cluster is fin and healthy and I have a couple of rbds exported 
to some fileserver and a nfs server.

The upgrade to V 12.2 documentation is clear regarding upgrading/restarting all 
MONs first, after that, the OSDs.

My question is: How to proceed with the serves which map the rbds?

Upgrading these after the ceph MONs/OSDs?

May I notice some interruption to the RBDs resp. the SMB and NFS Services? The 
NFS hosts some VMs for Xen …


Thanks for feedback dun comments . Regards . Götz


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Shutting down half / full cluster

[Götz Reinicke]

Thanks!

Götz

> Am 14.02.2018 um 11:16 schrieb Kai Wagner  <mailto:kwag...@suse.com>>:
> 
> Hi,
> 
> maybe it's worth looking at this:
> 
> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-April/017378.html 
> <http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-April/017378.html>
> Kai
> 
> On 02/14/2018 11:06 AM, Götz Reinicke wrote:
>> Hi,
>> 
>> We have some work to do on our power lines for all building and we have to 
>> shut down all systems. So there is also no traffic on any ceph client.
>> 
>> Pitty, we have to shot down some ceph nodes too in an affected building.
>> 
>> To avoid rebalancing - as I see there is no need for it, as there is no 
>> traffic on clients - how can I safely set the remaining cluster nodes in a 
>> „keep calm and wait“ state?
>> 
>> Is that the noout option?
>> 
>> Thanks for feedback and suggestions! Regards . Götz
>> 
>> 
>> 
>> 
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com>
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
>> <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>
> 
> -- 
> SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 
> 21284 (AG Nürnberg)
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com>
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

  <http://www.filmakademie.de/>   
Götz Reinicke 
IT-Koordinator
IT-OfficeNet
+49 7141 969 82420 
goetz.reini...@filmakademie.de <mailto:goetz.reini...@filmakademie.de>
Filmakademie Baden-Württemberg GmbH 
Akademiehof 10
71638 Ludwigsburg 
http://www.filmakademie.de <http://www.filmakademie.de/>
   <https://twitter.com/filmakademiebw>
<https://de-de.facebook.com/filmakademiebw> 
<https://www.youtube.com/user/TheFilmakademie>  
<https://vimeo.com/filmakademiebw> 
Eintragung Amtsgericht Stuttgart HRB 205016
Vorsitzende des Aufsichtsrates:
Petra Olschowski
Staatssekretärin im Ministerium für Wissenschaft,
Forschung und Kunst Baden-Württemberg
Geschäftsführer:
Prof. Thomas Schadt




  <http://www.filmakademie.de/>   
Götz Reinicke 
IT-Koordinator
IT-OfficeNet
+49 7141 969 82420 
goetz.reini...@filmakademie.de <mailto:goetz.reini...@filmakademie.de>
Filmakademie Baden-Württemberg GmbH 
Akademiehof 10
71638 Ludwigsburg 
http://www.filmakademie.de <http://www.filmakademie.de/>
   <https://twitter.com/filmakademiebw>
<https://de-de.facebook.com/filmakademiebw> 
<https://www.youtube.com/user/TheFilmakademie>  
<https://vimeo.com/filmakademiebw> 
Eintragung Amtsgericht Stuttgart HRB 205016
Vorsitzende des Aufsichtsrates:
Petra Olschowski
Staatssekretärin im Ministerium für Wissenschaft,
Forschung und Kunst Baden-Württemberg
Geschäftsführer:
Prof. Thomas Schadt




smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Shutting down half / full cluster

[Götz Reinicke]

Hi,

We have some work to do on our power lines for all building and we have to shut 
down all systems. So there is also no traffic on any ceph client.

Pitty, we have to shot down some ceph nodes too in an affected building.

To avoid rebalancing - as I see there is no need for it, as there is no traffic 
on clients - how can I safely set the remaining cluster nodes in a „keep calm 
and wait“ state?

Is that the noout option?

Thanks for feedback and suggestions! Regards . Götz




smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Suggestion fur naming RBDs

[Götz Reinicke]

Hi, I was wondering what naming scheme you use for naming RBDs in different 
pools. There are no strict rules I know, so what might be a best practice? 
Something like the target service like fileserver_students or webservers_xen, 
webservers_vmware?

A good naming scheme might be helpful :) 

Thanks for feedback and suggestions . Regards . Götz


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Different Ceph versions on OSD/MONs and Clients?

[Götz Reinicke]

Hi,

our OSD and MONs run on jewel and Centos7. Now I was wondering if an older 
fileserver with RHEL 6 for which I just found hammer RPMs on the official ceph 
site can use the RDBs creation on the cluster?

I think there might be problems with some kernel versions/ceph features. 

If it is not possible in that combination I have, nevertheless is it possible 
to run different versions?

Thanks for hints & suggestions . Regards Götz




smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Updating ceps client - what will happen to services like NFS on clients

[Götz Reinicke]

Hi Thanks David & David,

we don’t use the fuse code. And may be I was a bit unclear, but your feedback 
clears some other aspects in that context.

I did an update already on our OSD/MONs while a NFS Fileserver still had a rbd 
connected and was exporting files (Virtual disks for XEN server) online.

Now the question is, can I update the ceph packages on the NFS Fileserver while 
the export (and VM Images) are online? Or should I shutdown the VMs and unmount 
the NFS export …?

Lots of nuts and bolts, but I like to screw …. :)

Cheers . Götz


> Am 25.09.2017 um 17:05 schrieb David Turner :
> 
> It depends a bit on how you have the RBDs mapped.  If you're mapping them 
> using krbd, then they don't need to be updated to use the new rbd-fuse or 
> rbd-nbd code.  If you're using one of the latter, then you should schedule a 
> time to restart the mounts so that they're mapped with the new Ceph version.
> 
> In general RBDs are not affected by upgrades as long as you don't take down 
> too much of the cluster at once and are properly doing a rolling upgrade.
> 
> On Mon, Sep 25, 2017 at 8:07 AM David  <mailto:dclistsli...@gmail.com>> wrote:
> Hi Götz 
> 
> If you did a rolling upgrade, RBD clients shouldn't have experienced 
> interrupted IO and therefor IO to NFS exports shouldn't have been affected. 
> However, in the past when using kernel NFS over kernel RBD, I did have some 
> lockups when OSDs went down in the cluster so that's something to watch out 
> for.
> 
> 
> On Mon, Sep 25, 2017 at 8:38 AM, Götz Reinicke 
> mailto:goetz.reini...@filmakademie.de>> 
> wrote:
> Hi,
> 
> I updated our ceph OSD/MON Nodes from 10.2.7 to 10.2.9 and everything looks 
> good so far.
> 
> Now I was wondering (as I may have forgotten how this works) what will happen 
> to a  NFS server which has the nfs shares on a ceph rbd ? Will the update 
> interrupt any access to the NFS share or is it that smooth that e.g. clients 
> accessing the NFS share will not notice?
> 
> Thanks for some lecture on managing ceph and regards . Götz

<…>



smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Updating ceps client - what will happen to services like NFS on clients

[Götz Reinicke]

Hi,

I updated our ceph OSD/MON Nodes from 10.2.7 to 10.2.9 and everything looks 
good so far.

Now I was wondering (as I may have forgotten how this works) what will happen 
to a  NFS server which has the nfs shares on a ceph rbd ? Will the update 
interrupt any access to the NFS share or is it that smooth that e.g. clients 
accessing the NFS share will not notice?

Thanks for some lecture on managing ceph and regards . Götz

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Some OSDs are down after Server reboot

[Götz Reinicke]

Hi,

maybe someone has a hint: I do have a cephalopod cluster (6 nodes, 144 OSDs), 
Cents 7.3 ceph 10.2.7.

I did a kernel update to the recent centos 7.3 one on a node and did a reboot.

After that, 10 OSDs did not came up as the others. The disk did not get mounted 
and the OSD processes did nothing … even after a couple of minutes no more 
disks/OSDs showed up.

So I did a ceph-disk activate-all.

And all missing OSDs got back online.

Questions: Any hints on debugging why the disk did not get online after the 
reboot?

Thanks an regards . Götz


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] where is a RBD in use

[Götz Reinicke]

Hi,

Is it possible to see which clients are using an RBD? … I found an RBD in one 
of my pools but cant remember if I ever use / mounted it to a client.

Thx for feedback ! Regards . Götz


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] XFS attempt to access beyond end of device

[Götz Reinicke - IT Koordinator]

Hi,

Am 28.07.17 um 04:06 schrieb Brad Hubbard:
> An update on this.
>
> The "attempt to access beyond end of device" messages are created due to a
> kernel bug which is rectified by the following patches.
>
>   - 59d43914ed7b9625(vfs: make guard_bh_eod() more generic)
>   - 4db96b71e3caea(vfs: guard end of device for mpage interface)
>
> An upgraded Red Hat kernel including these patches is pending.
>
> There was also discussion of the following upstream tracker
> http://tracker.ceph.com/issues/14842 however that has been eliminated as being
> in play for any of the devices analysed whilst investigating this issue since
> these partitions are correctly aligned.
<...>

any news on the fixed kernel release date/version? (May be I missed it yet)

    Thanks . Götz



smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Installing ceph on Centos 7.3

[Götz Reinicke - IT Koordinator]

Hi,

Am 18.07.17 um 10:51 schrieb Brian Wallis:
> I’m failing to get an install of ceph to work on a new Centos 7.3.1611
> server. I’m following the instructions
> at http://docs.ceph.com/docs/master/start/quick-ceph-deploy/ to no
> avail. 
>
> First question, is it possible to install ceph on Centos 7.3 or should
> I choose a different version or different linux distribution to use
> for now?
<...>

we run CEPH /Jewel /10.2.7 on RHEL 7.3. It is working.

Maybe an other guide might help you trough the installation steps?

https://www.virtualtothecore.com/en/quickly-build-a-new-ceph-cluster-with-ceph-deploy-on-centos-7/


Regards . Götz


smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Degraded Cluster, some OSDs dont get mounted, dmesg confusion

[Götz Reinicke - IT Koordinator]

Hi,

we have a 144 OSD 6 node ceph cluster with some pools (2 repl and EC).

Today I did an CEPH (10.2.5 -> 10.2.7) and kernel update and rebooted
two nodes.

on both nodes some OSDs dont get mountet and on one node I get some
dmesg like:

attempt to access beyond end of device

Currently the Cluster is replacing and recovering, but I do have a
starnge feeling :)

a) "where" are my missing OSDs (fdisk -l lists them)

b) how to recover or remount or bring the missing OSDs back into the
cluster?

c) what about the message "attempt to access beyond end of device"


Thanks for any hints and suggestions!

Regards . Götz



smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] At what point are objects removed?

[Götz Reinicke - IT Koordinator]

Hi, may be I got something wrong or did not understend it yet in total.

I have some pools and created some test rbd images which are mounted to 
a samba server.


After the test I deleted all files from the on the samba server.

But "ceph df detail" and "ceph -s" show still used space.

The OSDs on the ceph osd nodes are also filled with data.

My question: At what point will the still existing but not needed/used 
objects be removed?



Thanks for feedback and suggestions and Kowtow . Götz





smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Jewel v10.2.6 released

[Götz Reinicke - IT Koordinator]

Hi,

Am 08.03.17 um 13:11 schrieb Abhishek L:

This point release fixes several important bugs in RBD mirroring, RGW 
multi-site, CephFS, and RADOS.

We recommend that all v10.2.x users upgrade.

For more detailed information, see the complete changelog[1] and the release 
notes[2]

I hope you can give me some quick advise on updating from 10.2.5 -> 
10.2.6 a running ceph cluster. Can I just do an yum update and I'm good?


Thanks for making that clear and regards . Götz




smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] To backup or not to backup the classic way - How to backup hundreds of TB?

[Götz Reinicke]

Hi,

I guess that's a question that pops up in different places, but I could not 
find any which fits to my thoughts.

Currently we start to use ceph for file shares of our films produced by our 
students and some xen/vmware VMs. Thd VM data is already backed up; the fils 
original footage is stored in other places.

We start with some 100TB rbd and mount smb/NFS shares from the clients. May be 
we look into ceph fs soon.

The question is: How would someone handle a backup of 100 TB data? Rsyncing 
that to an other system or having a commercial backup solution looks not that 
good e.g. regarding the price.

One thought is, is there some sort of best practice in the ceph world e.g. 
replicating to an other physical independent cluster? Or use more replicas, 
odds, nodes and do snapshots in one cluster?

Having productive data and backup on the same hardware currently makes me feel 
not that good too….But the world changes :)

Long story short: How do you do backup hundreds of TB?

Curious for suggestions and thoughts .. Thanks and Regards . Götz


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph -s require_jewel_osds pops up and disappears

[Götz Reinicke]

Hi,

Ceph -s shows like a direction indicator on of  require_jewel_osds.

I did recently an upgrade from centos 7.2 to 7.3 and ceph 10.2.3 to 10.2.5.

May be I forgot to set an option?

I thought I did a „ ceph osd set require_jewel_osds“ as described in the 
release notes https://ceph.com/geen-categorie/v10-2-4-jewel-released/

Thanks for hints what to checke and maybe switch on and off again.

Regards . Götz
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] suggestions on / how to update OS and Ceph in general

[Götz Reinicke - IT Koordinator]

Hi,

we have a 6 node ceph 10.2.3 cluster on centos 7.2 servers, currently no
hosting any rbds or anything else. MONs are on the OSD nodes.

My question is as centos 7.3 is out now for some time and there is a
ceph update to 10.2.5 available, what would be a good or the best path
to update everything?

First the OS or first ceph? What steps are recommended in the future
when the cluster is in productive use? For the kernel update the nodes
have to be rebooted; what happens during that downtime of a node to the
cluster?

I read the
http://docs.ceph.com/docs/jewel/install/upgrading-ceph/#upgrade-procedures
and it looks quiet simple ... doing the "yum update" and "ceph restart ..."

Thanks for feedback and suggestions and happy 2017 . Götz





smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] where is what in use ...

[Götz Reinicke - IT Koordinator]

Hi,

I started to play with our Ceph cluster and created some pools and rdbs
and did some performance test. Currently I'm up to understand and
interpret the different outputs of ceph -s or rados df etc.

So far so good so nice.

Now I was cleaning up (rbd rm ... ) and still see some space used on the
pools (rados df)

Can someone give me a short explanation, why do I still see space used?


Thx and regards . Götz




smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rbd pool:replica size choose: 2 vs 3

[Götz Reinicke - IT Koordinator]

Hi,

Am 23.09.16 um 05:55 schrieb Zhongyan Gu:
> Hi there,
> the default rbd pool replica size is 3. However, I found that in our
> all ssd environment, capacity become a cost issue. We want to save
> more capacity. So one option is change the replica size from 3 to 2.
> anyone can share the experience of pros vs cons regarding replica size
> 2 vs 3?
from my (still limited) POV, one main aspect is: how reliabel is your
hardware if you think off this? How often will a disk break, a server
crash, a datacenter burn down, a networkswitch fail? And if there is a
failure, how fast could that broken part be replaced or how fast is your
availabel hardware to replicate the lost OSD to the remaining system.

I dont have numbers, but for our first initial cluster we go as well
with a repl size of 2 and I dont have bad feelings yet when i look at
the server and network infrastrukture we got.

Others with more experiacne will give some other hints and may be
numbers. I never found some sort of calculator which can say "Oh you get
this hardware? Than a repl size of x y z is what you need."
 
HTH a bit . Regards . Götz




smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] thoughts about Cache Tier Levels

[Götz Reinicke - IT Koordinator]

Hi,

currently there are two levels I know of: storage- and cachepool. From
our workload I do expect an third "level" of data, which will stay
currently in the storagepool as well.

Has anyone as we been thinking of data which could be moved even deeper
in that tiering, e.g. have SSD cache, fast lots of HDD SAS something
2-4TB size for storage pool and e.g. 8TB so whatever for an "archive" level?

From my pov that would be for data which are good and nice to have in
the cluster, but dont have to be as reliable and fast as the storage
tier data.

Or may be that kind of data could be moved to a LTO tape library ...
just thinking ! :)

Thanks for feedback and suggestion on how to handle data you "never will
use again but you have to have them" :)

Regards . Götz






smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] 40Gb fileserver/NIC suggestions

[Götz Reinicke - IT Koordinator]

Am 13.07.16 um 17:08 schrieb c...@jack.fr.eu.org:
> I am using these for other stuff:
> http://www.supermicro.com/products/accessories/addon/AOC-STG-b4S.cfm
>
> If you want NIC, also think of the "network side" : SFP+ switch are very
> common, 40G is less common, 25G is really new (= really few products)
The networkside will be 10G for the OSD nodes, and 40Gb for the core layer.

Regards . Götz



smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] 40Gb fileserver/NIC suggestions

[Götz Reinicke - IT Koordinator]

Am 13.07.16 um 17:44 schrieb David:
> Aside from the 10GbE vs 40GbE question, if you're planning to export
> an RBD image over smb/nfs I think you are going to struggle to reach
> anywhere near 1GB/s in a single threaded read. This is because even
> with readahead cranked right up you're still only going be hitting a
> handful of disks at a time. There's a few threads on this list about
> sequential reads with the kernel rbd client. I think CephFS would be
> more appropriate in your use case.
Thanks for that hint; as soon as our nodes are online, we will do some
testing!

Regards . Götz




smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] 40Gb fileserver/NIC suggestions

[Götz Reinicke - IT Koordinator]

Am 13.07.16 um 14:59 schrieb Joe Landman:
>
>
> On 07/13/2016 08:41 AM, c...@jack.fr.eu.org wrote:
>> 40Gbps can be used as 4*10Gbps
>>
>> I guess welcome feedbacks should not be stuck by "usage of a 40Gbps
>> ports", but extented to "usage of more than a single 10Gbps port, eg
>> 20Gbps etc too"
>>
>> Is there people here that are using more than 10G on an ceph server ?
>
> We have built, and are building Ceph units for some of our customers
> with dual 100Gb links.  The storage box was one of our all flash
> Unison units for OSDs.  Similarly, we have several customers actively
> using multiple 40GbE links on our 60 bay Unison spinning rust disk
> (SRD) box.
>
Now we get closer. Can you tell me which 40G Nic you use?

/götz



smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] 40Gb fileserver/NIC suggestions

[Götz Reinicke - IT Koordinator]

Am 13.07.16 um 14:27 schrieb Wido den Hollander:
>> Op 13 juli 2016 om 12:00 schreef Götz Reinicke - IT Koordinator 
>> :
>>
>>
>> Am 13.07.16 um 11:47 schrieb Wido den Hollander:
>>>> Op 13 juli 2016 om 8:19 schreef Götz Reinicke - IT Koordinator 
>>>> :
>>>>
>>>>
>>>> Hi,
>>>>
>>>> can anybody give some realworld feedback on what hardware
>>>> (CPU/Cores/NIC) you use for a 40Gb (file)server (smb and nfs)? The Ceph
>>>> Cluster will be mostly rbd images. S3 in the future, CephFS we will see :)
>>>>
>>>> Thanks for some feedback and hints! Regadrs . Götz
>>>>
>>> Why do you think you need 40Gb? That's some serious traffic to the OSDs and 
>>> I doubt it's really needed.
>>>
>>> Latency-wise 40Gb isn't much better than 10Gb, so why not stick with that?
>>>
>>> It's also better to have more smaller nodes than a few big nodes with Ceph.
>>>
>>> Wido
>>>
>> Hi Wido,
>>
>> may be my post was misleading. The OSD Nodes do have 10G, the FIleserver
>> in front to the Clients/Destops should have 40G.
>>
> Ah, the fileserver will re-export RBD/Samba? Any Xeon E5 CPU will do just 
> fine I think.
True @re-export
> Still, 40GbE is a lot of bandwidth!
:) I know, but we have users which like to transfer e.g. raw movie
footage for a normal project which might be quick at 1TB and they dont
want to wait hours ;). Or others like to screen/stream 4K Video footage
raw which is +- 10Gb/second ... Thats the challenge :)

And yes our Ceph Cluster is well designed .. on the paper ;) SSDs
considered. With lot of helpful feedback from the List!!

I just try to find linux/ceph useres with 40Gb experiences :)

cheers . Götz



smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] 40Gb fileserver/NIC suggestions

[Götz Reinicke - IT Koordinator]

Am 13.07.16 um 11:47 schrieb Wido den Hollander:
>> Op 13 juli 2016 om 8:19 schreef Götz Reinicke - IT Koordinator 
>> :
>>
>>
>> Hi,
>>
>> can anybody give some realworld feedback on what hardware
>> (CPU/Cores/NIC) you use for a 40Gb (file)server (smb and nfs)? The Ceph
>> Cluster will be mostly rbd images. S3 in the future, CephFS we will see :)
>>
>> Thanks for some feedback and hints! Regadrs . Götz
>>
> Why do you think you need 40Gb? That's some serious traffic to the OSDs and I 
> doubt it's really needed.
>
> Latency-wise 40Gb isn't much better than 10Gb, so why not stick with that?
>
> It's also better to have more smaller nodes than a few big nodes with Ceph.
>
> Wido
>
Hi Wido,

may be my post was misleading. The OSD Nodes do have 10G, the FIleserver
in front to the Clients/Destops should have 40G.


OSD NODEs/Cluster 2*10Gb Bond  40G Fileserver 40G  1G/10G Clients

/Götz



smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] 40Gb fileserver/NIC suggestions

[Götz Reinicke - IT Koordinator]

Hi,

can anybody give some realworld feedback on what hardware
(CPU/Cores/NIC) you use for a 40Gb (file)server (smb and nfs)? The Ceph
Cluster will be mostly rbd images. S3 in the future, CephFS we will see :)

Thanks for some feedback and hints! Regadrs . Götz




smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Quick short survey which SSDs

[Götz Reinicke - IT Koordinator]

Hi,

we have offers for ceph storage nodes with different SSD types and some
are already mentioned as a very good choice but some are total new to me.

May be you could give some feedback on the SSDs in question or provide
just small information which you primarily us?

Regarding the three disk in questions it is not a question of the price
(good conditions :) ). More the question of reliability and real world
experiences.

We plan to use 48*800 SSDs, so this is not just for fun.


Well known SSD: Intel DC S3710

Unfamiliar: Samsung SM863

Unfamiliar: Seagate ST800FM0043 (amazing fact sheet, very good conditions!)


Thanks a lot for your feedback and Regards . Götz





smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] recommendations for file sharing

[Götz Reinicke]

Hi,

> Am 17.12.2015 um 09:43 schrieb Alex Leake  >:
> 
> ​Lin,
> 
> Thanks for this! I did not see the ownCloud RADOS implementation.
> 
> I maintain a local ownCloud environment anyway, so this is a really good idea.
> 
> Have you used it?

<…>

just a quick google look up:

https://doc.owncloud.org/server/8.2/admin_manual/enterprise_external_storage/s3_swift_as_primary_object_store_configuration.html
 


BTW: We plan also to use

a) a HA Samba Filer head and b) if it is time to use own cloud with cep 
connected….

Still we don’t have our server/OSD hardware … 

Regards . Götz



smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] K is for Kraken

[Götz Reinicke - IT Koordinator]

Am 08.02.16 um 20:09 schrieb Robert LeBlanc:
> Too bad K isn't an LTS. It was be fun to release the Kraken many times.

+1

:) https://www.youtube.com/watch?v=_lN2auTVavw

cheers . Götz


-- 
Götz Reinicke
IT-Koordinator

Tel. +49 7141 969 82420
E-Mail goetz.reini...@filmakademie.de

Filmakademie Baden-Württemberg GmbH
Akademiehof 10
71638 Ludwigsburg
www.filmakademie.de

Eintragung Amtsgericht Stuttgart HRB 205016

Vorsitzender des Aufsichtsrats: Jürgen Walter MdL
Staatssekretär im Ministerium für Wissenschaft,
Forschung und Kunst Baden-Württemberg

Geschäftsführer: Prof. Thomas Schadt



smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] SSD OSDs - more Cores or more GHz

[Götz Reinicke - IT Koordinator]

Am 20.01.16 um 11:30 schrieb Christian Balzer:
> 
> Hello,
> 
> On Wed, 20 Jan 2016 10:01:19 +0100 Götz Reinicke - IT Koordinator wrote:
> 
>> Hi folks,
>>
>> we plan to use more ssd OSDs in our first cluster layout instead of SAS
>> osds. (more IO is needed than space)
>>
>> short question: What would influence the performance more? more Cores or
>> more GHz/Core.
>>
>> Or is it as always: Depeds on the total of
>> OSDs/nodes/repl-level/etc ... :)
>>
> 
> While there certainly is a "depends" in there, my feeling is that faster
> cores are more helpful than many, slower ones.
> And this is how I spec'ed my first SSD nodes, 1 fast core (Intel, thus 2
> pseudo-cores) per OSD.
> The reasoning is simple, an individual OSD thread will run (hopefully) on
> one core and thus be faster, with less latency(!).
> 
>> If needed, I can give some more detailed information on the layout.
>>
> Might be interesting for other sanity checks, if you don't mind.

With pleasure. The basic setup was: 6*24 SATA Nodes, simple
"replication" (size = 2), two datacenters, 20Gbit bonded LAN.

We calculated with 2x Intel Xeon E5-2630v3 8-Core 2,4GHz per OSD Node =
32 vCPUs each. (hypertreaded)

On the other hand, if we talk about one core per osd two E5-2620v3
6-Core would do per node.

But regarding the total cost we are faced with (upgrading network, more
fibers etc) ... the cost for the more cores do not hurt. And having some
cores "as spare" sounded good as we would install the mons on the OSD
node for now.

Later we discussed the design with an external consultant and added the
new and "OMG I forgot to tell you that we need" requirements and ended
up with:

three pool storage classes: fast and big more small IO, fast and not so
big (VM images) and cachetiered ec-pool bg videofiles.

Using the 6-Node 2 datacenter layout we ended up with a nice OSD
spreading and layout which sounds good to me.

The most important "new" thing was: using SSD for the pools, not just
for journal and OS.

So I ended up with the question, if the CPUs would have any good or bad
influence in that new SSDish-design.

Thanks for all the feedback and thoughts/comments so far.


CONCLUSION so far: From my POV the 2630v3 is good for us.

Cheers . Götz

-- 
Götz Reinicke
IT-Koordinator

Tel. +49 7141 969 82420
E-Mail goetz.reini...@filmakademie.de

Filmakademie Baden-Württemberg GmbH
Akademiehof 10
71638 Ludwigsburg
www.filmakademie.de

Eintragung Amtsgericht Stuttgart HRB 205016

Vorsitzender des Aufsichtsrats: Jürgen Walter MdL
Staatssekretär im Ministerium für Wissenschaft,
Forschung und Kunst Baden-Württemberg

Geschäftsführer: Prof. Thomas Schadt




smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] SSD OSDs - more Cores or more GHz

[Götz Reinicke - IT Koordinator]

Hi folks,

we plan to use more ssd OSDs in our first cluster layout instead of SAS
osds. (more IO is needed than space)

short question: What would influence the performance more? more Cores or
more GHz/Core.

Or is it as always: Depeds on the total of OSDs/nodes/repl-level/etc ... :)

If needed, I can give some more detailed information on the layout.

Thansk for feedback . Götz
-- 
Götz Reinicke
IT-Koordinator

Tel. +49 7141 969 82420
E-Mail goetz.reini...@filmakademie.de

Filmakademie Baden-Württemberg GmbH
Akademiehof 10
71638 Ludwigsburg
www.filmakademie.de

Eintragung Amtsgericht Stuttgart HRB 205016

Vorsitzender des Aufsichtsrats: Jürgen Walter MdL
Staatssekretär im Ministerium für Wissenschaft,
Forschung und Kunst Baden-Württemberg

Geschäftsführer: Prof. Thomas Schadt



smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] network failover with public/custer network - is that possible

[Götz Reinicke - IT Koordinator]

Hi,

discussing some design questions we came across the failover possibility
of cephs network configuration.

If I just have a public network, all traffic is crossing that lan.

With public and cluster network I can separate the traffic and get some
benefits.

What if one of the networks fail? e.g. just on one host or the whole
network for all nodes?

Is there some sort of auto failover to use the other network for all
traffic than?

How dose that work in real life? :) Or do I have to interact by hand?

Thanks for feedback and regards . Götz




smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] which SSD / experiences with Samsung 843T vs. Intel s3700

[Götz Reinicke - IT Koordinator]

Hi,

most of the times I do get the recommendation from resellers to go with
the intel s3700 for the journalling.

Now I got an offer for systems with MLC 240 GB SATA Samsung 843T.

A quick research on google shows that that ssd is not as good as the
intel, but good, server grade 24/7 etc. and not that good in iops and
performance.

The samsung cost on the other hand nearly half in that offers ...

Dose anybody can give some feedback on the SSDs you use and in special
regarding the two I mention?

Thanks and regards. Götz




smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Real world benefit from SSD Journals for a more read than write cluster

[Götz Reinicke]

Hi Warren,

thanks for that feedback. regarding the 2 or 3 copies we had a lot of internal 
discussions and lots of pros and cons on 2 and 3 :) … and finally decided to 
give 2 copies in the first - now called evaluation cluster - a chance to prove.

I bet in 2016 we will see, if that was a good decision or bad and data los is 
in that scenario ok. We evaluate. :)

Regarding one P3700 for 12 SATA disks I do get it right, that if that P3700 
fails all 12 OSDs are lost… ? So that looks like a bigger risk to me from my 
current knowledge. Or are the P3700 so much more reliable than the eg. S3500 or 
S3700?

Or is the suggestion with the P3700 if we go in the direction of 20+ nodes and 
till than stay without SSDs for journaling.

I really appreciate your thoughts and feedback and I’m aware of the fact that 
building a ceph cluster is some sort of knowing the specs, configuration 
option, math, experience, modification and feedback from best practices real 
world clusters. Finally all clusters are unique in some way and what works for 
one will not work for an other.

Thanks for feedback, 100 kowtows . Götz


 
> Am 09.07.2015 um 16:58 schrieb Wang, Warren :
> 
> You'll take a noticeable hit on write latency. Whether or not it's tolerable 
> will be up to you and the workload you have to capture. Large file operations 
> are throughput efficient without an SSD journal, as long as you have enough 
> spindles.
> 
> About the Intel P3700, you will only need 1 to keep up with 12 SATA drives. 
> The 400 GB is probably okay if you keep the journal sizes small, but the 800 
> is probably safer if you plan on leaving these in production for a few years. 
> Depends on the turnover of data on the servers.
> 
> The dual disk failure comment is pointing out that you are more exposed for 
> data loss with 2 copies. You do need to understand that there is a 
> possibility for 2 drives to fail either simultaneously, or one before the 
> cluster is repaired. As usual, this is going to be a decision you need to 
> decide if it's acceptable or not. We have many clusters, and some are 2, and 
> others are 3. If your data resides nowhere else, then 3 copies is the safe 
> thing to do. That's getting harder and harder to justify though, when the 
> price of other storage solutions using erasure coding continues to plummet.
> 
> Warren
> 
> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Götz 
> Reinicke - IT Koordinator
> Sent: Thursday, July 09, 2015 4:47 AM
> To: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] Real world benefit from SSD Journals for a more 
> read than write cluster
> 
> Hi Christian,
> Am 09.07.15 um 09:36 schrieb Christian Balzer:
>> 
>> Hello,
>> 
>> On Thu, 09 Jul 2015 08:57:27 +0200 Götz Reinicke - IT Koordinator wrote:
>> 
>>> Hi again,
>>> 
>>> time is passing, so is my budget :-/ and I have to recheck the 
>>> options for a "starter" cluster. An expansion next year for may be an 
>>> openstack installation or more performance if the demands rise is 
>>> possible. The "starter" could always be used as test or slow dark archive.
>>> 
>>> At the beginning I was at 16SATA OSDs with 4 SSDs for journal per 
>>> node, but now I'm looking for 12 SATA OSDs without SSD journal. Less 
>>> performance, less capacity I know. But thats ok!
>>> 
>> Leave the space to upgrade these nodes with SSDs in the future.
>> If your cluster grows large enough (more than 20 nodes) even a single
>> P3700 might do the trick and will need only a PCIe slot.
> 
> If I get you right, the 12Disk is not a bad idea, if there would be the need 
> of SSD Journal I can add the PCIe P3700.
> 
> In the 12 OSD Setup I should get 2 P3700 one per 6 OSDs.
> 
> God or bad idea?
> 
>> 
>>> There should be 6 may be with the 12 OSDs 8 Nodes with a repl. of 2.
>>> 
>> Danger, Will Robinson.
>> This is essentially a RAID5 and you're plain asking for a double disk 
>> failure to happen.
> 
> May be I do not understand that. size = 2 I think is more sort of raid1 ... ? 
> And why am I asking for for a double disk failure?
> 
> To less nodes, OSDs or because of the size = 2.
> 
>> 
>> See this recent thread:
>> "calculating maximum number of disk and node failure that can be 
>> handled by cluster with out data loss"
>> for some discussion and python script which you will need to modify 
>> for
>> 2 disk replication.
>> 
>> With a RAID5 failure calculator you're at 1 data loss event per 3.5 
>> years...
>> 
> 
> Thanks for that thre

Re: [ceph-users] Real world benefit from SSD Journals for a more read than write cluster

[Götz Reinicke - IT Koordinator]

Hi Christian,
Am 09.07.15 um 09:36 schrieb Christian Balzer:
> 
> Hello,
> 
> On Thu, 09 Jul 2015 08:57:27 +0200 Götz Reinicke - IT Koordinator wrote:
> 
>> Hi again,
>>
>> time is passing, so is my budget :-/ and I have to recheck the options
>> for a "starter" cluster. An expansion next year for may be an openstack
>> installation or more performance if the demands rise is possible. The
>> "starter" could always be used as test or slow dark archive.
>>
>> At the beginning I was at 16SATA OSDs with 4 SSDs for journal per node,
>> but now I'm looking for 12 SATA OSDs without SSD journal. Less
>> performance, less capacity I know. But thats ok!
>>
> Leave the space to upgrade these nodes with SSDs in the future.
> If your cluster grows large enough (more than 20 nodes) even a single
> P3700 might do the trick and will need only a PCIe slot.

If I get you right, the 12Disk is not a bad idea, if there would be the
need of SSD Journal I can add the PCIe P3700.

In the 12 OSD Setup I should get 2 P3700 one per 6 OSDs.

God or bad idea?

> 
>> There should be 6 may be with the 12 OSDs 8 Nodes with a repl. of 2.
>>
> Danger, Will Robinson.
> This is essentially a RAID5 and you're plain asking for a double disk
> failure to happen.

May be I do not understand that. size = 2 I think is more sort of raid1
... ? And why am I asking for for a double disk failure?

To less nodes, OSDs or because of the size = 2.

> 
> See this recent thread:
> "calculating maximum number of disk and node failure that can be handled
> by cluster with out data loss"
> for some discussion and python script which you will need to modify for
> 2 disk replication.
> 
> With a RAID5 failure calculator you're at 1 data loss event per 3.5
> years...
> 

Thanks for that thread, but I dont get the point out of it for me.

I see that calculating the reliability is some sort of complex math ...

>> The workload I expect is more writes of may be some GB of Office files
>> per day and some TB of larger video Files from a few users per week.
>>
>> At the end of this year we calculate to have +- 60 to 80 TB of lager
>> videofiles in that cluster, which are accessed from time to time.
>>
>> Any suggestion on the drop of ssd journals?
>>
> You will miss them when the cluster does write, be it from clients or when
> re-balancing a lost OSD.

I can imagine, that I might miss the SSD Journal, but if I can add the
P3700 later I feel comfy with it for now. Budget and evaluation related.

Thanks for your helpful input and feedback. /Götz

-- 
Götz Reinicke
IT-Koordinator

Tel. +49 7141 969 82420
E-Mail goetz.reini...@filmakademie.de

Filmakademie Baden-Württemberg GmbH
Akademiehof 10
71638 Ludwigsburg
www.filmakademie.de

Eintragung Amtsgericht Stuttgart HRB 205016

Vorsitzender des Aufsichtsrats: Jürgen Walter MdL
Staatssekretär im Ministerium für Wissenschaft,
Forschung und Kunst Baden-Württemberg

Geschäftsführer: Prof. Thomas Schadt




smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Real world benefit from SSD Journals for a more read than write cluster

[Götz Reinicke - IT Koordinator]

Hi again,

time is passing, so is my budget :-/ and I have to recheck the options
for a "starter" cluster. An expansion next year for may be an openstack
installation or more performance if the demands rise is possible. The
"starter" could always be used as test or slow dark archive.

At the beginning I was at 16SATA OSDs with 4 SSDs for journal per node,
but now I'm looking for 12 SATA OSDs without SSD journal. Less
performance, less capacity I know. But thats ok!

There should be 6 may be with the 12 OSDs 8 Nodes with a repl. of 2.

The workload I expect is more writes of may be some GB of Office files
per day and some TB of larger video Files from a few users per week.

At the end of this year we calculate to have +- 60 to 80 TB of lager
videofiles in that cluster, which are accessed from time to time.

Any suggestion on the drop of ssd journals?

Thanks as always for your feedback . Götz

-- 
Götz Reinicke
IT-Koordinator

Tel. +49 7141 969 82420
E-Mail goetz.reini...@filmakademie.de

Filmakademie Baden-Württemberg GmbH
Akademiehof 10
71638 Ludwigsburg
www.filmakademie.de

Eintragung Amtsgericht Stuttgart HRB 205016

Vorsitzender des Aufsichtsrats: Jürgen Walter MdL
Staatssekretär im Ministerium für Wissenschaft,
Forschung und Kunst Baden-Württemberg

Geschäftsführer: Prof. Thomas Schadt



smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cisco UCS Blades as MONs? Pros cons ...?

[Götz Reinicke - IT Koordinator]

Hi Jake,

we have the fabric interconnects.

MONs as VM? What setup do you have? and what cluster size?

Regards . Götz


Am 13.05.15 um 15:20 schrieb Jake Young:
> I run my mons as VMs inside of UCS blade compute nodes. 
> 
> Do you use the fabric interconnects or the standalone blade chassis?
> 
> Jake
> 
> On Wednesday, May 13, 2015, Götz Reinicke - IT Koordinator
> mailto:goetz.reini...@filmakademie.de>>
> wrote:
> 
> Hi Christian,
> 
> currently we do get good discounts as an University and the bundles were
> worth it.
> 
> The chassis do have multiple PSUs and n 10Gb Ports (40Gb is possible).
> The switch connection is redundant.
> 
> Cuurrently we think of 10 SATA OSD nodes + x SSD Cache Pool Nodes and 5
> MONs. For a start.
> 
> The main focus with the blaids would be spacesaving in the rack. Till
> now I dont have any prize, but that woucld count to in our decision :)
> 
> Thanks and regards . Götz
> 
<...>


-- 
Götz Reinicke
IT-Koordinator

Tel. +49 7141 969 82 420
E-Mail goetz.reini...@filmakademie.de

Filmakademie Baden-Württemberg GmbH
Akademiehof 10
71638 Ludwigsburg
www.filmakademie.de

Eintragung Amtsgericht Stuttgart HRB 205016

Vorsitzender des Aufsichtsrats: Jürgen Walter MdL
Staatssekretär im Ministerium für Wissenschaft,
Forschung und Kunst Baden-Württemberg

Geschäftsführer: Prof. Thomas Schadt



smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cisco UCS Blades as MONs? Pros cons ...?

[Götz Reinicke - IT Koordinator]

Hi Christian,

currently we do get good discounts as an University and the bundles were
worth it.

The chassis do have multiple PSUs and n 10Gb Ports (40Gb is possible).
The switch connection is redundant.

Cuurrently we think of 10 SATA OSD nodes + x SSD Cache Pool Nodes and 5
MONs. For a start.

The main focus with the blaids would be spacesaving in the rack. Till
now I dont have any prize, but that woucld count to in our decision :)

Thanks and regards . Götz

Am 12.05.15 um 14:50 schrieb Christian Balzer:
> 
> Hello,
> 
> I'm not familiar with Cisco UCS gear (can you cite exact models?),
> but somehow the thought of buying compute gear from Cisco makes me think of
> having too much money or very steep discounts. ^o^
> 
> That said, I presume the chassis those blades are in have redundancy in
> terms of PSUs (we always have at least 2 independent power circuits per
> rack) and outside connectivity.
> So from where I'm standing (I have deployed plenty of SuperMicro
> MicroCloud chassis/blades) I'd consider the blade a PoF and be done.
> 
> What I would do (remembering the scale of your planned deployment) is to
> go with one dedicated MON that will be the primary (lowest IP) 99.8% of
> the time and 4 OSDs with MONs on them. If you want to feel extra good
> about this, give those OSDs a bit more CPU/RAM and most of all fast SSDs
> for the OS (/var/lib/ceph).
> 
> Christian
> 
> On Tue, 12 May 2015 14:30:58 +0200 Götz Reinicke - IT Koordinator wrote:
> 
>> Hi,
>>
>> we have some space in our two blade chassis, so I was thinking of the
>> pros and cons of using some blades as MONs. I thought about five MONs.
>>
>> Pro: space saving in our rack
>> Con: "just" two blade centers. Two points of failures.
>>
>> From the redundndency POV I'd go with standalone servers, but space
>> could be a bit of a problem currently 
>>
>> Waht do you think?
>>
>>  Regards . Götz
> 
> 


-- 
Götz Reinicke
IT-Koordinator

Tel. +49 7141 969 82 420
E-Mail goetz.reini...@filmakademie.de

Filmakademie Baden-Württemberg GmbH
Akademiehof 10
71638 Ludwigsburg
www.filmakademie.de

Eintragung Amtsgericht Stuttgart HRB 205016

Vorsitzender des Aufsichtsrats: Jürgen Walter MdL
Staatssekretär im Ministerium für Wissenschaft,
Forschung und Kunst Baden-Württemberg

Geschäftsführer: Prof. Thomas Schadt



smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Cisco UCS Blades as MONs? Pros cons ...?

[Götz Reinicke - IT Koordinator]

Hi,

we have some space in our two blade chassis, so I was thinking of the
pros and cons of using some blades as MONs. I thought about five MONs.

Pro: space saving in our rack
Con: "just" two blade centers. Two points of failures.

From the redundndency POV I'd go with standalone servers, but space
could be a bit of a problem currently 

Waht do you think?

Regards . Götz
-- 
Götz Reinicke
IT-Koordinator

Tel. +49 7141 969 82 420
E-Mail goetz.reini...@filmakademie.de

Filmakademie Baden-Württemberg GmbH
Akademiehof 10
71638 Ludwigsburg
www.filmakademie.de

Eintragung Amtsgericht Stuttgart HRB 205016

Vorsitzender des Aufsichtsrats: Jürgen Walter MdL
Staatssekretär im Ministerium für Wissenschaft,
Forschung und Kunst Baden-Württemberg

Geschäftsführer: Prof. Thomas Schadt



smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Dataflow/path Client <---> OSD

[Götz Reinicke - IT Koordinator]

Hi,

still designing and deciding, we asked ourself: How dose the data
travels from and to an OSD?

E.G. I have my Fileserver with a rbd mounted and a client workstation
writes/read to/from a share on that rbd.

Is the data directly going to an OSD (node) or is it e.g. "travelling"
trough the monitors as well.

The point is: If we connect our file servers and OSD nodes with 40Gb,
dose the monitor need 40Gb to? Or would be 10Gb "enough".

Oversize is ok :) ...

Thansk and regards . Götz

-- 
Götz Reinicke
IT-Koordinator

Tel. +49 7141 969 82 420
E-Mail goetz.reini...@filmakademie.de

Filmakademie Baden-Württemberg GmbH
Akademiehof 10
71638 Ludwigsburg
www.filmakademie.de

Eintragung Amtsgericht Stuttgart HRB 205016

Vorsitzender des Aufsichtsrats: Jürgen Walter MdL
Staatssekretär im Ministerium für Wissenschaft,
Forschung und Kunst Baden-Württemberg

Geschäftsführer: Prof. Thomas Schadt



smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] How to backup hundreds or thousands of TB

[Götz Reinicke - IT Koordinator]

Hi folks,

beside hardware and performance and failover design: How do you manage
to backup hundreds or thousands of TB :) ?

Any suggestions? Best practice?

A second ceph cluster at a different location? "bigger archive" Disks in
good boxes? Or tabe-libs?

What kind of backupsoftware can handle such volumes nicely?

Thanks and regards . Götz
-- 
Götz Reinicke
IT-Koordinator

Tel. +49 7141 969 82 420
E-Mail goetz.reini...@filmakademie.de

Filmakademie Baden-Württemberg GmbH
Akademiehof 10
71638 Ludwigsburg
www.filmakademie.de

Eintragung Amtsgericht Stuttgart HRB 205016

Vorsitzender des Aufsichtsrats: Jürgen Walter MdL
Staatssekretär im Ministerium für Wissenschaft,
Forschung und Kunst Baden-Württemberg

Geschäftsführer: Prof. Thomas Schadt



smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] capacity planing with SSD Cache Pool Tiering

[Götz Reinicke - IT Koordinator]

Thanks Marc & Nick, that makes things much more clear!

/Götz

Am 05.05.15 um 11:36 schrieb Nick Fisk:
> Just to add, the caching promote/demotes whole objects, so if you have
> lots of small random IO’s you will need a lot more cache than compared
> to the actual amount of hot data. Reducing the RBD object size can help
> with this, but YMMV
> 
>  
> 
> Also don’t try and compare Ceph Tiering to a generic cache. With a
> generic cache you tend to get a benefit even when the cache is too
> small, however due to the way Ceph promote/demotes, cache misses are
> very expensive and I have found that unless the bulk of you’re working
> set fits in the cache tier, then performance can actually be worse than
> without the cache.
> 
>  
> 
> *From:*ceph-users [mailto:ceph-users-boun...@lists.ceph.com] *On Behalf
> Of *Marc
> *Sent:* 05 May 2015 10:25
> *To:* ceph-users@lists.ceph.com; goetz.reini...@filmakademie.de
> *Subject:* Re: [ceph-users] capacity planing with SSD Cache Pool Tiering
> 
>  
> 
> Hi,
> 
> The cache doesn't give you any additional storage capacity as the cache
> can never store data, thats not on the tier below it (or store more
> writes than the underlying storage has room for).
> 
> As for how much you should go for... thats very much up to your use
> case. Try to come up with an estimate of how much data is frequently
> being accessed (this is the data most likely to remain in the cache).
> Then double that estimate - ALWAYS double your estimates ;) (this isn't
> Ceph-specific).
> 
> There might be additional magic, but in general the cache will store all
> the data that is being read from the underlying storage (in hopes of it
> being required again later) as well as any writes that may occur (if you
> don't configure the cache to be read-only that is). Do note that this
> also means that currently (afaik this is being worked on) pulling a
> backup of your RBDs will completely flush the cache. This only means
> that the files you'd want cached will have to be pulled back in after
> that and you may lose the performance advantage for a little while after
> each backup.
> 
> Hope that helps, dont hesitate with further inquiries!
> 
> 
> Marc


-- 
Götz Reinicke
IT-Koordinator

Tel. +49 7141 969 82 420
E-Mail goetz.reini...@filmakademie.de

Filmakademie Baden-Württemberg GmbH
Akademiehof 10
71638 Ludwigsburg
www.filmakademie.de

Eintragung Amtsgericht Stuttgart HRB 205016

Vorsitzender des Aufsichtsrats: Jürgen Walter MdL
Staatssekretär im Ministerium für Wissenschaft,
Forschung und Kunst Baden-Württemberg

Geschäftsführer: Prof. Thomas Schadt



smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] capacity planing with SSD Cache Pool Tiering

[Götz Reinicke - IT Koordinator]

Hi folks,

one more question:

after some more interanal discussions, I'm faced with the question how a
SSD Cache Pool Tiering is calculated in the "overall" usable storage space.

And how "big" do I calculate an SSD Cache Pool?

From my understanding, the cache pool is not calculated into the overall
usable space. It is a "cache".

E.g. The slow pool is 100 TB, the SSD Cache 10 TB, I dont have 110TB all
in all?

True? I'm wrong?

As always thanks a lot and regards! Götz

-- 
Götz Reinicke
IT-Koordinator

Tel. +49 7141 969 82 420
E-Mail goetz.reini...@filmakademie.de

Filmakademie Baden-Württemberg GmbH
Akademiehof 10
71638 Ludwigsburg
www.filmakademie.de

Eintragung Amtsgericht Stuttgart HRB 205016

Vorsitzender des Aufsichtsrats: Jürgen Walter MdL
Staatssekretär im Ministerium für Wissenschaft,
Forschung und Kunst Baden-Württemberg

Geschäftsführer: Prof. Thomas Schadt



smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] One more thing. Journal or not to journal or DB-what? Status?

[Götz Reinicke - IT Koordinator]

Dear folks,

I'm sorry for the strange subject, but that might show my current
confusion too.

From what I know the writes to an OSD are also journaled for speed and
consistency. Currently that is done to the/a filesystem, that's why a
lot of suggestion are to use SSD for journals.

So far, that's clear.

But I don't understand what the leveldb/rocksdb/LMDB have to do with it.

From what I'v read and understand, those DBs could make the journal with
SSD obsolete. Keyword Filestore/KeyValueStore.

True? Wrong? :)

May be someone can explain this to me? And may be there is a roadmap on
the progress?

We hope to reduce the systems complexity (dedicated journal SSDs) with that.

http://tracker.ceph.com/issues/11028 says "LMDB key/value backend for
Ceph" is done by 70% 15 days ago.


Kowtow, kowtow and thanks . Götz

-- 
Götz Reinicke
IT-Koordinator

Tel. +49 7141 969 82 420
E-Mail goetz.reini...@filmakademie.de

Filmakademie Baden-Württemberg GmbH
Akademiehof 10
71638 Ludwigsburg
www.filmakademie.de

Eintragung Amtsgericht Stuttgart HRB 205016

Vorsitzender des Aufsichtsrats: Jürgen Walter MdL
Staatssekretär im Ministerium für Wissenschaft,
Forschung und Kunst Baden-Württemberg

Geschäftsführer: Prof. Thomas Schadt



smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] inktank configuration guides are gone?

[Götz Reinicke - IT Koordinator]

Hi,

here I saw some links that sound interisting to me regarding Hardware
planing: https://ceph.com/category/resources/

The links redirect to Redhat, and I cant find the content.

May be someone has a new Guid? I found one from 2013 as pdf.

Regards and Thanks . Götz
-- 
Götz Reinicke
IT-Koordinator

Tel. +49 7141 969 82 420
E-Mail goetz.reini...@filmakademie.de

Filmakademie Baden-Württemberg GmbH
Akademiehof 10
71638 Ludwigsburg
www.filmakademie.de

Eintragung Amtsgericht Stuttgart HRB 205016

Vorsitzender des Aufsichtsrats: Jürgen Walter MdL
Staatssekretär im Ministerium für Wissenschaft,
Forschung und Kunst Baden-Württemberg

Geschäftsführer: Prof. Thomas Schadt



smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Some more numbers - CPU/Memory suggestions for OSDs and Monitors

[Götz Reinicke - IT Koordinator]

Hi folks,

thanks for the feedback regarding the network questions. Currently I try
to solve the question of how much memory, cores and GHz for OSD nodes
and Monitors.

My research so far:

OSD nodes: 2 GB RAM, 2 GHz, 1 Core (?) per OSD

+ enough power to handle the network load.

For the monitors I can't find a real base of information; some are much
stronger (more RAM, more CPU/GHz) some use less performant systems.

And the http://ceph.com/docs/master/start/hardware-recommendations/
looks a little bit "outdated" ... e.g. "... A recent (2012) Ceph cluster
project. ..."

How to calculate a good balanced? Is there a rule of thumb estimate :)?

BTW: The numbers I got are from the recommendation and sample
configurations from DELL, HP, Intel, Supermicron, Emulex, CERN and some
more... Like this list.

Thanks a lot for any suggestion and feedback . Regards . Götz

-- 
Götz Reinicke
IT-Koordinator

Tel. +49 7141 969 82 420
E-Mail goetz.reini...@filmakademie.de

Filmakademie Baden-Württemberg GmbH
Akademiehof 10
71638 Ludwigsburg
www.filmakademie.de

Eintragung Amtsgericht Stuttgart HRB 205016

Vorsitzender des Aufsichtsrats: Jürgen Walter MdL
Staatssekretär im Ministerium für Wissenschaft,
Forschung und Kunst Baden-Württemberg

Geschäftsführer: Prof. Thomas Schadt



smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Network redundancy pro and cons, best practice, suggestions?

[Götz Reinicke - IT Koordinator]

Hi Christian,
Am 13.04.15 um 12:54 schrieb Christian Balzer:
> 
> Hello,
> 
> On Mon, 13 Apr 2015 11:03:24 +0200 Götz Reinicke - IT Koordinator wrote:
> 
>> Dear ceph users,
>>
>> we are planing a ceph storage cluster from scratch. Might be up to 1 PB
>> within the next 3 years, multiple buildings, new network infrastructure
>> for the cluster etc.

<...>

> So at your storage node density of 12 HDDs (16 HDD chassis are not space
> efficient), 40GbE is overkill with a single link/network, insanely so with
> 2 networks.

What would you think, if we go with 20 OSDs, may be 22 OSd (24 HDD
chassis with 2 or 4 SSD (OS / Journal))

From the "Questions about an example of ceph infrastructure" topic ! got
the calculation of

HDDs x expected spped per disk = Max Performance =>

20 x 70 MB/s = 1400 MB/s ie 1.4 GB/s => 14 Gb/s redundant => 28 Gb/s

I like the suggestion from Robert LeBlanc, using two 40Gb ports with VLANs.

Currently we have to extend our LAN anyway, all 10Gb ports are in use.

Upgrading 10Gb ports costs more than buying new 10Gb hardware in our
case. (Good old Cisco 6500 vs. modern 4500x challenge :) )

Furthermore we will see a lot of more traffic and requirements regarding
speed within next year and than an other rise within the next 5 years.
(e.g. 4K/8K video realtime playback for some workstations. 4K is about 9
Gb/s ! per Workstation.)

Long story short, we have/will/should/and can start with 40Gb. The
question is how big :)

I'd say, after some more internal discussions to, redundant switches are
mandatory in our case, and the 40Gb-VLANs a good balance regarding
redundancy, cost and performance.

Thumbs up or down, your vote :D. Seriously what you think?

Thanks for your feedback and best regards . Götz

-- 
Götz Reinicke
IT-Koordinator

Tel. +49 7141 969 82 420
E-Mail goetz.reini...@filmakademie.de

Filmakademie Baden-Württemberg GmbH
Akademiehof 10
71638 Ludwigsburg
www.filmakademie.de

Eintragung Amtsgericht Stuttgart HRB 205016

Vorsitzender des Aufsichtsrats: Jürgen Walter MdL
Staatssekretär im Ministerium für Wissenschaft,
Forschung und Kunst Baden-Württemberg

Geschäftsführer: Prof. Thomas Schadt



smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Network redundancy pro and cons, best practice, suggestions?

[Götz Reinicke - IT Koordinator]

Hi Alexandre,

thanks for that suggestion. mellanox might be on our shoping list
already, but what regarding the redundandency design at all from your POV?

/Götz
Am 13.04.15 um 11:08 schrieb Alexandre DERUMIER:
>>> So what would you suggest, what are your experiences?
> 
> Hi, you can have a look at mellanox sx1012 for example
> http://www.mellanox.com/page/products_dyn?product_family=163
> 
> 12 ports 40GB for around 4000€
> 
> you can use breakout cables to have 4x12 10GB ports.
> 
> 
> They can be stacked with mlag and lacp
> 
> 
> - Mail original -
> De: "Götz Reinicke - IT Koordinator" 
> À: "ceph-users" 
> Envoyé: Lundi 13 Avril 2015 11:03:24
> Objet: [ceph-users] Network redundancy pro and cons, best practice,   
> suggestions?
> 
> Dear ceph users, 
> 
> we are planing a ceph storage cluster from scratch. Might be up to 1 PB 
> within the next 3 years, multiple buildings, new network infrastructure 
> for the cluster etc. 
> 
> I had some excellent trainings on ceph, so the essential fundamentals 
> are familiar to me, and I know our goals/dreams can be reached. :) 
> 
> There is just "one tiny piece" in the design I'm currently unsure about :) 
> 
> Ceph follows some sort of keep it small and simple, e.g. dont use raid 
> controllers, use more boxes and disks, fast network etc. 
> 
> So from our current design we plan 40Gb Storage and Client LAN. 
> 
> Would you suggest to connect the OSD nodes redundant to both networks? 
> That would end up with 4 * 40Gb ports in each box, two Switches to 
> connect to. 
> 
> I'd think of OSD nodes with 12 - 16 * 4TB SATA disks for "high" io 
> pools. (+ currently SSD for journal, but may be until we start, levelDB, 
> rocksDB are ready ... ?) 
> 
> Later some less io bound pools for data archiving/backup. (bigger and 
> more Disks per node) 
> 
> We would also do some Cache tiering for some pools. 
> 
> From HP, Intel, Supermicron etc reference documentations, they use 
> usually non-redundant network connection. (single 10Gb) 
> 
> I know: redundancy keeps some headaches small, but also adds some more 
> complexity and increases the budget. (add network adapters, other 
> server, more switches, etc) 
> 
> So what would you suggest, what are your experiences? 
> 
> Thanks for any suggestion and feedback . Regards . Götz 
> 


-- 
Götz Reinicke
IT-Koordinator

Tel. +49 7141 969 82 420
E-Mail goetz.reini...@filmakademie.de

Filmakademie Baden-Württemberg GmbH
Akademiehof 10
71638 Ludwigsburg
www.filmakademie.de

Eintragung Amtsgericht Stuttgart HRB 205016

Vorsitzender des Aufsichtsrats: Jürgen Walter MdL
Staatssekretär im Ministerium für Wissenschaft,
Forschung und Kunst Baden-Württemberg

Geschäftsführer: Prof. Thomas Schadt



smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Network redundancy pro and cons, best practice, suggestions?

[Götz Reinicke - IT Koordinator]

Dear ceph users,

we are planing a ceph storage cluster from scratch. Might be up to 1 PB
within the next 3 years, multiple buildings, new network infrastructure
for the cluster etc.

I had some excellent trainings on ceph, so the essential fundamentals
are familiar to me, and I know our goals/dreams can be reached. :)

There is just "one tiny piece" in the design I'm currently unsure about :)

Ceph follows some sort of keep it small and simple, e.g. dont use raid
controllers, use more boxes and disks, fast network etc.

So from our current design we plan 40Gb Storage and Client LAN.

Would you suggest to connect the OSD nodes redundant to both networks?
That would end up with 4 * 40Gb ports in each box, two Switches to
connect to.

I'd think of OSD nodes with 12 - 16 * 4TB SATA disks for "high" io
pools. (+ currently SSD for journal, but may be until we start, levelDB,
rocksDB are ready ... ?)

Later some less io bound pools for data archiving/backup. (bigger and
more Disks per node)

We would also do some Cache tiering for some pools.

From HP, Intel, Supermicron etc reference documentations, they use
usually non-redundant network connection. (single 10Gb)

I know: redundancy keeps some headaches small, but also adds some more
complexity and increases the budget. (add network adapters, other
server, more switches, etc)

So what would you suggest, what are your experiences?

Thanks for any suggestion and feedback . Regards . Götz
-- 
Götz Reinicke
IT-Koordinator

Tel. +49 7141 969 82 420
E-Mail goetz.reini...@filmakademie.de

Filmakademie Baden-Württemberg GmbH
Akademiehof 10
71638 Ludwigsburg
www.filmakademie.de

Eintragung Amtsgericht Stuttgart HRB 205016

Vorsitzender des Aufsichtsrats: Jürgen Walter MdL
Staatssekretär im Ministerium für Wissenschaft,
Forschung und Kunst Baden-Württemberg

Geschäftsführer: Prof. Thomas Schadt



smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com