Re: [ceph-users] OSD killed by OOM when many cache available

2017-11-17 Thread Eric Nelson 
As I understand it the memory would show up in /proc/meminfo or vmstat as
cached, however since it's being used for the page cache the kernel cannot
allocate it at the time of oom-killer invocation since it's paged out
(likely for cached reads/writes to your storage device). You could call #
sync ; echo 3 > /proc/sys/vm/drop_caches to immediately free up the memory
but that's interactive. Upping cache pressure and min_free_kbytes makes the
kernel keep things at bay and the system usable.

I had some way of monitoring this but forget since it's been quite a few
fires to fight since then! If I find it I'll send it your way.

Cheers,
E

On Fri, Nov 17, 2017 at 6:03 PM, Sam Huracan 
wrote:

> @Eric: How can I check status of fscache? Why can it be root cause?
>
> Thanks
>
> 2017-11-18 7:30 GMT+07:00 Eric Nelson :
>
>> One thing that doesn't show up is fs cache, which is likely the cause
>> here. We went through this on our SSDs and had to add the following to stop
>> the crashes. I believe vm.vfs_cache_pressure and min_free_kbytes were the
>> really helpful things in getting the crashes to stop. HTH!
>>
>> sysctl_param 'vm.vfs_cache_pressure' do
>>
>>   value 400
>>
>> end
>>
>> sysctl_param 'vm.dirty_ratio' do
>>
>>   value 20
>>
>> end
>>
>> sysctl_param 'vm.dirty_background_ratio' do
>>
>>   value 2
>>
>> end
>>
>> sysctl_param 'vm.min_free_kbytes' do
>>
>>   value 4194304
>>
>> end
>>
>>
>>
>> On Fri, Nov 17, 2017 at 4:24 PM, Sam Huracan 
>> wrote:
>>
>>> I see some more logs about memory in syslog:
>>>
>>> Nov 17 10:47:17 ceph1 kernel: [2810698.553749] Node 0 DMA free:14828kB
>>> min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB
>>> active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB
>>> isolated(file):0kB present:15996kB managed:15896kB mlocked:0kB dirty:0kB
>>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB
>>> slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB
>>> bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB
>>> pages_scanned:0 all_unreclaimable? yes
>>> Nov 17 10:47:17 ceph1 kernel: [2810698.553753] lowmem_reserve[]: 0 1830
>>> 32011 32011 32011
>>> Nov 17 10:47:17 ceph1 kernel: [2810698.553755] Node 0 DMA32
>>> free:134848kB min:3860kB low:4824kB high:5788kB active_anon:120628kB
>>> inactive_anon:121792kB active_file:653404kB inactive_file:399272kB
>>> unevictable:0kB isolated(anon):0kB isolated(file):36kB present:1981184kB
>>> managed:1900752kB mlocked:0kB dirty:344kB writeback:0kB mapped:1200kB
>>> shmem:0kB slab_reclaimable:239900kB slab_unreclaimable:154560kB
>>> kernel_stack:43376kB pagetables:1176kB unstable:0kB bounce:0kB free_pcp:0kB
>>> local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0
>>> all_unreclaimable? no
>>> Nov 17 10:47:17 ceph1 kernel: [2810698.553758] lowmem_reserve[]: 0 0
>>> 30180 30180 30180
>>> Nov 17 10:47:17 ceph1 kernel: [2810698.553760] Node 0 Normal
>>> free:237232kB min:63684kB low:79604kB high:95524kB active_anon:3075228kB
>>> inactive_anon:629052kB active_file:12544336kB inactive_file:12570716kB
>>> unevictable:0kB isolated(anon):0kB isolated(file):0kB present:31457280kB
>>> managed:30905084kB mlocked:0kB dirty:74368kB writeback:3796kB
>>> mapped:48516kB shmem:1276kB slab_reclaimable:713684kB
>>> slab_unreclaimable:416404kB kernel_stack:40896kB pagetables:25288kB
>>> unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
>>> writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
>>> Nov 17 10:47:17 ceph1 kernel: [2810698.553763] lowmem_reserve[]: 0 0 0 0
>>> 0
>>> Nov 17 10:47:17 ceph1 kernel: [2810698.553765] Node 0 DMA: 1*4kB (U)
>>> 1*8kB (U) 0*16kB 1*32kB (U) 3*64kB (U) 2*128kB (U) 0*256kB 0*512kB 0*1024kB
>>> 1*2048kB (M) 3*4096kB (M) = 14828kB
>>> Nov 17 10:47:17 ceph1 kernel: [2810698.553772] Node 0 DMA32: 10756*4kB
>>> (UME) 11442*8kB (UME) 25*16kB (UE) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB
>>> 0*1024kB 0*2048kB 0*4096kB = 134960kB
>>> Nov 17 10:47:17 ceph1 kernel: [2810698.553777] Node 0 Normal: 59473*4kB
>>> (UME) 77*8kB (U) 10*16kB (H) 8*32kB (H) 6*64kB (H) 2*128kB (H) 1*256kB (H)
>>> 1*512kB (H) 0*1024kB 0*2048kB 0*4096kB = 240332kB
>>> Nov 17 10:47:17 ceph1 kernel: [2810698.553784] Node 0 hugepages_total=0
>>> hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
>>> Nov 17 10:47:17 ceph1 kernel: [2810698.553784] Node 0 hugepages_total=0
>>> hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
>>> Nov 17 10:47:17 ceph1 kernel: [2810698.553785] 6544612 total pagecache
>>> pages
>>> Nov 17 10:47:17 ceph1 kernel: [2810698.553786] 2347 pages in swap cache
>>> Nov 17 10:47:17 ceph1 kernel: [2810698.553787] Swap cache stats: add
>>> 2697126, delete 2694779, find 38874122/39241548
>>> Nov 17 10:47:17 ceph1 kernel: [2810698.553788] Free swap  = 61498092kB
>>> Nov 17 10:47:17 ceph1 kernel: [2810698.553788] Total swap = 62498812kB
>>> Nov 17 10:47:17 ceph1 kernel: [2810698.553789] 8363615 pages RAM
>>> Nov 17 10:47:17 ceph1 kernel: [2810698.553790] 0 pages
>>>

[ceph-users] Use case for multiple "zonegroup"s in a realm

2017-11-17 Thread Richard Chan 
What are good use cases for multiple zonegroups: is this mainly a
namespacing thing? What do you use it for?

Please help my understanding:

Within a realm, all zonegroups share the same namespace(object IDs?) for
tenants, users, and buckets(is this correct)? All these entities are
created in the master zone of the master zonegroup.

However separate zonegroups are used if you want to avoid blob sync'ing,
i.e, there is intra-zonegroup sync'ing but no inter-zonegroup sync.

RHCS2 documentation uses only one zonegroup:
https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2/html/object_gateway_guide_for_red_hat_enterprise_linux/multi_site

SES3 documentation uses only one zonegroup:
https://www.suse.com/documentation/ses-3/book_storage_admin/data/ceph_rgw_fed.html

Regards

-- 
Richard Chan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] OSD killed by OOM when many cache available

2017-11-17 Thread Sam Huracan 
@Eric: How can I check status of fscache? Why can it be root cause?

Thanks

2017-11-18 7:30 GMT+07:00 Eric Nelson :

> One thing that doesn't show up is fs cache, which is likely the cause
> here. We went through this on our SSDs and had to add the following to stop
> the crashes. I believe vm.vfs_cache_pressure and min_free_kbytes were the
> really helpful things in getting the crashes to stop. HTH!
>
> sysctl_param 'vm.vfs_cache_pressure' do
>
>   value 400
>
> end
>
> sysctl_param 'vm.dirty_ratio' do
>
>   value 20
>
> end
>
> sysctl_param 'vm.dirty_background_ratio' do
>
>   value 2
>
> end
>
> sysctl_param 'vm.min_free_kbytes' do
>
>   value 4194304
>
> end
>
>
>
> On Fri, Nov 17, 2017 at 4:24 PM, Sam Huracan 
> wrote:
>
>> I see some more logs about memory in syslog:
>>
>> Nov 17 10:47:17 ceph1 kernel: [2810698.553749] Node 0 DMA free:14828kB
>> min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB
>> active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB
>> isolated(file):0kB present:15996kB managed:15896kB mlocked:0kB dirty:0kB
>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB
>> slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB
>> bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB
>> pages_scanned:0 all_unreclaimable? yes
>> Nov 17 10:47:17 ceph1 kernel: [2810698.553753] lowmem_reserve[]: 0 1830
>> 32011 32011 32011
>> Nov 17 10:47:17 ceph1 kernel: [2810698.553755] Node 0 DMA32 free:134848kB
>> min:3860kB low:4824kB high:5788kB active_anon:120628kB
>> inactive_anon:121792kB active_file:653404kB inactive_file:399272kB
>> unevictable:0kB isolated(anon):0kB isolated(file):36kB present:1981184kB
>> managed:1900752kB mlocked:0kB dirty:344kB writeback:0kB mapped:1200kB
>> shmem:0kB slab_reclaimable:239900kB slab_unreclaimable:154560kB
>> kernel_stack:43376kB pagetables:1176kB unstable:0kB bounce:0kB free_pcp:0kB
>> local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0
>> all_unreclaimable? no
>> Nov 17 10:47:17 ceph1 kernel: [2810698.553758] lowmem_reserve[]: 0 0
>> 30180 30180 30180
>> Nov 17 10:47:17 ceph1 kernel: [2810698.553760] Node 0 Normal
>> free:237232kB min:63684kB low:79604kB high:95524kB active_anon:3075228kB
>> inactive_anon:629052kB active_file:12544336kB inactive_file:12570716kB
>> unevictable:0kB isolated(anon):0kB isolated(file):0kB present:31457280kB
>> managed:30905084kB mlocked:0kB dirty:74368kB writeback:3796kB
>> mapped:48516kB shmem:1276kB slab_reclaimable:713684kB
>> slab_unreclaimable:416404kB kernel_stack:40896kB pagetables:25288kB
>> unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
>> writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
>> Nov 17 10:47:17 ceph1 kernel: [2810698.553763] lowmem_reserve[]: 0 0 0 0 0
>> Nov 17 10:47:17 ceph1 kernel: [2810698.553765] Node 0 DMA: 1*4kB (U)
>> 1*8kB (U) 0*16kB 1*32kB (U) 3*64kB (U) 2*128kB (U) 0*256kB 0*512kB 0*1024kB
>> 1*2048kB (M) 3*4096kB (M) = 14828kB
>> Nov 17 10:47:17 ceph1 kernel: [2810698.553772] Node 0 DMA32: 10756*4kB
>> (UME) 11442*8kB (UME) 25*16kB (UE) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB
>> 0*1024kB 0*2048kB 0*4096kB = 134960kB
>> Nov 17 10:47:17 ceph1 kernel: [2810698.553777] Node 0 Normal: 59473*4kB
>> (UME) 77*8kB (U) 10*16kB (H) 8*32kB (H) 6*64kB (H) 2*128kB (H) 1*256kB (H)
>> 1*512kB (H) 0*1024kB 0*2048kB 0*4096kB = 240332kB
>> Nov 17 10:47:17 ceph1 kernel: [2810698.553784] Node 0 hugepages_total=0
>> hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
>> Nov 17 10:47:17 ceph1 kernel: [2810698.553784] Node 0 hugepages_total=0
>> hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
>> Nov 17 10:47:17 ceph1 kernel: [2810698.553785] 6544612 total pagecache
>> pages
>> Nov 17 10:47:17 ceph1 kernel: [2810698.553786] 2347 pages in swap cache
>> Nov 17 10:47:17 ceph1 kernel: [2810698.553787] Swap cache stats: add
>> 2697126, delete 2694779, find 38874122/39241548
>> Nov 17 10:47:17 ceph1 kernel: [2810698.553788] Free swap  = 61498092kB
>> Nov 17 10:47:17 ceph1 kernel: [2810698.553788] Total swap = 62498812kB
>> Nov 17 10:47:17 ceph1 kernel: [2810698.553789] 8363615 pages RAM
>> Nov 17 10:47:17 ceph1 kernel: [2810698.553790] 0 pages HighMem/MovableOnly
>> Nov 17 10:47:17 ceph1 kernel: [2810698.553790] 158182 pages reserved
>> Nov 17 10:47:17 ceph1 kernel: [2810698.553791] 0 pages cma reserved
>>
>> Is it relate to page caches?
>>
>> 2017-11-18 7:22 GMT+07:00 Sam Huracan :
>>
>>> Today, one of our Ceph OSDs was down, I've check syslog and see this OSD
>>> process was killed by OMM
>>>
>>>
>>> Nov 17 10:01:06 ceph1 kernel: [2807926.762304] Out of memory: Kill
>>> process 3330 (ceph-osd) score 7 or sacrifice child
>>> Nov 17 10:01:06 ceph1 kernel: [2807926.763745] Killed process 3330
>>> (ceph-osd) total-vm:2372392kB, anon-rss:559084kB, file-rss:7268kB
>>> Nov 17 10:01:06 ceph1 kernel: [2807926.985830] init: ceph-osd (ceph/6)
>>> main process (3330) killed by KILL signal
>>> Nov 17 10:01:06 ceph1 kernel: [2807926.985844] i

Re: [ceph-users] OSD killed by OOM when many cache available

2017-11-17 Thread Eric Nelson 
One thing that doesn't show up is fs cache, which is likely the cause here.
We went through this on our SSDs and had to add the following to stop the
crashes. I believe vm.vfs_cache_pressure and min_free_kbytes were the
really helpful things in getting the crashes to stop. HTH!

sysctl_param 'vm.vfs_cache_pressure' do

  value 400

end

sysctl_param 'vm.dirty_ratio' do

  value 20

end

sysctl_param 'vm.dirty_background_ratio' do

  value 2

end

sysctl_param 'vm.min_free_kbytes' do

  value 4194304

end



On Fri, Nov 17, 2017 at 4:24 PM, Sam Huracan 
wrote:

> I see some more logs about memory in syslog:
>
> Nov 17 10:47:17 ceph1 kernel: [2810698.553749] Node 0 DMA free:14828kB
> min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB
> active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB
> isolated(file):0kB present:15996kB managed:15896kB mlocked:0kB dirty:0kB
> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB
> slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB
> bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB
> pages_scanned:0 all_unreclaimable? yes
> Nov 17 10:47:17 ceph1 kernel: [2810698.553753] lowmem_reserve[]: 0 1830
> 32011 32011 32011
> Nov 17 10:47:17 ceph1 kernel: [2810698.553755] Node 0 DMA32 free:134848kB
> min:3860kB low:4824kB high:5788kB active_anon:120628kB
> inactive_anon:121792kB active_file:653404kB inactive_file:399272kB
> unevictable:0kB isolated(anon):0kB isolated(file):36kB present:1981184kB
> managed:1900752kB mlocked:0kB dirty:344kB writeback:0kB mapped:1200kB
> shmem:0kB slab_reclaimable:239900kB slab_unreclaimable:154560kB
> kernel_stack:43376kB pagetables:1176kB unstable:0kB bounce:0kB free_pcp:0kB
> local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0
> all_unreclaimable? no
> Nov 17 10:47:17 ceph1 kernel: [2810698.553758] lowmem_reserve[]: 0 0 30180
> 30180 30180
> Nov 17 10:47:17 ceph1 kernel: [2810698.553760] Node 0 Normal free:237232kB
> min:63684kB low:79604kB high:95524kB active_anon:3075228kB
> inactive_anon:629052kB active_file:12544336kB inactive_file:12570716kB
> unevictable:0kB isolated(anon):0kB isolated(file):0kB present:31457280kB
> managed:30905084kB mlocked:0kB dirty:74368kB writeback:3796kB
> mapped:48516kB shmem:1276kB slab_reclaimable:713684kB
> slab_unreclaimable:416404kB kernel_stack:40896kB pagetables:25288kB
> unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
> writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> Nov 17 10:47:17 ceph1 kernel: [2810698.553763] lowmem_reserve[]: 0 0 0 0 0
> Nov 17 10:47:17 ceph1 kernel: [2810698.553765] Node 0 DMA: 1*4kB (U) 1*8kB
> (U) 0*16kB 1*32kB (U) 3*64kB (U) 2*128kB (U) 0*256kB 0*512kB 0*1024kB
> 1*2048kB (M) 3*4096kB (M) = 14828kB
> Nov 17 10:47:17 ceph1 kernel: [2810698.553772] Node 0 DMA32: 10756*4kB
> (UME) 11442*8kB (UME) 25*16kB (UE) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB
> 0*1024kB 0*2048kB 0*4096kB = 134960kB
> Nov 17 10:47:17 ceph1 kernel: [2810698.553777] Node 0 Normal: 59473*4kB
> (UME) 77*8kB (U) 10*16kB (H) 8*32kB (H) 6*64kB (H) 2*128kB (H) 1*256kB (H)
> 1*512kB (H) 0*1024kB 0*2048kB 0*4096kB = 240332kB
> Nov 17 10:47:17 ceph1 kernel: [2810698.553784] Node 0 hugepages_total=0
> hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
> Nov 17 10:47:17 ceph1 kernel: [2810698.553784] Node 0 hugepages_total=0
> hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> Nov 17 10:47:17 ceph1 kernel: [2810698.553785] 6544612 total pagecache
> pages
> Nov 17 10:47:17 ceph1 kernel: [2810698.553786] 2347 pages in swap cache
> Nov 17 10:47:17 ceph1 kernel: [2810698.553787] Swap cache stats: add
> 2697126, delete 2694779, find 38874122/39241548
> Nov 17 10:47:17 ceph1 kernel: [2810698.553788] Free swap  = 61498092kB
> Nov 17 10:47:17 ceph1 kernel: [2810698.553788] Total swap = 62498812kB
> Nov 17 10:47:17 ceph1 kernel: [2810698.553789] 8363615 pages RAM
> Nov 17 10:47:17 ceph1 kernel: [2810698.553790] 0 pages HighMem/MovableOnly
> Nov 17 10:47:17 ceph1 kernel: [2810698.553790] 158182 pages reserved
> Nov 17 10:47:17 ceph1 kernel: [2810698.553791] 0 pages cma reserved
>
> Is it relate to page caches?
>
> 2017-11-18 7:22 GMT+07:00 Sam Huracan :
>
>> Today, one of our Ceph OSDs was down, I've check syslog and see this OSD
>> process was killed by OMM
>>
>>
>> Nov 17 10:01:06 ceph1 kernel: [2807926.762304] Out of memory: Kill
>> process 3330 (ceph-osd) score 7 or sacrifice child
>> Nov 17 10:01:06 ceph1 kernel: [2807926.763745] Killed process 3330
>> (ceph-osd) total-vm:2372392kB, anon-rss:559084kB, file-rss:7268kB
>> Nov 17 10:01:06 ceph1 kernel: [2807926.985830] init: ceph-osd (ceph/6)
>> main process (3330) killed by KILL signal
>> Nov 17 10:01:06 ceph1 kernel: [2807926.985844] init: ceph-osd (ceph/6)
>> main process ended, respawning
>> Nov 17 10:03:39 ceph1 bash: root [15524]: sudo ceph health detail [0]
>> Nov 17 10:03:40 ceph1 bash: root [15524]: sudo ceph health detail [0]
>> Nov 17 10:17:01 ceph1 CRON[75167]: (r

Re: [ceph-users] OSD killed by OOM when many cache available

2017-11-17 Thread Sam Huracan 
I see some more logs about memory in syslog:

Nov 17 10:47:17 ceph1 kernel: [2810698.553749] Node 0 DMA free:14828kB
min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB
active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:15996kB managed:15896kB mlocked:0kB dirty:0kB
writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB
slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB
bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:0 all_unreclaimable? yes
Nov 17 10:47:17 ceph1 kernel: [2810698.553753] lowmem_reserve[]: 0 1830
32011 32011 32011
Nov 17 10:47:17 ceph1 kernel: [2810698.553755] Node 0 DMA32 free:134848kB
min:3860kB low:4824kB high:5788kB active_anon:120628kB
inactive_anon:121792kB active_file:653404kB inactive_file:399272kB
unevictable:0kB isolated(anon):0kB isolated(file):36kB present:1981184kB
managed:1900752kB mlocked:0kB dirty:344kB writeback:0kB mapped:1200kB
shmem:0kB slab_reclaimable:239900kB slab_unreclaimable:154560kB
kernel_stack:43376kB pagetables:1176kB unstable:0kB bounce:0kB free_pcp:0kB
local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0
all_unreclaimable? no
Nov 17 10:47:17 ceph1 kernel: [2810698.553758] lowmem_reserve[]: 0 0 30180
30180 30180
Nov 17 10:47:17 ceph1 kernel: [2810698.553760] Node 0 Normal free:237232kB
min:63684kB low:79604kB high:95524kB active_anon:3075228kB
inactive_anon:629052kB active_file:12544336kB inactive_file:12570716kB
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:31457280kB
managed:30905084kB mlocked:0kB dirty:74368kB writeback:3796kB
mapped:48516kB shmem:1276kB slab_reclaimable:713684kB
slab_unreclaimable:416404kB kernel_stack:40896kB pagetables:25288kB
unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 17 10:47:17 ceph1 kernel: [2810698.553763] lowmem_reserve[]: 0 0 0 0 0
Nov 17 10:47:17 ceph1 kernel: [2810698.553765] Node 0 DMA: 1*4kB (U) 1*8kB
(U) 0*16kB 1*32kB (U) 3*64kB (U) 2*128kB (U) 0*256kB 0*512kB 0*1024kB
1*2048kB (M) 3*4096kB (M) = 14828kB
Nov 17 10:47:17 ceph1 kernel: [2810698.553772] Node 0 DMA32: 10756*4kB
(UME) 11442*8kB (UME) 25*16kB (UE) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB
0*1024kB 0*2048kB 0*4096kB = 134960kB
Nov 17 10:47:17 ceph1 kernel: [2810698.553777] Node 0 Normal: 59473*4kB
(UME) 77*8kB (U) 10*16kB (H) 8*32kB (H) 6*64kB (H) 2*128kB (H) 1*256kB (H)
1*512kB (H) 0*1024kB 0*2048kB 0*4096kB = 240332kB
Nov 17 10:47:17 ceph1 kernel: [2810698.553784] Node 0 hugepages_total=0
hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 17 10:47:17 ceph1 kernel: [2810698.553784] Node 0 hugepages_total=0
hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 17 10:47:17 ceph1 kernel: [2810698.553785] 6544612 total pagecache pages
Nov 17 10:47:17 ceph1 kernel: [2810698.553786] 2347 pages in swap cache
Nov 17 10:47:17 ceph1 kernel: [2810698.553787] Swap cache stats: add
2697126, delete 2694779, find 38874122/39241548
Nov 17 10:47:17 ceph1 kernel: [2810698.553788] Free swap  = 61498092kB
Nov 17 10:47:17 ceph1 kernel: [2810698.553788] Total swap = 62498812kB
Nov 17 10:47:17 ceph1 kernel: [2810698.553789] 8363615 pages RAM
Nov 17 10:47:17 ceph1 kernel: [2810698.553790] 0 pages HighMem/MovableOnly
Nov 17 10:47:17 ceph1 kernel: [2810698.553790] 158182 pages reserved
Nov 17 10:47:17 ceph1 kernel: [2810698.553791] 0 pages cma reserved

Is it relate to page caches?

2017-11-18 7:22 GMT+07:00 Sam Huracan :

> Today, one of our Ceph OSDs was down, I've check syslog and see this OSD
> process was killed by OMM
>
>
> Nov 17 10:01:06 ceph1 kernel: [2807926.762304] Out of memory: Kill process
> 3330 (ceph-osd) score 7 or sacrifice child
> Nov 17 10:01:06 ceph1 kernel: [2807926.763745] Killed process 3330
> (ceph-osd) total-vm:2372392kB, anon-rss:559084kB, file-rss:7268kB
> Nov 17 10:01:06 ceph1 kernel: [2807926.985830] init: ceph-osd (ceph/6)
> main process (3330) killed by KILL signal
> Nov 17 10:01:06 ceph1 kernel: [2807926.985844] init: ceph-osd (ceph/6)
> main process ended, respawning
> Nov 17 10:03:39 ceph1 bash: root [15524]: sudo ceph health detail [0]
> Nov 17 10:03:40 ceph1 bash: root [15524]: sudo ceph health detail [0]
> Nov 17 10:17:01 ceph1 CRON[75167]: (root) CMD (   cd / && run-parts
> --report /etc/cron.hourly)
> Nov 17 10:47:17 ceph1 kernel: [2810698.553690] ceph-status.sh invoked
> oom-killer: gfp_mask=0x26000c0, order=2, oom_score_adj=0
> Nov 17 10:47:17 ceph1 kernel: [2810698.553693] ceph-status.sh cpuset=/
> mems_allowed=0
> Nov 17 10:47:17 ceph1 kernel: [2810698.553697] CPU: 5 PID: 194271 Comm:
> ceph-status.sh Not tainted 4.4.0-62-generic #83~14.04.1-Ubuntu
> Nov 17 10:47:17 ceph1 kernel: [2810698.553698] Hardware name: Dell Inc.
> PowerEdge R730xd/072T6D, BIOS 2.5.5 08/16/2017
> Nov 17 10:47:17 ceph1 kernel: [2810698.553699]  
> 88001a857b38 813dc4ac 88001a857cf0
> Nov 17 10:47:17 ceph1 kernel: [281069

[ceph-users] OSD killed by OOM when many cache available

2017-11-17 Thread Sam Huracan 
Today, one of our Ceph OSDs was down, I've check syslog and see this OSD
process was killed by OMM


Nov 17 10:01:06 ceph1 kernel: [2807926.762304] Out of memory: Kill process
3330 (ceph-osd) score 7 or sacrifice child
Nov 17 10:01:06 ceph1 kernel: [2807926.763745] Killed process 3330
(ceph-osd) total-vm:2372392kB, anon-rss:559084kB, file-rss:7268kB
Nov 17 10:01:06 ceph1 kernel: [2807926.985830] init: ceph-osd (ceph/6) main
process (3330) killed by KILL signal
Nov 17 10:01:06 ceph1 kernel: [2807926.985844] init: ceph-osd (ceph/6) main
process ended, respawning
Nov 17 10:03:39 ceph1 bash: root [15524]: sudo ceph health detail [0]
Nov 17 10:03:40 ceph1 bash: root [15524]: sudo ceph health detail [0]
Nov 17 10:17:01 ceph1 CRON[75167]: (root) CMD (   cd / && run-parts
--report /etc/cron.hourly)
Nov 17 10:47:17 ceph1 kernel: [2810698.553690] ceph-status.sh invoked
oom-killer: gfp_mask=0x26000c0, order=2, oom_score_adj=0
Nov 17 10:47:17 ceph1 kernel: [2810698.553693] ceph-status.sh cpuset=/
mems_allowed=0
Nov 17 10:47:17 ceph1 kernel: [2810698.553697] CPU: 5 PID: 194271 Comm:
ceph-status.sh Not tainted 4.4.0-62-generic #83~14.04.1-Ubuntu
Nov 17 10:47:17 ceph1 kernel: [2810698.553698] Hardware name: Dell Inc.
PowerEdge R730xd/072T6D, BIOS 2.5.5 08/16/2017
Nov 17 10:47:17 ceph1 kernel: [2810698.553699]  
88001a857b38 813dc4ac 88001a857cf0
Nov 17 10:47:17 ceph1 kernel: [2810698.553701]  
88001a857bc8 811fb066 88083d29e200
Nov 17 10:47:17 ceph1 kernel: [2810698.553703]  88001a857cf0
88001a857c00 8808453cf000 
Nov 17 10:47:17 ceph1 kernel: [2810698.553704] Call Trace:
Nov 17 10:47:17 ceph1 kernel: [2810698.553709]  []
dump_stack+0x63/0x87
Nov 17 10:47:17 ceph1 kernel: [2810698.553713]  []
dump_header+0x5b/0x1d5
Nov 17 10:47:17 ceph1 kernel: [2810698.553717]  []
oom_kill_process+0x205/0x3d0
Nov 17 10:47:17 ceph1 kernel: [2810698.553718]  [] ?
oom_unkillable_task+0x9e/0xc0
Nov 17 10:47:17 ceph1 kernel: [2810698.553720]  []
out_of_memory+0x40b/0x460
Nov 17 10:47:17 ceph1 kernel: [2810698.553722]  []
__alloc_pages_slowpath.constprop.87+0x742/0x7ad
Nov 17 10:47:17 ceph1 kernel: [2810698.553725]  []
__alloc_pages_nodemask+0x237/0x240
Nov 17 10:47:17 ceph1 kernel: [2810698.553727]  []
alloc_kmem_pages_node+0x4d/0xd0
Nov 17 10:47:17 ceph1 kernel: [2810698.553730]  []
copy_process+0x185/0x1ce0
Nov 17 10:47:17 ceph1 kernel: [2810698.553732]  [] ?
security_file_alloc+0x33/0x50
Nov 17 10:47:17 ceph1 kernel: [2810698.553734]  []
_do_fork+0x8a/0x310
Nov 17 10:47:17 ceph1 kernel: [2810698.553737]  [] ?
sigprocmask+0x51/0x80
Nov 17 10:47:17 ceph1 kernel: [2810698.553738]  []
SyS_clone+0x19/0x20
Nov 17 10:47:17 ceph1 kernel: [2810698.553743]  []
entry_SYSCALL_64_fastpath+0x16/0x75
Nov 17 10:47:17 ceph1 kernel: [2810698.553744] Mem-Info:
Nov 17 10:47:17 ceph1 kernel: [2810698.553747] active_anon:798964
inactive_anon:187711 isolated_anon:0
Nov 17 10:47:17 ceph1 kernel: [2810698.553747]  active_file:3299435
inactive_file:3242497 isolated_file:9
Nov 17 10:47:17 ceph1 kernel: [2810698.553747]  unevictable:0 dirty:18678
writeback:949 unstable:0
Nov 17 10:47:17 ceph1 kernel: [2810698.553747]  slab_reclaimable:238396
slab_unreclaimable:142741
Nov 17 10:47:17 ceph1 kernel: [2810698.553747]  mapped:12429 shmem:319
pagetables:6616 bounce:0
Nov 17 10:47:17 ceph1 kernel: [2810698.553747]  free:96727 free_pcp:0
free_cma:0
Nov 17 10:47:17 ceph1 kernel: [2810698.553749] Node 0 DMA free:14828kB
min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB
active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:15996kB managed:15896kB mlocked:0kB dirty:0kB
writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB
slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB
bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:0 all_unreclaimable? yes
Nov 17 10:47:17 ceph1 kernel: [2810698.553753] lowmem_reserve[]: 0 1830
32011 32011 32011
Nov 17 10:47:17 ceph1 kernel: [2810698.553755] Node 0 DMA32 free:134848kB
min:3860kB low:4824kB high:5788kB active_anon:120628kB
inactive_anon:121792kB active_file:653404kB inactive_file:399272kB
unevictable:0kB isolated(anon):0kB isolated(file):36kB present:1981184kB
managed:1900752kB mlocked:0kB dirty:344kB writeback:0kB mapped:1200kB
shmem:0kB slab_reclaimable:239900kB slab_unreclaimable:154560kB
kernel_stack:43376kB pagetables:1176kB unstable:0kB bounce:0kB free_pcp:0kB
local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0
all_unreclaimable? no


We check free Memory, Cache is free 24,705 MB

root@ceph1:/var/log# free -m
 total   used   free sharedbuffers cached
Mem: 32052  31729323  1125  24256
-/+ buffers/cache:   7347  24705
Swap:61033120  60913



It's so weird. Could you help me solve this problem, I'm afraid it 'll come
again.

Thanks in a

[ceph-users] I/O stalls when doing fstrim on large RBD

2017-11-17 Thread Brendan Moloney 
Hi,

I guess this isn't strictly about Ceph, but I feel like other folks here must 
have run into the same issues.

I am trying to keep my thinly provisioned RBD volumes thin.  I use virtio-scsi 
to attach the RBD volumes to my VMs with the "discard=unmap" option. The RBD is 
formatted as XFS and some of them can be quite large (16TB+).  I have a cron 
job that runs "fstrim" commands twice a week in the evenings.

The issue is that I see massive I/O stalls on the VM during the fstrim.  To the 
point where I am getting kernel panics from hung tasks and other timeouts.  I 
have tried a number of things to lessen the impact:

- Switching from deadline to CFQ (initially I thought this helped, but now 
I am not convinced)
- Running fstrim with "ionice -c idle" (this doesn't seem to make a 
difference)
- Chunking the fstrim with the offset/length options (helps reduce worst 
case, but I can't trim less than 1TB at a time and that can still cause a pause 
for several minutes)

Is there anything else I can do to avoid this issue?

Thanks,
Brendan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Bluestore performance 50% of filestore

2017-11-17 Thread Milanov, Radoslav Nikiforov 
Here's some more results, I'm reading 12.2.2 will have performance improvements 
for bluestore and should be released soon? 

Iodepth=not specified
Filestore
  write: io=3511.9MB, bw=19978KB/s, iops=4994, runt=180001msec
  write: io=3525.6MB, bw=20057KB/s, iops=5014, runt=180001msec
  write: io=3554.1MB, bw=20222KB/s, iops=5055, runt=180016msec

  read : io=1995.7MB, bw=11353KB/s, iops=2838, runt=180001msec
  read : io=1824.5MB, bw=10379KB/s, iops=2594, runt=180001msec
  read : io=1966.5MB, bw=11187KB/s, iops=2796, runt=180001msec

Bluestore
  write: io=1621.2MB, bw=9222.3KB/s, iops=2305, runt=180002msec
  write: io=1576.3MB, bw=8965.6KB/s, iops=2241, runt=180029msec
  write: io=1531.9MB, bw=8714.3KB/s, iops=2178, runt=180001msec

  read : io=1279.4MB, bw=7276.5KB/s, iops=1819, runt=180006msec
  read : io=773824KB, bw=4298.9KB/s, iops=1074, runt=180010msec
  read : io=1018.5MB, bw=5793.7KB/s, iops=1448, runt=180001msec

Iodepth=10
Filestore
  write: io=5045.1MB, bw=28706KB/s, iops=7176, runt=180001msec
  write: io=4764.7MB, bw=27099KB/s, iops=6774, runt=180021msec
  write: io=4626.2MB, bw=26318KB/s, iops=6579, runt=180031msec

  read : io=1745.3MB, bw=9928.6KB/s, iops=2482, runt=180001msec
  read : io=1933.7MB, bw=11000KB/s, iops=2749, runt=180001msec
  read : io=1952.7MB, bw=11108KB/s, iops=2777, runt=180001msec

Bluestore
  write: io=1578.8MB, bw=8980.9KB/s, iops=2245, runt=180006msec
  write: io=1583.9MB, bw=9010.2KB/s, iops=2252, runt=180002msec
  write: io=1591.5MB, bw=9050.9KB/s, iops=2262, runt=180009msec

  read : io=412104KB, bw=2289.5KB/s, iops=572, runt=180002msec
  read : io=718108KB, bw=3989.5KB/s, iops=997, runt=180003msec
  read : io=968388KB, bw=5379.7KB/s, iops=1344, runt=180009msec

Iodpeth=20
Filestore
  write: io=4671.2MB, bw=26574KB/s, iops=6643, runt=180001msec
  write: io=4583.4MB, bw=26066KB/s, iops=6516, runt=180054msec
  write: io=4641.6MB, bw=26347KB/s, iops=6586, runt=180395msec

  read : io=2094.3MB, bw=11914KB/s, iops=2978, runt=180001msec
  read : io=1997.6MB, bw=11364KB/s, iops=2840, runt=180001msec
  read : io=2028.4MB, bw=11539KB/s, iops=2884, runt=180001msec

Bluestore
  write: io=1595.8MB, bw=9078.2KB/s, iops=2269, runt=180001msec
  write: io=1596.2MB, bw=9080.6KB/s, iops=2270, runt=180001msec
  write: io=1588.3MB, bw=9035.4KB/s, iops=2258, runt=180002msec

  read : io=1126.9MB, bw=6410.5KB/s, iops=1602, runt=180004msec
  read : io=1282.4MB, bw=7295.3KB/s, iops=1823, runt=180003msec
  read : io=1380.9MB, bw=7854.1KB/s, iops=1963, runt=180007msec


- Rado

-Original Message-
From: Mark Nelson [mailto:mnel...@redhat.com] 
Sent: Thursday, November 16, 2017 2:04 PM
To: Milanov, Radoslav Nikiforov ; David Turner 

Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Bluestore performance 50% of filestore

It depends on what you expect your typical workload to be like.  Ceph (and 
distributed storage in general) likes high io depths so writes can hit all of 
the drives at the same time.  There are tricks (like journals, writahead logs, 
centralized caches, etc) that can help mitigate this, but I suspect you'll see 
much better performance with more concurrent writes.

Regarding file size, the smaller the file, the more likely those tricks 
mentioned above are to help you.  Based on your results, it appears filestore 
may be doing a better job of it than bluestore.  The question you have to ask 
is whether or not this kind of test represents what you are likely to see for 
real on your cluster.

Doing writes over a much larger file, say 3-4x over the total amount of RAM in 
all of the nodes, helps you get a better idea of what the behavior is like when 
those tricks are less effective.  I think that's probably a more likely 
scenario in most production environments, but it's up to you which workload you 
think better represents what you are going to see in practice.  A while back 
Nick Fisk showed some results wehre bluestore was slower than filestore at 
small sync writes and it could be that we simply have more work to do in this 
area.  On the other hand, we pretty consistently see bluestore doing better 
than filestore with 4k random writes and higher IO depths, which is why I'd be 
curious to see how it goes if you try that.

Mark

On 11/16/2017 10:11 AM, Milanov, Radoslav Nikiforov wrote:
> No,
> What test parameters (iodepth/file size/numjobs) would make sense  for 3 
> node/27OSD@4TB ?
> - Rado
>
> -Original Message-
> From: Mark Nelson [mailto:mnel...@redhat.com]
> Sent: Thursday, November 16, 2017 10:56 AM
> To: Milanov, Radoslav Nikiforov ; David Turner 
> 
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] Bluestore performance 50% of filestore
>
> Did you happen to have a chance to try with a higher io depth?
>
> Mark
>
> On 11/16/2017 09:53 AM, Milanov, Radoslav Nikiforov wrote:
>> FYI
>>
>> Having 50GB bock.db made no difference on the performance.
>>
>>
>>
>> - Rado
>>
>>
>>
>> *From:*David Turner [mailto:drakonst...@gmail.com]
>> *Se

Re: [ceph-users] Moving bluestore WAL and DB after bluestore creation

2017-11-17 Thread Ronny Aasen 

On 16.11.2017 09:45, Loris Cuoghi wrote:

Le Wed, 15 Nov 2017 19:46:48 +,
Shawn Edwards  a écrit :


On Wed, Nov 15, 2017, 11:07 David Turner 
wrote:


I'm not going to lie.  This makes me dislike Bluestore quite a
bit.  Using multiple OSDs to an SSD journal allowed for you to
monitor the write durability of the SSD and replace it without
having to out and re-add all of the OSDs on the device.  Having to
now out and backfill back onto the HDDs is awful and would have
made a time when I realized that 20 journal SSDs all ran low on
writes at the same time nearly impossible to recover from.

Flushing journals, replacing SSDs, and bringing it all back online
was a slick process.  Formatting the HDDs and backfilling back onto
the same disks sounds like a big regression.  A process to migrate
the WAL and DB onto the HDD and then back off to a new device would
be very helpful.

On Wed, Nov 15, 2017 at 10:51 AM Mario Giammarco
 wrote:
  

It seems it is not possible. I recreated the OSD

2017-11-12 17:44 GMT+01:00 Shawn Edwards :
  

I've created some Bluestore OSD with all data (wal, db, and data)
all on the same rotating disk.  I would like to now move the wal
and db onto an nvme disk.  Is that possible without re-creating
the OSD?

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

  

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
  

This.  Exactly this.  Not being able to move the .db and .wal data on
and off the main storage disk on Bluestore is a regression.


Hello,

What stops you from dd'ing the DB/WAL's partitions on another disk and
updating the symlinks in the OSD's mount point under /var/lib/ceph/osd?



this probably works when you deployed bluestore with partitions, but if 
you did not create partitions for block.db on orginal bluestore creation 
there is no block.db symlink, db and wal are mixed into the block 
partition and not easy to extract.  also just dd the block device may 
not help if you want to change the size of the db partition. this needs 
more testing.  probably tools can be created in the future for resizing  
db and wal partitions, and for extracting db data from block into a 
separate block.db partition.


dd block.db would probably work when you need to replace a worn out ssd 
drive. but not so much if you want to deploy separate block.db from a 
bluestore made without block.db



kind regards
Ronny Aasen





___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Rename iscsi target_iqn

2017-11-17 Thread Jason Dillaman 
On Fri, Nov 17, 2017 at 8:01 AM, Frank Brendel
 wrote:
> Hi,
>
> how can I rename an iscsi target_iqn?

That operation is not supported via gwcli.

> And where is the configuration that I made with gwcli stored?

It's stored in a JSON object within the 'rbd' pool named "gateway.conf".

>
> Thank you
> Frank
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Rename iscsi target_iqn

2017-11-17 Thread Frank Brendel 

Hi,

how can I rename an iscsi target_iqn?
And where is the configuration that I made with gwcli stored?


Thank you
Frank
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Is the 12.2.1 really stable? Anybody have production cluster with Luminous Bluestore?

2017-11-17 Thread Cassiano Pilipavicius 
I have a small cluster on 12.2.1, used only for storing VHDS on RBD and 
it is pretty stable so far. I've upgraded from jewel to luminous and the 
only thing that caused me instability right after the upgrade is that I 
was using JEMalloc for the OSDs and after converting the OSDs to 
bluestore they have started crashing... commenting out the line on 
/etc/sysconfig/ceph solved all the issues with my crashing OSDs.




Em 11/16/2017 9:55 PM, Linh Vu escreveu:


We have a small prod cluster with 12.2.1 and bluestore, running just 
cephfs, for HPC use. It's been in prod for about 7 weeks now, and 
pretty stable. 😊



*From:* ceph-users  on behalf of 
Eric Nelson 

*Sent:* Friday, 17 November 2017 7:29:54 AM
*To:* Ashley Merrick
*Cc:* ceph-users@lists.ceph.com
*Subject:* Re: [ceph-users] Is the 12.2.1 really stable? Anybody have 
production cluster with Luminous Bluestore?
We upgraded to it a few weeks ago in order to get some of the new 
indexing features, but have also had a few nasty bugs in the process 
(including this one) as we have been upgrading osds from filestore to 
bluestore. Currently these are isolated to our SSD cache tier so I've 
been evicting everything in hopes of removing the tier and getting a 
functional cluster again :-/


On Thu, Nov 16, 2017 at 6:33 AM, Ashley Merrick > wrote:


Currently experiencing a nasty bug
http://tracker.ceph.com/issues/21142


I would say wait a while for the next point release.

,Ashley

-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com
] On Behalf Of Jack
Sent: 16 November 2017 22:22
To: ceph-users@lists.ceph.com 
Subject: Re: [ceph-users] Is the 12.2.1 really stable? Anybody
have production cluster with Luminous Bluestore?

My cluster (55 OSDs) runs 12.2.x since the release, and bluestore
too All good so far

On 16/11/2017 15:14, Konstantin Shalygin wrote:
> Hi cephers.
> Some thoughts...
> At this time my cluster on Kraken 11.2.0 - works smooth with
FileStore
> and RBD only.
> I want upgrade to Luminous 12.2.1 and go to Bluestore because this
> cluster want grows double with new disks, so is best opportunity
> migrate to Bluestore.
>
> In ML I was found two problems:
> 1. Increased memory usage, should be fixed in upstream
>

(http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-October/021676.html

).
>
> 2. OSD drops and goes cluster offline
>

(http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-November/022494.html

).
> Don't know this Bluestore or FileStore OSD'.s.
>
> If the first case I can safely survive - hosts has enough memory
to go
> to Bluestore and with the growing I can wait until the next
stable release.
> That second case really scares me. As I understood clusters with
this
> problem for now not in production.
>
> By this point I have completed all the preparations for the
update and
> now I need to figure out whether I should update to 12.2.1 or
wait for
> the next stable release, because my cluster is in production and I
> can't fail. Or I can upgrade and use FileStore until next release,
> this is acceptable for me.
>
> Thanks.
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com





___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com