date:20171226

[ceph-users] How to monitor slow request?

2017-12-26 Thread shadow_lin

I am building a ceph moninter dashboard and I want to moniter how many slow 
requests are on each node. But I find the ceph.log some time only log like 
below:
2017-12-27 14:59:47.852396 mon.mnc000 mon.0 192.168.99.80:6789/0 2147 : cluster 
[WRN] Health check failed: 4 slow requests are blocked > 32 sec (REQUEST_SLOW)

There is no osd id info about where the slow reqeust happenened.

what would be a proper way to moniter which osd caused the slow request and how 
many slow requests are one that osd?



2017-12-27


shadow_lin ___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [luminous 12.2.2] Cluster write performance degradation problem(possibly tcmalloc related)

2017-12-26 Thread shadow_lin

I have disabled scrub before the test.

2017-12-27 

shadow_lin 



发件人：Webert de Souza Lima 
发送时间：2017-12-22 20:37
主题：Re: [ceph-users] [luminous 12.2.2] Cluster write performance degradation 
problem(possibly tcmalloc related)
收件人："ceph-users"
抄送：


On Thu, Dec 21, 2017 at 12:52 PM, shadow_lin  wrote:
After 18:00 suddenly the write throughput dropped and the osd latency 
increased. TCmalloc started relcaim page heap freelist much more frequently.All 
of this happened very fast and every osd had the indentical pattern.
Could that be caused by OSD scrub?  Check your "osd_scrub_begin_hour"

  ceph daemon osd.$ID config show | grep osd_scrub




Regards,


Webert Lima
DevOps Engineer at MAV Tecnologia
Belo Horizonte - Brasil
IRC NICK - WebertRLZ___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Bluestore: inaccurate disk usage statistics problem?

2017-12-26 Thread Zhi Zhang

Hi Sage,

Thanks for the quick reply. I read the code and our test also proved
that disk space was wasted due to min_alloc_size.

Very look forward to the "inline" data feature for small objects. We
will also look into this feature and hopefully work with community on
it.

Regards,
Zhi Zhang (David)
Contact: zhang.david2...@gmail.com
  zhangz.da...@outlook.com


On Wed, Dec 27, 2017 at 6:36 AM, Sage Weil  wrote:
> On Tue, 26 Dec 2017, Zhi Zhang wrote:
>> Hi,
>>
>> We recently started to test bluestore with huge amount of small files
>> (only dozens of bytes per file). We have 22 OSDs in a test cluster
>> using ceph-12.2.1 with 2 replicas and each OSD disk is 2TB size. After
>> we wrote about 150 million files through cephfs, we found each OSD
>> disk usage reported by "ceph osd df" was more than 40%, which meant
>> more than 800GB was used for each disk, but the actual total file size
>> was only about 5.2 GB, which was reported by "ceph df" and also
>> calculated by ourselves.
>>
>> The test is ongoing. I wonder whether the cluster would report OSD
>> full after we wrote about 300 million files, however the actual total
>> file size would be far far less than the disk usage. I will update the
>> result when the test is done.
>>
>> My question is, whether the disk usage statistics in bluestore is
>> inaccurate, or the padding, alignment stuff or something else in
>> bluestore wastes the disk space?
>
> Bluestore isn't making any attempt to optimize for small files, so a
> one byte file will consume min_alloc_size (64kb on HDD, 16kb on SSD,
> IIRC).
>
> It probably wouldn't be too difficult to add an "inline" data for small
> objects feature that puts small objects in rocksdb...
>
> sage
>
>>
>> Thanks!
>>
>> $ ceph osd df
>> ID CLASS WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE  VAR  PGS
>>  0   hdd 1.49728  1.0  1862G   853G  1009G 45.82 1.00 110
>>  1   hdd 1.69193  1.0  1862G   807G  1054G 43.37 0.94 105
>>  2   hdd 1.81929  1.0  1862G   811G  1051G 43.57 0.95 116
>>  3   hdd 2.00700  1.0  1862G   839G  1023G 45.04 0.98 122
>>  4   hdd 2.06334  1.0  1862G   886G   976G 47.58 1.03 130
>>  5   hdd 1.99051  1.0  1862G   856G  1006G 45.95 1.00 118
>>  6   hdd 1.67519  1.0  1862G   881G   981G 47.32 1.03 114
>>  7   hdd 1.81929  1.0  1862G   874G   988G 46.94 1.02 120
>>  8   hdd 2.08881  1.0  1862G   885G   976G 47.56 1.03 130
>>  9   hdd 1.64265  1.0  1862G   852G  1010G 45.78 0.99 106
>> 10   hdd 1.81929  1.0  1862G   873G   989G 46.88 1.02 109
>> 11   hdd 2.20041  1.0  1862G   915G   947G 49.13 1.07 131
>> 12   hdd 1.45694  1.0  1862G   874G   988G 46.94 1.02 110
>> 13   hdd 2.03847  1.0  1862G   821G  1041G 44.08 0.96 113
>> 14   hdd 1.53812  1.0  1862G   810G  1052G 43.50 0.95 112
>> 15   hdd 1.52914  1.0  1862G   874G   988G 46.94 1.02 111
>> 16   hdd 1.99176  1.0  1862G   810G  1052G 43.51 0.95 114
>> 17   hdd 1.81929  1.0  1862G   841G  1021G 45.16 0.98 119
>> 18   hdd 1.70901  1.0  1862G   831G  1031G 44.61 0.97 113
>> 19   hdd 1.67519  1.0  1862G   875G   987G 47.02 1.02 115
>> 20   hdd 2.03847  1.0  1862G   864G   998G 46.39 1.01 115
>> 21   hdd 2.18794  1.0  1862G   920G   942G 49.39 1.07 127
>> TOTAL 40984G 18861G 22122G 46.02
>>
>> $ ceph df
>> GLOBAL:
>> SIZE   AVAIL  RAW USED %RAW USED
>> 40984G 22122G   18861G 46.02
>> POOLS:
>> NAMEID USED  %USED MAX AVAIL OBJECTS
>> cephfs_metadata 5   160M 0 6964G 77342
>> cephfs_data 6  5193M  0.04 6964G 151292669
>>
>>
>> Regards,
>> Zhi Zhang (David)
>> Contact: zhang.david2...@gmail.com
>>   zhangz.da...@outlook.com
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] 答复: 答复: 答复: Can't delete file in cephfs with "No space left on device"

2017-12-26 Thread 周威

The client they are using is mainly fuse(10.2.9 and 0.94.9)

-邮件原件-
发件人: Yan, Zheng [mailto:uker...@gmail.com] 
发送时间: 2017年12月27日 10:32
收件人: 周 威 
抄送: Cary ; ceph-users@lists.ceph.com
主题: Re: [ceph-users] 答复: 答复: Can't delete file in cephfs with "No space left on 
device"

On Tue, Dec 26, 2017 at 2:28 PM, 周 威  wrote:
> We don't use hardlink.
> I reduced the mds_cache_size from 1000 to 200.
> After that, the num_strays reduce to about 100k The cluster is normal 
> now. I think there is some bug about it.
> Anyway, thanks for your reply!
>

This seems like a client bug. which client do you use (kclient or fuse, 
version)?


> -邮件原件-
> 发件人: Cary [mailto:dynamic.c...@gmail.com]
> 发送时间: 2017年12月26日 14:08
> 收件人: 周 威 
> 抄送: ceph-users@lists.ceph.com
> 主题: Re: 答复: [ceph-users] Can't delete file in cephfs with "No space left on 
> device"
>
> https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists
> .ceph.com%2Fpipermail%2Fceph-users-ceph.com%2F2016-October%2F013646.ht
> ml=02%7C01%7C%7C571893beb17d4d643a8e08d54c26fbed%7C84df9e7fe9f640
> afb435%7C1%7C0%7C636498652669125646=2Y9JT%2BoksSglve
> XPttiR7y3nwAmADhLwxTUoH4lxQAI%3D=0
>
> On Tue, Dec 26, 2017 at 6:07 AM, Cary  wrote:
>> Are you using hardlinks in cephfs?
>>
>>
>> On Tue, Dec 26, 2017 at 3:42 AM, 周 威  wrote:
>>> The out put of ceph osd df
>>>
>>>
>>>
>>> ID WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE  VAR  PGS
>>>
>>> 0 1.62650  1.0  1665G  1279G   386G 76.82 1.05 343
>>>
>>> 1 1.62650  1.0  1665G  1148G   516G 68.97 0.94 336
>>>
>>> 2 1.62650  1.0  1665G  1253G   411G 75.27 1.03 325
>>>
>>> 3 1.62650  1.0  1665G  1192G   472G 71.60 0.98 325
>>>
>>> 4 1.62650  1.0  1665G  1205G   460G 72.35 0.99 341
>>>
>>> 5 1.62650  1.0  1665G  1381G   283G 82.95 1.13 364
>>>
>>> 6 1.62650  1.0  1665G  1069G   595G 64.22 0.88 322
>>>
>>> 7 1.62650  1.0  1665G  1222G   443G 73.38 1.00 337
>>>
>>> 8 1.62650  1.0  1665G  1120G   544G 67.29 0.92 312
>>>
>>> 9 1.62650  1.0  1665G  1166G   498G 70.04 0.96 336
>>>
>>> 10 1.62650  1.0  1665G  1254G   411G 75.31 1.03 348
>>>
>>> 11 1.62650  1.0  1665G  1352G   313G 81.19 1.11 341
>>>
>>> 12 1.62650  1.0  1665G  1174G   490G 70.52 0.96 328
>>>
>>> 13 1.62650  1.0  1665G  1281G   383G 76.95 1.05 345
>>>
>>> 14 1.62650  1.0  1665G  1147G   518G 68.88 0.94 339
>>>
>>> 15 1.62650  1.0  1665G  1236G   429G 74.24 1.01 334
>>>
>>> 20 1.62650  1.0  1665G  1166G   499G 70.03 0.96 325
>>>
>>> 21 1.62650  1.0  1665G  1371G   293G 82.35 1.13 377
>>>
>>> 22 1.62650  1.0  1665G  1110G   555G 66.67 0.91 341
>>>
>>> 23 1.62650  1.0  1665G  1221G   443G 73.36 1.00 327
>>>
>>> 16 1.62650  1.0  1665G  1354G   310G 81.34 1.11 352
>>>
>>> 17 1.62650  1.0  1665G  1250G   415G 75.06 1.03 341
>>>
>>> 18 1.62650  1.0  1665G  1179G   486G 70.80 0.97 316
>>>
>>> 19 1.62650  1.0  1665G  1236G   428G 74.26 1.01 333
>>>
>>> 24 1.62650  1.0  1665G  1146G   518G 68.86 0.94 325
>>>
>>> 25 1.62650  1.0  1665G  1033G   632G 62.02 0.85 309
>>>
>>> 26 1.62650  1.0  1665G  1234G   431G 74.11 1.01 334
>>>
>>> 27 1.62650  1.0  1665G  1342G   322G 80.62 1.10 352
>>>
>>>   TOTAL 46635G 34135G 12500G 73.20
>>>
>>> MIN/MAX VAR: 0.85/1.13  STDDEV: 5.28
>>>
>>>
>>>
>>> 发件人: Cary [mailto:dynamic.c...@gmail.com]
>>> 发送时间: 2017年12月26日 11:40
>>> 收件人: ? ? 
>>> 抄送: ceph-users@lists.ceph.com
>>> 主题: Re: [ceph-users] Can't delete file in cephfs with "No space left 
>>> on device"
>>>
>>>
>>>
>>> Could you post the output of “ceph osd df”?
>>>
>>>
>>> On Dec 25, 2017, at 19:46, ? ?  wrote:
>>>
>>> Hi all:
>>>
>>>
>>>
>>> Ceph version:
>>> ceph version 10.2.9 (2ee413f77150c0f375ff6f10edd6c8f9c7d060d0)
>>>
>>>
>>>
>>> Ceph df:
>>>
>>> GLOBAL:
>>>
>>> SIZE   AVAIL  RAW USED %RAW USED
>>>
>>> 46635G 12500G   34135G 73.19
>>>
>>>
>>>
>>> rm d
>>>
>>> rm: cannot remove `d': No space left on device
>>>
>>>
>>>
>>> and mds_cache:
>>>
>>> {
>>>
>>> "mds_cache": {
>>>
>>> "num_strays": 999713,
>>>
>>> "num_strays_purging": 0,
>>>
>>> "num_strays_delayed": 0,
>>>
>>> "num_purge_ops": 0,
>>>
>>> "strays_created": 999723,
>>>
>>> "strays_purged": 10,
>>>
>>> "strays_reintegrated": 0,
>>>
>>> "strays_migrated": 0,
>>>
>>> "num_recovering_processing": 0,
>>>
>>> "num_recovering_enqueued": 0,
>>>
>>> "num_recovering_prioritized": 0,
>>>
>>> "recovery_started": 107,
>>>
>>> "recovery_completed": 107
>>>
>>> }
>>>
>>> }
>>>
>>>
>>>
>>> It seems starys num are stuck, what should I do?
>>>
>>> Thanks all.
>>>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>>

Re: [ceph-users] 答复: 答复: Can't delete file in cephfs with "No space left on device"

2017-12-26 Thread Yan, Zheng

On Tue, Dec 26, 2017 at 2:28 PM, 周 威  wrote:
> We don't use hardlink.
> I reduced the mds_cache_size from 1000 to 200.
> After that, the num_strays reduce to about 100k
> The cluster is normal now. I think there is some bug about it.
> Anyway, thanks for your reply!
>

This seems like a client bug. which client do you use (kclient or
fuse, version)?


> -邮件原件-
> 发件人: Cary [mailto:dynamic.c...@gmail.com]
> 发送时间: 2017年12月26日 14:08
> 收件人: 周 威 
> 抄送: ceph-users@lists.ceph.com
> 主题: Re: 答复: [ceph-users] Can't delete file in cephfs with "No space left on 
> device"
>
> https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ceph.com%2Fpipermail%2Fceph-users-ceph.com%2F2016-October%2F013646.html=02%7C01%7C%7C571893beb17d4d643a8e08d54c26fbed%7C84df9e7fe9f640afb435%7C1%7C0%7C636498652669125646=2Y9JT%2BoksSglveXPttiR7y3nwAmADhLwxTUoH4lxQAI%3D=0
>
> On Tue, Dec 26, 2017 at 6:07 AM, Cary  wrote:
>> Are you using hardlinks in cephfs?
>>
>>
>> On Tue, Dec 26, 2017 at 3:42 AM, 周 威  wrote:
>>> The out put of ceph osd df
>>>
>>>
>>>
>>> ID WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE  VAR  PGS
>>>
>>> 0 1.62650  1.0  1665G  1279G   386G 76.82 1.05 343
>>>
>>> 1 1.62650  1.0  1665G  1148G   516G 68.97 0.94 336
>>>
>>> 2 1.62650  1.0  1665G  1253G   411G 75.27 1.03 325
>>>
>>> 3 1.62650  1.0  1665G  1192G   472G 71.60 0.98 325
>>>
>>> 4 1.62650  1.0  1665G  1205G   460G 72.35 0.99 341
>>>
>>> 5 1.62650  1.0  1665G  1381G   283G 82.95 1.13 364
>>>
>>> 6 1.62650  1.0  1665G  1069G   595G 64.22 0.88 322
>>>
>>> 7 1.62650  1.0  1665G  1222G   443G 73.38 1.00 337
>>>
>>> 8 1.62650  1.0  1665G  1120G   544G 67.29 0.92 312
>>>
>>> 9 1.62650  1.0  1665G  1166G   498G 70.04 0.96 336
>>>
>>> 10 1.62650  1.0  1665G  1254G   411G 75.31 1.03 348
>>>
>>> 11 1.62650  1.0  1665G  1352G   313G 81.19 1.11 341
>>>
>>> 12 1.62650  1.0  1665G  1174G   490G 70.52 0.96 328
>>>
>>> 13 1.62650  1.0  1665G  1281G   383G 76.95 1.05 345
>>>
>>> 14 1.62650  1.0  1665G  1147G   518G 68.88 0.94 339
>>>
>>> 15 1.62650  1.0  1665G  1236G   429G 74.24 1.01 334
>>>
>>> 20 1.62650  1.0  1665G  1166G   499G 70.03 0.96 325
>>>
>>> 21 1.62650  1.0  1665G  1371G   293G 82.35 1.13 377
>>>
>>> 22 1.62650  1.0  1665G  1110G   555G 66.67 0.91 341
>>>
>>> 23 1.62650  1.0  1665G  1221G   443G 73.36 1.00 327
>>>
>>> 16 1.62650  1.0  1665G  1354G   310G 81.34 1.11 352
>>>
>>> 17 1.62650  1.0  1665G  1250G   415G 75.06 1.03 341
>>>
>>> 18 1.62650  1.0  1665G  1179G   486G 70.80 0.97 316
>>>
>>> 19 1.62650  1.0  1665G  1236G   428G 74.26 1.01 333
>>>
>>> 24 1.62650  1.0  1665G  1146G   518G 68.86 0.94 325
>>>
>>> 25 1.62650  1.0  1665G  1033G   632G 62.02 0.85 309
>>>
>>> 26 1.62650  1.0  1665G  1234G   431G 74.11 1.01 334
>>>
>>> 27 1.62650  1.0  1665G  1342G   322G 80.62 1.10 352
>>>
>>>   TOTAL 46635G 34135G 12500G 73.20
>>>
>>> MIN/MAX VAR: 0.85/1.13  STDDEV: 5.28
>>>
>>>
>>>
>>> 发件人: Cary [mailto:dynamic.c...@gmail.com]
>>> 发送时间: 2017年12月26日 11:40
>>> 收件人: ? ? 
>>> 抄送: ceph-users@lists.ceph.com
>>> 主题: Re: [ceph-users] Can't delete file in cephfs with "No space left
>>> on device"
>>>
>>>
>>>
>>> Could you post the output of “ceph osd df”?
>>>
>>>
>>> On Dec 25, 2017, at 19:46, ? ?  wrote:
>>>
>>> Hi all:
>>>
>>>
>>>
>>> Ceph version:
>>> ceph version 10.2.9 (2ee413f77150c0f375ff6f10edd6c8f9c7d060d0)
>>>
>>>
>>>
>>> Ceph df:
>>>
>>> GLOBAL:
>>>
>>> SIZE   AVAIL  RAW USED %RAW USED
>>>
>>> 46635G 12500G   34135G 73.19
>>>
>>>
>>>
>>> rm d
>>>
>>> rm: cannot remove `d': No space left on device
>>>
>>>
>>>
>>> and mds_cache:
>>>
>>> {
>>>
>>> "mds_cache": {
>>>
>>> "num_strays": 999713,
>>>
>>> "num_strays_purging": 0,
>>>
>>> "num_strays_delayed": 0,
>>>
>>> "num_purge_ops": 0,
>>>
>>> "strays_created": 999723,
>>>
>>> "strays_purged": 10,
>>>
>>> "strays_reintegrated": 0,
>>>
>>> "strays_migrated": 0,
>>>
>>> "num_recovering_processing": 0,
>>>
>>> "num_recovering_enqueued": 0,
>>>
>>> "num_recovering_prioritized": 0,
>>>
>>> "recovery_started": 107,
>>>
>>> "recovery_completed": 107
>>>
>>> }
>>>
>>> }
>>>
>>>
>>>
>>> It seems starys num are stuck, what should I do?
>>>
>>> Thanks all.
>>>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Flist
>>> s.ceph.com%2Flistinfo.cgi%2Fceph-users-ceph.com=02%7C01%7C%7C571
>>> 893beb17d4d643a8e08d54c26fbed%7C84df9e7fe9f640afb435%7C1%
>>> 7C0%7C636498652669125646=%2BUv%2BbpXBv%2B4INeEyjNLML1e%2BEjPMVH
>>> k%2FD5iU8a7a1DA%3D=0
> ___
>

Re: [ceph-users] Bluestore: inaccurate disk usage statistics problem?

2017-12-26 Thread Sage Weil

On Tue, 26 Dec 2017, Zhi Zhang wrote:
> Hi,
> 
> We recently started to test bluestore with huge amount of small files
> (only dozens of bytes per file). We have 22 OSDs in a test cluster
> using ceph-12.2.1 with 2 replicas and each OSD disk is 2TB size. After
> we wrote about 150 million files through cephfs, we found each OSD
> disk usage reported by "ceph osd df" was more than 40%, which meant
> more than 800GB was used for each disk, but the actual total file size
> was only about 5.2 GB, which was reported by "ceph df" and also
> calculated by ourselves.
> 
> The test is ongoing. I wonder whether the cluster would report OSD
> full after we wrote about 300 million files, however the actual total
> file size would be far far less than the disk usage. I will update the
> result when the test is done.
> 
> My question is, whether the disk usage statistics in bluestore is
> inaccurate, or the padding, alignment stuff or something else in
> bluestore wastes the disk space?

Bluestore isn't making any attempt to optimize for small files, so a 
one byte file will consume min_alloc_size (64kb on HDD, 16kb on SSD, 
IIRC).

It probably wouldn't be too difficult to add an "inline" data for small 
objects feature that puts small objects in rocksdb...

sage

> 
> Thanks!
> 
> $ ceph osd df
> ID CLASS WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE  VAR  PGS
>  0   hdd 1.49728  1.0  1862G   853G  1009G 45.82 1.00 110
>  1   hdd 1.69193  1.0  1862G   807G  1054G 43.37 0.94 105
>  2   hdd 1.81929  1.0  1862G   811G  1051G 43.57 0.95 116
>  3   hdd 2.00700  1.0  1862G   839G  1023G 45.04 0.98 122
>  4   hdd 2.06334  1.0  1862G   886G   976G 47.58 1.03 130
>  5   hdd 1.99051  1.0  1862G   856G  1006G 45.95 1.00 118
>  6   hdd 1.67519  1.0  1862G   881G   981G 47.32 1.03 114
>  7   hdd 1.81929  1.0  1862G   874G   988G 46.94 1.02 120
>  8   hdd 2.08881  1.0  1862G   885G   976G 47.56 1.03 130
>  9   hdd 1.64265  1.0  1862G   852G  1010G 45.78 0.99 106
> 10   hdd 1.81929  1.0  1862G   873G   989G 46.88 1.02 109
> 11   hdd 2.20041  1.0  1862G   915G   947G 49.13 1.07 131
> 12   hdd 1.45694  1.0  1862G   874G   988G 46.94 1.02 110
> 13   hdd 2.03847  1.0  1862G   821G  1041G 44.08 0.96 113
> 14   hdd 1.53812  1.0  1862G   810G  1052G 43.50 0.95 112
> 15   hdd 1.52914  1.0  1862G   874G   988G 46.94 1.02 111
> 16   hdd 1.99176  1.0  1862G   810G  1052G 43.51 0.95 114
> 17   hdd 1.81929  1.0  1862G   841G  1021G 45.16 0.98 119
> 18   hdd 1.70901  1.0  1862G   831G  1031G 44.61 0.97 113
> 19   hdd 1.67519  1.0  1862G   875G   987G 47.02 1.02 115
> 20   hdd 2.03847  1.0  1862G   864G   998G 46.39 1.01 115
> 21   hdd 2.18794  1.0  1862G   920G   942G 49.39 1.07 127
> TOTAL 40984G 18861G 22122G 46.02
> 
> $ ceph df
> GLOBAL:
> SIZE   AVAIL  RAW USED %RAW USED
> 40984G 22122G   18861G 46.02
> POOLS:
> NAMEID USED  %USED MAX AVAIL OBJECTS
> cephfs_metadata 5   160M 0 6964G 77342
> cephfs_data 6  5193M  0.04 6964G 151292669
> 
> 
> Regards,
> Zhi Zhang (David)
> Contact: zhang.david2...@gmail.com
>   zhangz.da...@outlook.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cache tiering on Erasure coded pools

2017-12-26 Thread David Turner

Please use the version of the docs for your installed version of ceph.  Now
the Jewel in your URL and the Luminous in mine.  In Luminous you no longer
need a cache tier to use EC with RBDs.

http://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/

On Tue, Dec 26, 2017, 4:21 PM Karun Josy  wrote:

> Hi,
>
> We are using Erasure coded pools in a ceph cluster for RBD images.
> Ceph version is 12.2.2 Luminous.
>
> -
> http://docs.ceph.com/docs/jewel/rados/operations/cache-tiering/
> -
>
> Here it says we can use a Cache tiering infront of ec pools.
> To use erasure code with RBD we  have a replicated pool to store metadata
> and  ecpool as data pool .
>
> Is it possible to setup cache tiering since there is already a replicated
> pool that is being used ?
>
>
>
>
>
>
>
>
>
>
> Karun Josy
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Cache tiering on Erasure coded pools

2017-12-26 Thread Karun Josy

Hi,

We are using Erasure coded pools in a ceph cluster for RBD images.
Ceph version is 12.2.2 Luminous.

-
http://docs.ceph.com/docs/jewel/rados/operations/cache-tiering/
-

Here it says we can use a Cache tiering infront of ec pools.
To use erasure code with RBD we  have a replicated pool to store metadata
and  ecpool as data pool .

Is it possible to setup cache tiering since there is already a replicated
pool that is being used ?










Karun Josy
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] slow 4k writes, Luminous with bluestore backend

2017-12-26 Thread kevin parrikar

Hi All,
I upgraded my cluster from Hammer to Jewel and then to Luminous , changed
from filestore to bluestore backend.

on a KVM vm with 4 cpu /2 Gb RAM i have attached a 20gb rbd volume as vdc
and performed following test.

dd if=/dev/zero of=/dev/vdc bs=4k count=1000 oflag=direct
1000+0 records in
1000+0 records out
4096000 bytes (4.1 MB) copied, 3.08965 s, *1.3 MB/s*

and its consistently giving 1.3MB/s which i feel is too low.I have 3 ceph
osd nodes each with 24 x15k RPM with a replication of 2 ,connected 2x10G
LACP bonded NICs with an MTU of 9100.

Rados Bench results:

rados bench -p volumes 4 write
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size
4194304 for up to 4 seconds or 0 objects
Object prefix: benchmark_data_ceph3.sapiennetworks.com_820994
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg
lat(s)
0   0 0 0 0 0   -
0
1  16   276   260   1039.98  1040   0.0165053
0.0381299
2  16   545   529   1057.92  10760.043151
0.0580376
3  16   847   831   1107.91  1208   0.0394811
0.0567684
4  16  1160  11441143.9  1252 0.63265
0.0541888
Total time run: 4.099801
Total writes made:  1161
Write size: 4194304
Object size:4194304
Bandwidth (MB/sec): 1132.74
Stddev Bandwidth:   101.98
Max bandwidth (MB/sec): 1252
Min bandwidth (MB/sec): 1040
Average IOPS:   283
Stddev IOPS:25
Max IOPS:   313
Min IOPS:   260
Average Latency(s): 0.0560897
Stddev Latency(s):  0.107352
Max latency(s): 1.02123
Min latency(s): 0.00920514
Cleaning up (deleting benchmark objects)
Removed 1161 objects
Clean up completed and total clean up time :0.079850


After upgrading to Luminous i have executed

ceph osd crush tunables optimal

ceph.conf

[global]
fsid = 06c5c906-fc43-499f-8a6f-6c8e21807acf
mon_initial_members = node-16 node-30 node-31
mon_host = 172.16.1.9 172.16.1.3 172.16.1.11
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
log_to_syslog_level = info
log_to_syslog = True
osd_pool_default_size = 2
osd_pool_default_min_size = 1
osd_pool_default_pg_num = 64
public_network = 172.16.1.0/24
log_to_syslog_facility = LOG_LOCAL0
osd_journal_size = 2048
auth_supported = cephx
osd_pool_default_pgp_num = 64
osd_mkfs_type = xfs
cluster_network = 172.16.1.0/24
osd_recovery_max_active = 1
osd_max_backfills = 1
max_open_files = 131072
debug_default = False


[client]
rbd_cache_writethrough_until_flush = True
rbd_cache = True

[client.radosgw.gateway]
rgw_keystone_accepted_roles = _member_, Member, admin, swiftoperator
keyring = /etc/ceph/keyring.radosgw.gateway
rgw_frontends = fastcgi socket_port=9000 socket_host=127.0.0.1
rgw_socket_path = /tmp/radosgw.sock
rgw_keystone_revocation_interval = 100
rgw_keystone_url = http://192.168.1.3:35357
rgw_keystone_admin_token = jaJSmlTNxgsFp1ttq5SuAT1R
rgw_init_timeout = 36
host = controller2
rgw_dns_name = *.sapiennetworks.com
rgw_print_continue = True
rgw_keystone_token_cache_size = 10
rgw_data = /var/lib/ceph/radosgw
user = www-data

[osd]
journal_queue_max_ops = 3000
objecter_inflight_ops = 10240
journal_queue_max_bytes = 1048576000
filestore_queue_max_ops = 500
osd_mkfs_type = xfs
osd_mount_options_xfs = rw,relatime,inode64,logbsize=256k,allocsize=4M
osd_op_threads = 20
filestore_queue_committing_max_ops = 5000
journal_max_write_entries = 1000
objecter_infilght_op_bytes = 1048576000
filestore_queue_max_bytes = 1048576000
filestore_max_sync_interval = 10
journal_max_write_bytes = 1048576000
filestore_queue_committing_max_bytes = 1048576000
ms_dispatch_throttle_bytes = 1048576000

 ceph -s
  cluster:
id: 06c5c906-fc43-499f-8a6f-6c8e21807acf
health: HEALTH_WARN
application not enabled on 2 pool(s)

  services:
mon: 3 daemons, quorum controller3,controller2,controller1
mgr: controller1(active)
osd: 72 osds: 72 up, 72 in
rgw: 1 daemon active

  data:
pools:   5 pools, 6240 pgs
objects: 12732 objects, 72319 MB
usage:   229 GB used, 39965 GB / 40195 GB avail
pgs: 6240 active+clean

can some one suggest a way to improve this.

Thanks,

Kevin
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph status doesnt show available and used disk space after upgrade

2017-12-26 Thread kevin parrikar

It was a firewall issue on the controller nodes.After allowing ceph-mgr
port in iptables everything is displaying correctly.Thanks to people on
IRC.

Thanks alot,
Kevin

On Thu, Dec 21, 2017 at 5:24 PM, kevin parrikar 
wrote:

> accidently removed mailing list email
>
> ++ceph-users
>
> Thanks a lot JC for looking into this issue. I am really out of ideas.
>
>
> ceph.conf on mgr node which is also monitor node.
>
> [global]
> fsid = 06c5c906-fc43-499f-8a6f-6c8e21807acf
> mon_initial_members = node-16 node-30 node-31
> mon_host = 172.16.1.9 172.16.1.3 172.16.1.11
> auth_cluster_required = cephx
> auth_service_required = cephx
> auth_client_required = cephx
> filestore_xattr_use_omap = true
> log_to_syslog_level = info
> log_to_syslog = True
> osd_pool_default_size = 2
> osd_pool_default_min_size = 1
> osd_pool_default_pg_num = 64
> public_network = 172.16.1.0/24
> log_to_syslog_facility = LOG_LOCAL0
> osd_journal_size = 2048
> auth_supported = cephx
> osd_pool_default_pgp_num = 64
> osd_mkfs_type = xfs
> cluster_network = 172.16.1.0/24
> osd_recovery_max_active = 1
> osd_max_backfills = 1
> mon allow pool delete = true
>
> [client]
> rbd_cache_writethrough_until_flush = True
> rbd_cache = True
>
> [client.radosgw.gateway]
> rgw_keystone_accepted_roles = _member_, Member, admin, swiftoperator
> keyring = /etc/ceph/keyring.radosgw.gateway
> rgw_frontends = fastcgi socket_port=9000 socket_host=127.0.0.1
> rgw_socket_path = /tmp/radosgw.sock
> rgw_keystone_revocation_interval = 100
> rgw_keystone_url = http://192.168.1.3:35357
> rgw_keystone_admin_token = jaJSmlTNxgsFp1ttq5SuAT1R
> rgw_init_timeout = 36
> host = controller3
> rgw_dns_name = *.sapiennetworks.com
> rgw_print_continue = True
> rgw_keystone_token_cache_size = 10
> rgw_data = /var/lib/ceph/radosgw
> user = www-data
>
>
>
>
> ceph auth list
>
>
> osd.100
> key: AQAtZjpaVZOFBxAAwl0yFLdUOidLzPFjv+HnjA==
> caps: [mgr] allow profile osd
> caps: [mon] allow profile osd
> caps: [osd] allow *
> osd.101
> key: AQA4ZjpaS4wwGBAABwgoXQRc1J8sav4MUkWceQ==
> caps: [mgr] allow profile osd
> caps: [mon] allow profile osd
> caps: [osd] allow *
> osd.102
> key: AQBDZjpaBS2tEBAAtFiPKBzh8JGi8Nh3PtAGCg==
> caps: [mgr] allow profile osd
> caps: [mon] allow profile osd
> caps: [osd] allow *
>
> client.admin
> key: AQD0yXFYflnYFxAAEz/2XLHO/6RiRXQ5HXRAnw==
> caps: [mds] allow *
> caps: [mgr] allow *
> caps: [mon] allow *
> caps: [osd] allow *
> client.backups
> key: AQC0y3FY4YQNNhAAs5fludq0yvtp/JJt7RT4HA==
> caps: [mgr] allow r
> caps: [mon] allow r
> caps: [osd] allow class-read object_prefix rbd_children, allow rwx
> pool=backups, allow rwx pool=volumes
> client.bootstrap-mds
> key: AQD5yXFYyIxiFxAAyoqLPnxxqWmUr+zz7S+qVQ==
> caps: [mgr] allow r
> caps: [mon] allow profile bootstrap-mds
> client.bootstrap-mgr
> key: AQBmOTpaXqHQDhAAyDXoxlPmG9QovfmmUd8gIg==
> caps: [mon] allow profile bootstrap-mgr
> client.bootstrap-osd
> key: AQD0yXFYuGkSIhAAelSb3TCPuXRFoFJTBh7Vdg==
> caps: [mgr] allow r
> caps: [mon] allow profile bootstrap-osd
> client.bootstrap-rbd
> key: AQBnOTpafDS/IRAAnKzuI9AYEF81/6mDVv0QgQ==
> caps: [mon] allow profile bootstrap-rbd
>
> client.bootstrap-rgw
> key: AQD3yXFYxt1mLRAArxOgRvWmmzT9pmsqTLpXKw==
> caps: [mgr] allow r
> caps: [mon] allow profile bootstrap-rgw
> client.compute
> key: AQCbynFYRcNWOBAAPzdAKfP21GvGz1VoHBimGQ==
> caps: [mgr] allow r
> caps: [mon] allow r
> caps: [osd] allow class-read object_prefix rbd_children, allow rwx
> pool=volumes, allow rx pool=images, allow rwx pool=compute
> client.images
> key: AQCyy3FYSMtlJRAAbJ8/U/R82NXvWBC5LmkPGw==
> caps: [mgr] allow r
> caps: [mon] allow r
> caps: [osd] allow class-read object_prefix rbd_children, allow rwx
> pool=images
> client.radosgw.gateway
> key: AQA3ynFYAYMSAxAApvfe/booa9KhigpKpLpUOA==
> caps: [mgr] allow r
> caps: [mon] allow rw
> caps: [osd] allow rwx
> client.volumes
> key: AQCzy3FYa3paKBAA9BlYpQ1PTeR770ghVv1jKQ==
> caps: [mgr] allow r
> caps: [mon] allow r
> caps: [osd] allow class-read object_prefix rbd_children, allow rwx
> pool=volumes, allow rx pool=images
> mgr.controller2
> key: AQAmVTpaA+9vBhAApD3rMs//Qri+SawjUF4U4Q==
> caps: [mds] allow *
> caps: [mgr] allow *
> caps: [mon] allow *
> caps: [osd] allow *
> mgr.controller3
> key: AQByfDparprIEBAAj7Pxdr/87/v0kmJV49aKpQ==
> caps: [mds] allow *
> caps: [mgr] allow *
> caps: [mon] allow *
> caps: [osd] allow *
>
> Regards,
> Kevin
>
> On Thu, Dec 21, 2017 at 8:10 AM, kevin parrikar  > wrote:
>
>>

Re: [ceph-users] Ceph as an Alternative to HDFS for Hadoop

2017-12-26 Thread Aristeu Gil Alves Jr

In a recent thread on the list, I received various important answers to my
questions on hadoop plugin. Maybe this thread will help you.
https://www.spinics.net/lists/ceph-users/msg40790.html

One of the most important answers is about data locality. The last message
lead me to this article.
https://www.bluedata.com/blog/2015/05/data-locality-is-irrelevant-for-hadoop/

Regards,
--
Aristeu

2017-12-22 2:04 GMT-02:00 Serkan Çoban :

> >Also, are there any benchmark comparisons between hdfs and ceph
> specifically around performance of apps benefiting from data locality ?
> There will be no data locality in ceph, because all the data is
> accessed through network.
>
> On Fri, Dec 22, 2017 at 4:52 AM, Traiano Welcome 
> wrote:
> > Hi List
> >
> > I'm researching the possibility os using ceph as a drop in replacement
> for
> > hdfs for applications using spark and hadoop.
> >
> > I note that the jewel documentation states that it requires hadoop 1.1.x,
> > which seems a little dated and would be of concern for peopel:
> >
> > http://docs.ceph.com/docs/jewel/cephfs/hadoop/
> >
> > What about the 2.x series?
> >
> > Also, are there any benchmark comparisons between hdfs and ceph
> specifically
> > around performance of apps benefiting from data locality ?
> >
> > Many thanks in advance for any feedback!
> >
> > Regards,
> > Traiano
> >
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How to evict a client in rbd

2017-12-26 Thread Hamid EDDIMA


Hello

try : ceph osd blacklist add 10.255.0.17:0/3495340192 



Hamid.

Le 26/12/2017 à 12:16, Karun Josy a écrit :

Any help is really appreciated.

Karun Josy

On Sun, Dec 24, 2017 at 2:18 AM, Karun Josy > wrote:


Hello,

The image is not mapped.

# ceph --version
ceph version 12.2.1  luminous (stable)
# uname -r
4.14.0-1.el7.elrepo.x86_64


Karun Josy

On Sat, Dec 23, 2017 at 6:51 PM, Jason Dillaman
> wrote:

What Ceph and what kernel version are you using? Are you
positive that
the image has been unmapped from 10.255.0.17?

On Fri, Dec 22, 2017 at 7:14 PM, Karun Josy
> wrote:
> Hello,
>
> I am unable to delete this abandoned image.Rbd info shows a
watcher ip
> Image is not mapped
> Image has no snapshots
>
>
> rbd status cvm/image  --id clientuser
> Watchers:
> watcher=10.255.0.17:0/3495340192
 client.390908
> cookie=18446462598732841114
>
> How can I  evict or black list a watcher client so that
image can be deleted
> http://docs.ceph.com/docs/master/cephfs/eviction/

> I see this is possible in Cephfs
>
>
>
> Karun
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

>



--
Jason





___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How to evict a client in rbd

2017-12-26 Thread Karun Josy

Any help is really appreciated.

Karun Josy

On Sun, Dec 24, 2017 at 2:18 AM, Karun Josy  wrote:

> Hello,
>
> The image is not mapped.
>
> # ceph --version
> ceph version 12.2.1  luminous (stable)
> # uname -r
> 4.14.0-1.el7.elrepo.x86_64
>
>
> Karun Josy
>
> On Sat, Dec 23, 2017 at 6:51 PM, Jason Dillaman 
> wrote:
>
>> What Ceph and what kernel version are you using? Are you positive that
>> the image has been unmapped from 10.255.0.17?
>>
>> On Fri, Dec 22, 2017 at 7:14 PM, Karun Josy  wrote:
>> > Hello,
>> >
>> > I am unable to delete this abandoned image.Rbd info shows a watcher ip
>> > Image is not mapped
>> > Image has no snapshots
>> >
>> >
>> > rbd status cvm/image  --id clientuser
>> > Watchers:
>> > watcher=10.255.0.17:0/3495340192 client.390908
>> > cookie=18446462598732841114
>> >
>> > How can I  evict or black list a watcher client so that image can be
>> deleted
>> > http://docs.ceph.com/docs/master/cephfs/eviction/
>> > I see this is possible in Cephfs
>> >
>> >
>> >
>> > Karun
>> >
>> > ___
>> > ceph-users mailing list
>> > ceph-users@lists.ceph.com
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >
>>
>>
>>
>> --
>> Jason
>>
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] pass through commands via ceph-mgr restful plugin's request endpoint

2017-12-26 Thread zhenhua.zhang

Hi, all
By the comment of ceph-mgr restful plugin's /request post method, it should 
take ceph commands and fetch result back. However, I am having trouble of 
writing a curl example. Running Ceph 12.2.2 and havn't found any documentation 
on posting to this endpoint. Can someone shed some light on it?

Below example produces ValueError in log.curl --request POST --silent 
--insecure --user testuser:0244dc6e-0fcb-4b89-bf21-571de094a7a6  
https://192.168.52.132:8003/request -d "{'prefix': 'status'}"
ceph-mgr's error message:2017-12-26 05:08:20.200251 7fd496c20700  0 
mgr[restful] Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pecan/core.py", line 570, in __call__
    self.handle_request(req, resp)
  File "/usr/lib/python2.7/site-packages/pecan/core.py", line 508, in 
handle_request
    result = controller(*args, **kwargs)
  File "/usr/lib64/ceph/mgr/restful/decorators.py", line 33, in decorated
    return f(*args, **kwargs)
  File "/usr/lib64/ceph/mgr/restful/api/request.py", line 87, in post
    return module.instance.submit_request([[request.json]], **kwargs)
  File "/usr/lib/python2.7/site-packages/pecan/core.py", line 35, in __getattr__
    return getattr(obj, attr)
  File "/usr/lib/python2.7/site-packages/webob/request.py", line 701, in 
_json_body__get
    return json.loads(self.body.decode(self.charset))
  File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

Any help is much appreciated.Thanks,Zhenhua

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] rbd map failed when ms_public_type=async+rdma

2017-12-26 Thread Yang, Liang

Hi all,

Rbd map will fail when ms_public_type=async+rdma, and network of ceph cluster 
is blocked.  Is this cause by that rbd in kernel does not support rdma?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] How to monitor slow request?

Re: [ceph-users] [luminous 12.2.2] Cluster write performance degradation problem(possibly tcmalloc related)

Re: [ceph-users] Bluestore: inaccurate disk usage statistics problem?

[ceph-users] 答复: 答复: 答复: Can't delete file in cephfs with "No space left on device"

Re: [ceph-users] 答复: 答复: Can't delete file in cephfs with "No space left on device"

Re: [ceph-users] Bluestore: inaccurate disk usage statistics problem?

Re: [ceph-users] Cache tiering on Erasure coded pools

[ceph-users] Cache tiering on Erasure coded pools

[ceph-users] slow 4k writes, Luminous with bluestore backend

Re: [ceph-users] ceph status doesnt show available and used disk space after upgrade

Re: [ceph-users] Ceph as an Alternative to HDFS for Hadoop

Re: [ceph-users] How to evict a client in rbd

Re: [ceph-users] How to evict a client in rbd

[ceph-users] pass through commands via ceph-mgr restful plugin's request endpoint

[ceph-users] rbd map failed when ms_public_type=async+rdma

15 matches

Site Navigation

Mail list logo

Footer information