Re: [ceph-users] Cephfs I/O when no I/O operations are submitted

[xiafei]

Hi Chirstian, 

My configuration is as follows:
Two nodes: node0 (10.10.0.23), node1(10.10.0.24)
node0: Monitor, MDS, OSD0 in /dev/sdd1
node1: OSD1 in /dev/sdd1
The cephfs is mounted using:  mount.ceph 10.10.0.23:6789:/  /mnt/mycephfs/
After finishing configuration (PGs are active+clean), I use the iostat cmd to 
monitor the /dev/sdd1 in node0. 
The iostat shows that there are write I/Os for about 15minutes. Then there are 
no I/Os in the OSDs. 
I am not sure where these write I/Os come from. 
Thanks.

Best regards,
Fei Xia



> 在 2015年12月14日，11:58，Christian Balzer  写道：
> 
> 
> Hello,
> 
> On Mon, 14 Dec 2015 11:46:43 +0800 xiafei wrote:
> 
>> Hi all,
>>  I have a question about the I/O of cephfs.
>> I configure cephfs with 2 OSDs, and 300PGs in two HDDs. Then I use the
>> iostat (iostat -kx 1 /dev/sdd1) to monitor the I/Os of the HDDs
>> (/dev/sdd1). The result is as follows:
>> 
>> 
>> The iostat shows that there are write requests every second. However I
>> do not execute any applications. What the source of the write I/O? 
>> 
> That's why you use atop and/or iotop to determine what program is writing
> to the disks.
> 
> If it is indeed Ceph (ceph-osd process) writing to the disk then the
> question is indeed what causes that. 
> I have no experience with CephFS, but quiescent RBD volume (mounted VM
> images) does not produce I/O by itself.
> 
> You say that you're not running any application, but is the CephFS mounted
> somewhere?
> 
> Regards,
> 
> Christian
> -- 
> Christian BalzerNetwork/Systems Engineer
> ch...@gol.com Global OnLine Japan/Rakuten Communications
> http://www.gol.com/

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] All pgs stuck peering

[Chris Dunlop]

On Sun, Dec 13, 2015 at 09:10:34PM -0700, Robert LeBlanc wrote:
> I've had something similar to this when there was an MTU mismatch, the
> smaller I/O would get through, but the larger I/O would be blocked and
> prevent peering.
> 
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1


AUGHHH!

That was it.

Thanks Robert and Varada!

Cheers,

Chris
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] All pgs stuck peering

[Robert LeBlanc]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

I've had something similar to this when there was an MTU mismatch, the
smaller I/O would get through, but the larger I/O would be blocked and
prevent peering.
- 
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1


On Sun, Dec 13, 2015 at 9:01 PM, Chris Dunlop  wrote:
> Hi Varada,
>
> On Mon, Dec 14, 2015 at 03:23:20AM +, Varada Kari wrote:
>> Can get the details of
>>
>> 1. ceph health detail
>> 2. ceph pg query
>>
>> of any one PG stuck peering
>>
>>
>> Varada
>
> The full health detail is over 9000 lines, but here's a summary:
>
> # ceph health detail | head
> HEALTH_WARN 3072 pgs peering; 3072 pgs stuck inactive; 3072 pgs stuck 
> unclean; 1570 requests are blocked > 32 sec; 25 osds have slow requests; 
> noout flag(s) set
> pg 3.1ae is stuck inactive for 23264.342056, current state peering, last 
> acting [16,4,8]
> pg 2.1af is stuck inactive for 23621.565024, current state peering, last 
> acting [6,0]
> pg 6.1ab is stuck inactive for 22843.875498, current state peering, last 
> acting [27,18,54]
> pg 3.1af is stuck inactive for 23315.971276, current state peering, last 
> acting [17,16,24]
> pg 2.1ae is stuck inactive for 19278.004657, current state peering, last 
> acting [7,1]
> pg 6.1aa is stuck inactive for 19321.668092, current state peering, last 
> acting [31,39,56]
> pg 3.1a8 is stuck inactive for 22897.969982, current state peering, last 
> acting [16,17,24]
> pg 2.1a9 is stuck inactive for 23516.554757, current state peering, last 
> acting [14,7]
> pg 6.1ad is stuck inactive for 23105.915508, current state peering, last 
> acting [33,47,20]
> # ceph health detail | grep -v peering
> 34 ops are blocked > 16777.2 sec
> 1289 ops are blocked > 8388.61 sec
> 50 ops are blocked > 4194.3 sec
> 34 ops are blocked > 2097.15 sec
> 68 ops are blocked > 1048.58 sec
> 13 ops are blocked > 524.288 sec
> 11 ops are blocked > 16777.2 sec on osd.0
> 4 ops are blocked > 8388.61 sec on osd.0
> 5 ops are blocked > 8388.61 sec on osd.1
> 100 ops are blocked > 8388.61 sec on osd.2
> 100 ops are blocked > 8388.61 sec on osd.3
> 100 ops are blocked > 8388.61 sec on osd.4
> 80 ops are blocked > 8388.61 sec on osd.5
> 34 ops are blocked > 8388.61 sec on osd.6
> 27 ops are blocked > 4194.3 sec on osd.6
> 15 ops are blocked > 2097.15 sec on osd.6
> 6 ops are blocked > 1048.58 sec on osd.6
> 9 ops are blocked > 524.288 sec on osd.6
> 2 ops are blocked > 16777.2 sec on osd.7
> 20 ops are blocked > 4194.3 sec on osd.7
> 16 ops are blocked > 2097.15 sec on osd.7
> 62 ops are blocked > 1048.58 sec on osd.7
> 85 ops are blocked > 8388.61 sec on osd.8
> 80 ops are blocked > 8388.61 sec on osd.9
> 13 ops are blocked > 16777.2 sec on osd.10
> 3 ops are blocked > 8388.61 sec on osd.10
> 1 ops are blocked > 4194.3 sec on osd.10
> 1 ops are blocked > 2097.15 sec on osd.10
> 6 ops are blocked > 8388.61 sec on osd.11
> 5 ops are blocked > 8388.61 sec on osd.12
> 4 ops are blocked > 8388.61 sec on osd.13
> 2 ops are blocked > 8388.61 sec on osd.14
> 4 ops are blocked > 524.288 sec on osd.14
> 7 ops are blocked > 16777.2 sec on osd.15
> 12 ops are blocked > 8388.61 sec on osd.15
> 2 ops are blocked > 4194.3 sec on osd.15
> 2 ops are blocked > 2097.15 sec on osd.15
> 100 ops are blocked > 8388.61 sec on osd.16
> 82 ops are blocked > 8388.61 sec on osd.17
> 1 ops are blocked > 16777.2 sec on osd.18
> 100 ops are blocked > 8388.61 sec on osd.21
> 86 ops are blocked > 8388.61 sec on osd.24
> 100 ops are blocked > 8388.61 sec on osd.38
> 100 ops are blocked > 8388.61 sec on osd.42
> 100 ops are blocked > 8388.61 sec on osd.44
> 1 ops are blocked > 8388.61 sec on osd.51
> 25 osds have slow requests
> noout flag(s) set
>
>
> # ceph pg 3.1ae query
> <<< hung, until ^c >>>
> # ceph pg 2.1af query
> {
> "state": "peering",
> "snap_trimq": "[]",
> "epoch": 357236,
> "up": [
> 6,
> 0
> ],
> "acting": [
> 6,
> 0
> ],
> "info": {
> "pgid": "2.1af",
> "last_update": "356361'1923761",
> "last_complete": "356361'1923761",
> "log_tail": "341349'1920757",
> "last_user_version": 1923761,
> "last_backfill": "MAX",
> "purged_snaps": "[1~34,38~1b,55~2,59~2a,84~68,ee~62]",
> "history": {
> "epoch_created": 1,
> "last_epoch_started": 356496,
> "last_epoch_clean": 356496,
> "last_epoch_split": 0,
> "same_up_since": 357218,
> "same_interval_since": 357218,
> "same_primary_since": 357218,
> "last_scrub": "356347'1923757",
> "last_scrub_stamp": "2015-12-12 12:18:54.719534",
> "last_deep_scrub": "356347'1923757",
> "last_deep_scrub_stamp": "2015-12-12 12:18:54.719534",
> "last_clean_scrub_stamp": "2015-12-12 12:18:54.719534"
> },
> "stats": {
> "version": "356361'19237

Re: [ceph-users] All pgs stuck peering

[Chris Dunlop]

Hi Varada,

On Mon, Dec 14, 2015 at 03:23:20AM +, Varada Kari wrote:
> Can get the details of 
> 
> 1. ceph health detail
> 2. ceph pg query  
> 
> of any one PG stuck peering
> 
> 
> Varada

The full health detail is over 9000 lines, but here's a summary:

# ceph health detail | head
HEALTH_WARN 3072 pgs peering; 3072 pgs stuck inactive; 3072 pgs stuck unclean; 
1570 requests are blocked > 32 sec; 25 osds have slow requests; noout flag(s) 
set
pg 3.1ae is stuck inactive for 23264.342056, current state peering, last acting 
[16,4,8]
pg 2.1af is stuck inactive for 23621.565024, current state peering, last acting 
[6,0]
pg 6.1ab is stuck inactive for 22843.875498, current state peering, last acting 
[27,18,54]
pg 3.1af is stuck inactive for 23315.971276, current state peering, last acting 
[17,16,24]
pg 2.1ae is stuck inactive for 19278.004657, current state peering, last acting 
[7,1]
pg 6.1aa is stuck inactive for 19321.668092, current state peering, last acting 
[31,39,56]
pg 3.1a8 is stuck inactive for 22897.969982, current state peering, last acting 
[16,17,24]
pg 2.1a9 is stuck inactive for 23516.554757, current state peering, last acting 
[14,7]
pg 6.1ad is stuck inactive for 23105.915508, current state peering, last acting 
[33,47,20]
# ceph health detail | grep -v peering
34 ops are blocked > 16777.2 sec
1289 ops are blocked > 8388.61 sec
50 ops are blocked > 4194.3 sec
34 ops are blocked > 2097.15 sec
68 ops are blocked > 1048.58 sec
13 ops are blocked > 524.288 sec
11 ops are blocked > 16777.2 sec on osd.0
4 ops are blocked > 8388.61 sec on osd.0
5 ops are blocked > 8388.61 sec on osd.1
100 ops are blocked > 8388.61 sec on osd.2
100 ops are blocked > 8388.61 sec on osd.3
100 ops are blocked > 8388.61 sec on osd.4
80 ops are blocked > 8388.61 sec on osd.5
34 ops are blocked > 8388.61 sec on osd.6
27 ops are blocked > 4194.3 sec on osd.6
15 ops are blocked > 2097.15 sec on osd.6
6 ops are blocked > 1048.58 sec on osd.6
9 ops are blocked > 524.288 sec on osd.6
2 ops are blocked > 16777.2 sec on osd.7
20 ops are blocked > 4194.3 sec on osd.7
16 ops are blocked > 2097.15 sec on osd.7
62 ops are blocked > 1048.58 sec on osd.7
85 ops are blocked > 8388.61 sec on osd.8
80 ops are blocked > 8388.61 sec on osd.9
13 ops are blocked > 16777.2 sec on osd.10
3 ops are blocked > 8388.61 sec on osd.10
1 ops are blocked > 4194.3 sec on osd.10
1 ops are blocked > 2097.15 sec on osd.10
6 ops are blocked > 8388.61 sec on osd.11
5 ops are blocked > 8388.61 sec on osd.12
4 ops are blocked > 8388.61 sec on osd.13
2 ops are blocked > 8388.61 sec on osd.14
4 ops are blocked > 524.288 sec on osd.14
7 ops are blocked > 16777.2 sec on osd.15
12 ops are blocked > 8388.61 sec on osd.15
2 ops are blocked > 4194.3 sec on osd.15
2 ops are blocked > 2097.15 sec on osd.15
100 ops are blocked > 8388.61 sec on osd.16
82 ops are blocked > 8388.61 sec on osd.17
1 ops are blocked > 16777.2 sec on osd.18
100 ops are blocked > 8388.61 sec on osd.21
86 ops are blocked > 8388.61 sec on osd.24
100 ops are blocked > 8388.61 sec on osd.38
100 ops are blocked > 8388.61 sec on osd.42
100 ops are blocked > 8388.61 sec on osd.44
1 ops are blocked > 8388.61 sec on osd.51
25 osds have slow requests
noout flag(s) set


# ceph pg 3.1ae query
<<< hung, until ^c >>>
# ceph pg 2.1af query
{
"state": "peering",
"snap_trimq": "[]",
"epoch": 357236,
"up": [
6,
0
],
"acting": [
6,
0
],
"info": {
"pgid": "2.1af",
"last_update": "356361'1923761",
"last_complete": "356361'1923761",
"log_tail": "341349'1920757",
"last_user_version": 1923761,
"last_backfill": "MAX",
"purged_snaps": "[1~34,38~1b,55~2,59~2a,84~68,ee~62]",
"history": {
"epoch_created": 1,
"last_epoch_started": 356496,
"last_epoch_clean": 356496,
"last_epoch_split": 0,
"same_up_since": 357218,
"same_interval_since": 357218,
"same_primary_since": 357218,
"last_scrub": "356347'1923757",
"last_scrub_stamp": "2015-12-12 12:18:54.719534",
"last_deep_scrub": "356347'1923757",
"last_deep_scrub_stamp": "2015-12-12 12:18:54.719534",
"last_clean_scrub_stamp": "2015-12-12 12:18:54.719534"
},
"stats": {
"version": "356361'1923761",
"reported_seq": "37552607",
"reported_epoch": "357218",
"state": "peering",
"last_fresh": "2015-12-14 12:54:41.084804",
"last_change": "2015-12-14 12:54:41.084804",
"last_active": "2015-12-14 07:53:05.850772",
"last_peered": "2015-12-14 07:53:05.850772",
"last_clean": "2015-12-14 07:53:05.850772",
"last_became_active": "2013-09-11 09:13:39.309600",
"last_became_peered": "2013-09-11 09:13:39.309600",
"last_unstale": "2015-12-14 12:54:41.084804",

Re: [ceph-users] Cephfs I/O when no I/O operations are submitted

[Christian Balzer]

Hello,

On Mon, 14 Dec 2015 11:46:43 +0800 xiafei wrote:

> Hi all,
>   I have a question about the I/O of cephfs.
> I configure cephfs with 2 OSDs, and 300PGs in two HDDs. Then I use the
> iostat (iostat -kx 1 /dev/sdd1) to monitor the I/Os of the HDDs
> (/dev/sdd1). The result is as follows:
> 
> 
> The iostat shows that there are write requests every second. However I
> do not execute any applications. What the source of the write I/O? 
> 
That's why you use atop and/or iotop to determine what program is writing
to the disks.

If it is indeed Ceph (ceph-osd process) writing to the disk then the
question is indeed what causes that. 
I have no experience with CephFS, but quiescent RBD volume (mounted VM
images) does not produce I/O by itself.

You say that you're not running any application, but is the CephFS mounted
somewhere?

Regards,

Christian
-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Global OnLine Japan/Rakuten Communications
http://www.gol.com/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Cephfs I/O when no I/O operations are submitted

[xiafei]

Hi all,
I have a question about the I/O of cephfs.
I configure cephfs with 2 OSDs, and 300PGs in two HDDs. Then I use the iostat 
(iostat -kx 1 /dev/sdd1) to monitor the I/Os of the HDDs (/dev/sdd1). 
The result is as follows:


The iostat shows that there are write requests every second. However I do not 
execute any applications.  
What the source of the write I/O? 

Thanks.


Best regards,
Fei Xia

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] All pgs stuck peering

[Varada Kari]

Can get the details of 

1. ceph health detail
2. ceph pg query  

of any one PG stuck peering


Varada

> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
> Chris Dunlop
> Sent: Monday, December 14, 2015 8:22 AM
> To: ceph-users@lists.ceph.com
> Subject: [ceph-users] All pgs stuck peering
> 
> Hi,
> 
> ceph 0.94.5
> 
> After restarting one of our three osd hosts to increase the RAM and change
> from linux 3.18.21 to 4.1., the cluster is stuck with all pgs peering:
> 
> # ceph -s
> cluster c6618970-0ce0-4cb2-bc9a-dd5f29b62e24
>  health HEALTH_WARN
> 3072 pgs peering
> 3072 pgs stuck inactive
> 3072 pgs stuck unclean
> 1450 requests are blocked > 32 sec
> noout flag(s) set
>  monmap e9: 3 mons at
> {b2=10.200.63.130:6789/0,b4=10.200.63.132:6789/0,b5=10.200.63.133:6789/0}
> election epoch 74462, quorum 0,1,2 b2,b4,b5
>  osdmap e356963: 59 osds: 59 up, 59 in
> flags noout
>   pgmap v69385733: 3072 pgs, 3 pools, 11973 GB data, 3340 kobjects
> 31768 GB used, 102 TB / 133 TB avail
> 3072 peering
> 
> What can I do to diagnose (or better yet, fix!) this?
> 
> Downgrading back to 3.18.21 hasn't helped.
> 
> Each host (now) has 192G RAM. One has 17 osds, the other two have 21 osds
> each.
> 
> I can see there's traffic going between the osd ports on the various osd
> hosts, but all small packets (122 or 131 bytes).
> 
> Just prior to upgrading this osd host another one had also been upgraded
> (RAM + linux). The cluster had no trouble at that point and was healthy
> within a few minutes of that server starting up.
> 
> The cluster has been working fine for years up to now, having had rolling
> upgrades since dumpling.
> 
> Cheers,
> 
> Chris
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] All pgs stuck peering

[Chris Dunlop]

Hi,

ceph 0.94.5

After restarting one of our three osd hosts to increase the RAM and change
from linux 3.18.21 to 4.1., the cluster is stuck with all pgs peering:

# ceph -s
cluster c6618970-0ce0-4cb2-bc9a-dd5f29b62e24
 health HEALTH_WARN
3072 pgs peering
3072 pgs stuck inactive
3072 pgs stuck unclean
1450 requests are blocked > 32 sec
noout flag(s) set
 monmap e9: 3 mons at 
{b2=10.200.63.130:6789/0,b4=10.200.63.132:6789/0,b5=10.200.63.133:6789/0}
election epoch 74462, quorum 0,1,2 b2,b4,b5
 osdmap e356963: 59 osds: 59 up, 59 in
flags noout
  pgmap v69385733: 3072 pgs, 3 pools, 11973 GB data, 3340 kobjects
31768 GB used, 102 TB / 133 TB avail
3072 peering

What can I do to diagnose (or better yet, fix!) this?

Downgrading back to 3.18.21 hasn't helped.

Each host (now) has 192G RAM. One has 17 osds, the other two have 21 osds
each.

I can see there's traffic going between the osd ports on the various osd
hosts, but all small packets (122 or 131 bytes).

Just prior to upgrading this osd host another one had also been upgraded
(RAM + linux). The cluster had no trouble at that point and was healthy
within a few minutes of that server starting up.

The cluster has been working fine for years up to now, having had rolling
upgrades since dumpling.

Cheers,

Chris
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Monitor rename / recreate issue -- probing state

[deeepdish]

Perhaps I’m not understanding something..

The “extra_probe_peers” ARE the other working monitors in quorum out of the 
mon_host line in ceph.conf.

In the example below 10.20.1.8 = b20s08; 10.20.10.251 = smon01s; 10.20.10.252 = 
smon02s

The monitor is not reaching out to the other IPs and syncing.   I’m able to 
ping all IPs in the extra_probe_peers list.

# ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.smg01.asok 
mon_status
{
"name": "smg01",
"rank": 0,
"state": "probing",
"election_epoch": 0,
"quorum": [],
"outside_quorum": [
"smg01"
],
"extra_probe_peers": [
"10.20.1.8:6789\/0",
"10.20.10.251:6789\/0",
"10.20.10.252:6789\/0"
],
"sync_provider": [],
"monmap": {
"epoch": 0,
"fsid": "693834c1-1f95-4237-ab97-a767b0c0e6e7",
"modified": "0.00",
"created": "0.00",
"mons": [
{
"rank": 0,
"name": "smg01",
"addr": "10.20.10.250:6789\/0"
},
{
"rank": 1,
"name": "smon01s",
"addr": "0.0.0.0:0\/1"
},
{
"rank": 2,
"name": "smon02s",
"addr": "0.0.0.0:0\/2"
},
{
"rank": 3,
"name": "b02s08",
"addr": "0.0.0.0:0\/3"
}
]
}
}


> On Dec 13, 2015, at 19:18 , Joao Eduardo Luis  wrote:
> 
> On 12/13/2015 12:26 PM, deeepdish wrote:
>>> 
>>> This appears to be consistent with a wrongly populated 'mon_host' and
>>> 'mon_initial_members' in your ceph.conf.
>>> 
>>> -Joao
>> 
>> 
>> Thanks Joao.   I had a look but my other 3 monitors are working just
>> fine.   To be clear, I’ve confirmed the same behaviour on other monitor
>> nodes that have been removed from the cluster and rebuild with a new IP
>> (however same name).
> 
> I'm not entirely sure what you mean, but let me clarify what I meant a bit.
> 
> Existing monitors take their monmap from their own stores. All monitors
> in a quorum will see the same monmap. Existing monitors do not care
> about the configuration file for their monmap.
> 
> 'mon_host' and 'mon_initial_members' are only used by clients trying to
> reach the monitors AND when creating a new monitor.
> 
> Therefore, when creating a new monitor, 'mon_host' must contain the ips
> of the existing monitors PLUS the monitor you are creating, and
> 'mon_initial_members' must contain the hosts of the existing monitors
> PLUS the host of the monitor you are creating.
> 
> Your initial email reflected a lot of other ips on the
> 'extra_probe_peers' (which is basically the contents of mon_host during
> the probing phase, while the monitor tries to find the other monitors),
> which is consistent with mon_host being wrongly populated.
> 
>  -Joao

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Monitor rename / recreate issue -- probing state

[Joao Eduardo Luis]

On 12/13/2015 12:26 PM, deeepdish wrote:
>>
>> This appears to be consistent with a wrongly populated 'mon_host' and
>> 'mon_initial_members' in your ceph.conf.
>>
>>  -Joao
> 
> 
> Thanks Joao.   I had a look but my other 3 monitors are working just
> fine.   To be clear, I’ve confirmed the same behaviour on other monitor
> nodes that have been removed from the cluster and rebuild with a new IP
> (however same name).

I'm not entirely sure what you mean, but let me clarify what I meant a bit.

Existing monitors take their monmap from their own stores. All monitors
in a quorum will see the same monmap. Existing monitors do not care
about the configuration file for their monmap.

'mon_host' and 'mon_initial_members' are only used by clients trying to
reach the monitors AND when creating a new monitor.

Therefore, when creating a new monitor, 'mon_host' must contain the ips
of the existing monitors PLUS the monitor you are creating, and
'mon_initial_members' must contain the hosts of the existing monitors
PLUS the host of the monitor you are creating.

Your initial email reflected a lot of other ips on the
'extra_probe_peers' (which is basically the contents of mon_host during
the probing phase, while the monitor tries to find the other monitors),
which is consistent with mon_host being wrongly populated.

  -Joao
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] where is the client

[Linux Chips]

  
  
Hi all,
i have been trying to send this to the dev mailing list, but the
mail was rejected! for what ever reason, though i am subscribed. any
one facing this issue with the dev list? i thought it is related to
the dev the most since i was digging inside the code for a wile now.
never the less, I have noticed if there are some clients misbehaving
it will effect the whole cluster, causing blocked requests and
slowing down every thing. particularly when a client is having a
network related issue, like a congested network or a faulty optic.
so i am trying to track the clients on our cluster. any good method
of locating/listing all the currently connected clients?
and, i am trying to find the clients from there requests, i have
been through the source code for the past couple of days but could
not find what i wanted. the dump operations, e.g. from running (ceph
daemon osd.xxx dump_historic_ops), gives me something like: (running
ceph v0.94.2)

    "description": "osd_op(client.36076101.0:62833746
default.36046497.2__shadow_2092280179.2~MFtK8tZp8-UmvkM0qbVkf6B4xQbLT-p.50_1



[] 108.f7ba6ce0 ack+ondisk+write+known_if_redirected e331887)",
    "initiated_at": "2015-12-05 15:26:08.159832",
    "age": 465.722853,
    "duration": 1.256738,
    "type_data": [
    "commit sent; apply or cleanup",
    {
    "client": "client.36076101",
    "tid": 62833746
    },
.
.
.
.
    {
    "time": "2015-12-05 15:26:09.416570",
    "event": "done"
    }
    ]
    ]
    },

the only peace of info that it might be useful (i assumed) is the
description, i can see the client id in it
(client.36076101.0:62833746, it is the client id, right?) but could
not understand from where it comes. my head was spinning inside the
source, though its the first time i go there ;). first i thought it
is related to the client ip or something, but could not confirm
that. any one have a clue whether this can get me somewhere? any
other ideas are appreciated.
thanks
Ali
  

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] about federated gateway

[孙方臣]

Hi, All,

I'm setting up federated gateway. One is master zone, the other is slave
zone. Radosgw-agent is running in slave zone. I have encountered some
problems, can anybody help answering this:

1.  When put a object to radosgw, there are two bilogs to generate. One is
"pending" state, the other is "complete" state.This should be ignored when
the entry is "pending" state, otherwise the same object will be copied
twice. I have a pull request that is at
https://github.com/ceph/radosgw-agent/pull/39, please give some suggestions
about it.

2.  When the "rgw_num_rados_handles" is set as 16, the radosgw-agent
caannot unlock, the error code is 404.  the log is following:
..
2015-12-13 21:52:33,373 26594 [radosgw_agent.lock][WARNING] failed to
unlock shard 115 in zone zone-a: Http error code 404 content Not Found
..
2015-12-13 21:53:00,732 26594 [radosgw_agent.lock][ERROR ] locking shard
116 in zone zone-a failed: Http error code 423 content
..

I can find the locker with the "rados lock info" command, and can break the
lock with  "rados lock break" command.
I find the reason finally, the reason is that the lock request from
radosgw-agent is processed by rados client and the unlock request from
radosgw-agent is processed by anther rados client. When  the
"rgw_num_rados_handles"
is set as 1, the warning message did not appeared.
Can anybody help giving some suggestions about this, and can the warning
message be ignored?

Thanks !
sunfch
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Monitor rename / recreate issue -- probing state

[deeepdish]

On 12/10/2015 04:00 AM, deeepdish wrote:
> Hello,
> 
> I encountered a strange issue when rebuilding monitors reusing same
> hostnames, however different IPs.
> 
> Steps to reproduce:
> 
> - Build monitor using ceph-deploy create mon 
> - Remove monitor
> via http://docs.ceph.com/docs/master/rados/operations/add-or-rm-mons/ 
> 
> (remove monitor) ? I didn?t realize there was a ceph-deploy mon destroy
> command at this point.
> - Build a new monitor on same hardware using ceph-deploy create mon
>   # reason = to rename / change IP of monitor as per above link
> - Monitor ends up in probing mode.   When connecting via the admin
> socket, I see that there are no peers avail.   
> 
> The above behavior of only when reinstalling monitors.   I even tried
> reinstalling the OS, however there?s a monmap embedded somewhere causing
> the previous monitor hostnames / IPs to conflict with the new monitor?s
> peering ability.  

> [root@b02s08 ~]# 
> 
> On a reinstalled (not working) monitor:
> 
> sudo ceph --cluster=ceph --admin-daemon
> /var/run/ceph/ceph-mon.smg01.asok mon_status
> {
>"name": "smg01",
>"rank": 0,
>"state": "probing",
>"election_epoch": 0,
>"quorum": [],
>"outside_quorum": [
>"smg01"
>],
>"extra_probe_peers": [
>"10.20.1.8:6789\/0",
>"10.20.10.14:6789\/0",
>"10.20.10.16:6789\/0",
>"10.20.10.18:6789\/0",
>"10.20.10.251:6789\/0",
>"10.20.10.252:6789\/0"
>],
[snip]
> }


> 
> This appears to be consistent with a wrongly populated 'mon_host' and
> 'mon_initial_members' in your ceph.conf.
> 
>  -Joao



Thanks Joao.   I had a look but my other 3 monitors are working just fine.   To 
be clear, I’ve confirmed the same behaviour on other monitor nodes that have 
been removed from the cluster and rebuild with a new IP (however same name).

[global]
fsid = (hidden)
mon_initial_members = smg01, smon01s, smon02s, b02s08
mon_host = 10.20.10.250, 10.20.10.251, 10.20.10.252, 10.20.1.8
public network = 10.20.10.0/24, 10.20.1.0/24
cluster network = 10.20.41.0/24

. . . 

[mon.smg01s]
#host = smg01s.erbus.kupsta.net
host = smg01s
addr = 10.20.10.250:6789

[mon.smon01s]
#host = smon01s.erbus.kupsta.net
host = smon01s
addr = 10.20.10.251:6789

[mon.smon02s]
#host = smon02s.erbus.kupsta.net
host = smon02s
addr = 10.20.10.252:6789

[mon.b02s08]
#host = b02s08.erbus.kupsta.net
host = b02s08
addr = 10.20.1.8:6789

# sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.smg01.asok 
mon_status
{
"name": "smg01",
"rank": 0,
"state": "probing",
"election_epoch": 0,
"quorum": [],
"outside_quorum": [
"smg01"
],
"extra_probe_peers": [
"10.20.1.8:6789\/0",
"10.20.10.251:6789\/0",
"10.20.10.252:6789\/0"
],
"sync_provider": [],
"monmap": {
"epoch": 0,
"fsid": “(hidden)",
"modified": "0.00",
"created": "0.00",
"mons": [
{
"rank": 0,
"name": "smg01",
"addr": "10.20.10.250:6789\/0"
},
{
"rank": 1,
"name": "smon01s",
"addr": "0.0.0.0:0\/1"
},
{
"rank": 2,
"name": "smon02s",
"addr": "0.0.0.0:0\/2"
},
{
"rank": 3,
"name": "b02s08",
"addr": "0.0.0.0:0\/3"
}
]
}
}

Processes running on the monitor node that’s in probing state:

# ps -ef | grep ceph
root  1140 1  0 Dec11 ?00:05:07 python 
/usr/sbin/ceph-create-keys --cluster ceph -i smg01
root  6406 1  0 Dec11 ?00:05:10 python 
/usr/sbin/ceph-create-keys --cluster ceph -i smg01
root  7712 1  0 Dec11 ?00:05:09 python 
/usr/sbin/ceph-create-keys --cluster ceph -i smg01
root  9105 1  0 Dec11 ?00:05:11 python 
/usr/sbin/ceph-create-keys --cluster ceph -i smg01
root 13098 30548  0 07:18 pts/100:00:00 grep --color=auto ceph
root 14243 1  0 Dec11 ?00:05:09 python 
/usr/sbin/ceph-create-keys --cluster ceph -i smg01
root 31222 1  0 05:39 ?00:00:00 /bin/bash -c ulimit -n 32768; 
/usr/bin/ceph-mon -i smg01 --pid-file /var/run/ceph/mon.smg01.pid -c 
/etc/ceph/ceph.conf --cluster ceph -f
root 31226 31222  1 05:39 ?00:01:39 /usr/bin/ceph-mon -i smg01 
--pid-file /var/run/ceph/mon.smg01.pid -c /etc/ceph/ceph.conf --cluster ceph -f
root 31228 1  0 05:39 pts/100:00:15 python 
/usr/sbin/ceph-create-keys --cluster ceph -i smg01

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com