Re: [ceph-users] PGs stuck activating after adding new OSDs

[Jakub Jaszewski]

Hi Jon,  can you reweight one OSD to default value and share outcome of "ceph
osd df tree; ceph -s; ceph health detail"  ?

Recently I was adding new node, 12x 4TB, one disk at a time and faced
activating+remapped state for few hours.

Not sure but maybe that was caused by "osd_max_backfills" value and
backfill awaiting PGs queue.

# ceph -s
>   cluster:
> id: 1023c49f-3a10-42de-9f62-9b122db21e1e
> health: HEALTH_WARN
> noscrub,nodeep-scrub flag(s) set
> 1 nearfull osd(s)
> 19 pool(s) nearfull
> 6982/289660233 objects misplaced (11.509%)
> Reduced data availability: 29 pgs inactive
> Degraded data redundancy: 788023/289660233 objects degraded
> (0.272%), 782 pgs unclean, 54 pgs degraded, 48 pgs undersized
>
>   services:
> mon: 3 daemons, quorum mon1,mon2,mon3
> mgr: mon2(active), standbys: mon3, mon1
> osd: 120 osds: 120 up, 120 in; 779 remapped pgs
>  flags noscrub,nodeep-scrub
> rgw: 3 daemons active
>
>   data:
> pools:   19 pools, 3760 pgs
> objects: 38285k objects, 146 TB
> usage:   285 TB used, 150 TB / 436 TB avail
> pgs: 0.771% pgs not active
>  788023/289660233 objects degraded (0.272%)
>  6982/289660233 objects misplaced (11.509%)
>  2978 active+clean
>  646  active+remapped+backfill_wait
>  57   active+remapped+backfilling
>  27   active+undersized+degraded+remapped+backfill_wait
>  25   activating+remapped
>  17   active+undersized+degraded+remapped+backfilling
>  4activating+undersized+degraded+remapped
>  3active+recovery_wait+degraded
>  3active+recovery_wait+degraded+remapped
>
>   io:
> client:   2228 kB/s rd, 54831 kB/s wr, 539 op/s rd, 756 op/s wr
> recovery: 1360 MB/s, 348 objects/s


Now all PGs are active+clean.


Regards
Jakub
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] 1 mon unable to join the quorum

[Brad Hubbard]

Can you update with the result of the following commands from all of the MONs?

# ceph --admin-daemon /var/run/ceph/ceph-mon.[whatever].asok mon_status
# ceph --admin-daemon /var/run/ceph/ceph-mon.[whatever].asok quorum_status

On Thu, Mar 29, 2018 at 3:11 PM, Gauvain Pocentek
 wrote:
> Hello Ceph users,
>
> We are having a problem on a ceph cluster running Jewel: one of the mons
> left the quorum, and we  have not been able to make it join again. The two
> other monitors are running just fine, but obviously we need this third one.
>
> The problem happened before Jewel, when the cluster was running Infernalis.
> We upgraded hoping that it would solve the problem, but no luck.
>
> We've validated several things: no network problem, no clock skew, same OS
> and ceph version everywhere. We've also removed the mon completely, and
> recreated it. We also tried to run an additional mon on one of the OSD
> machines, this mon didn't join the quorum either.
>
> We've opened https://tracker.ceph.com/issues/23403 with logs from the 3 mons
> during a fresh startup of the problematic logs.
>
> Is there anything we could try to do to resolve this issue? We are getting
> out of ideas.
>
> We'd appreciate any suggestion!
>
> Gauvain Pocentek
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
Cheers,
Brad
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] 1 mon unable to join the quorum

[Gauvain Pocentek]

Hello Ceph users,

We are having a problem on a ceph cluster running Jewel: one of the 
mons left the quorum, and we  have not been able to make it join again. 
The two other monitors are running just fine, but obviously we need this 
third one.


The problem happened before Jewel, when the cluster was running 
Infernalis. We upgraded hoping that it would solve the problem, but no 
luck.


We've validated several things: no network problem, no clock skew, same 
OS and ceph version everywhere. We've also removed the mon completely, 
and recreated it. We also tried to run an additional mon on one of the 
OSD machines, this mon didn't join the quorum either.


We've opened https://tracker.ceph.com/issues/23403 with logs from the 3 
mons during a fresh startup of the problematic logs.


Is there anything we could try to do to resolve this issue? We are 
getting out of ideas.


We'd appreciate any suggestion!

Gauvain Pocentek

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Random individual OSD failures with "connection refused reported by" another OSD?

[Kjetil Joergensen]

Hi,

another possibility - the osd's "refusing connections" crashed, there's a
window of time where connection attempts will fail with connection refused,
in between osd died, the osd being re-started by upstart/systemd, and the
OSD gets far enough into it's init process to start listening for new
connections.

While your symptoms look the same, there's no guarantee that you're
suffering from the same problem, but.. we're currently suffering from
ceph-osd v12.2.4 sporadically segfaulting. Either for config reasons or the
signal handler fails to do it's thing, we don't get the typical "oops I
crashed" reports in the osd log, although journald/systemd did capture
stdout which mentions it and there's a kernel log message left behind
saying that ceph-osd segfaulted. (http://tracker.ceph.com/issues/23352 ).

-KJ

On Wed, Mar 28, 2018 at 10:50 AM, Andre Goree  wrote:

> On 2018/03/28 1:39 pm, Subhachandra Chandra wrote:
>
> We have seen similar behavior when there are network issues. AFAIK, the
>> OSD is being reported down by an OSD that cannot reach it. But either
>> another OSD that can reach it or the heartbeat between the OSD and the
>> monitor declares it up. The OSD "boot" message does not seem to indicate an
>> actual OSD restart.
>>
>> Subhachandra
>>
>> On Wed, Mar 28, 2018 at 10:30 AM, Andre Goree  wrote:
>>
>> Hello,
>>>
>>> I've recently had a minor issue come up where random individual OSDs are
>>> failed due to a connection refused on another OSD.  I say minor, bc it's
>>> not a node-wide issue, and appears to be random nodes -- and besides that,
>>> the OSD comes up within less than a second, as if the OSD is sent a
>>> "restart," or something.
>>>
>>> ...
>
>
> Great!  Thank you!  Yes I found it funny that it "restarted" so quickly,
> and from my readings I remember that it takes more than a single OSD
> heartbeat failing to produce and _actual_ failure, so as to prevent false
> positives.  Thanks for the insight!
>
>
>
> --
> Andre Goree
> -=-=-=-=-=-
> Email - andre at drenet.net
> Website   - http://blog.drenet.net
> PGP key   - http://www.drenet.net/pubkey.html
> -=-=-=-=-=-
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 
Kjetil Joergensen 
SRE, Medallia Inc
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] session lost, hunting for new mon / session established : every 30s until unmount/remount

[Jean-Charles Lopez]

Hi,

if I read you crrectly you have 3 MONs on each data center. This means that 
when the link goes down you will loose quorum making the cluster unavailable.

If my perception is correct, you’d have to start a 7th MON somewhere else 
accessible from both sites for your cluster to maintain quorum during this 
event.

Regards
JC

> On Mar 28, 2018, at 15:40, Nicolas Huillard  wrote:
> 
> Hi all,
> 
> I didn't find much information regarding this kernel client loop in the
> ML. Here are my observation, around which I'll try to investigate.
> 
> My setup:
> * 2 datacenters connected using an IPsec tunnel configured for routing
> (2 subnets)
> * connection to the WAN using PPPoE and the pppd kernel module
> * the PPP connection lasts exactly 7 days, after which the provider
> kills it, and my PPP client restarts it (the WAN/inter-cluster
> communication is thus disconnected during ~30s)
> * 3 MON+n×OSD+MGR+MDS on each datacenter
> * 2 client servers using cephfs/kernel module; one of them on each
> datacenter runs the pppd client and the IPSec endpoint (Pacemaker
> manages this front-end aspect of the cluster)
> * a single cephfs mount which is not managed by Pacemaker
> 
> Observations:
> * when the ppp0 connection stops, the pppd restores the default route
> from "using the PPP tunnel" to "using a virtual IP which happens to be
> on the same host" (but could move to the other peer)
> 
> Mar 28 19:07:09 neon pppd[5543]: restoring old default route to eth0 
> [172.21.0.254]
> 
> * IPsec et al. react cleanly (remove the tunnel, recreate it when PPP
> is up again)
> 
> Mar 28 19:07:43 neon pppd[5543]: Connected to 02::42 via interface 
> eth1.835
> Mar 28 19:07:43 neon pppd[5543]: CHAP authentication succeeded
> Mar 28 19:07:43 neon pppd[5543]: peer from calling number 02::42 
> authorized
> Mar 28 19:07:43 neon pppd[5543]: replacing old default route to eth0 
> [172.21.0.254]
> 
> * 20s after the PPP link is up and IPsec is restored, libceph starts to
> complain (neon is the client/gateway on 172.21.0.0/16 which lost its
> PPP, sodium is the remote side of the same IPsec tunnel) :
> 
> Mar 28 19:08:03 neon kernel: [1232455.656828] libceph: mon1 172.21.0.18:6789 
> socket closed (con state OPEN)
> Mar 28 19:08:12 neon kernel: [1232463.846633] ceph: mds0 caps stale
> Mar 28 19:08:16 neon kernel: [1232468.128577] ceph: mds0 caps went stale, 
> renewing
> Mar 28 19:08:16 neon kernel: [1232468.128581] ceph: mds0 caps stale
> Mar 28 19:08:30 neon kernel: [1232482.601183] libceph: mon3 172.22.0.16:6789 
> session established
> Mar 28 19:09:01 neon kernel: [1232513.256059] libceph: mon3 172.22.0.16:6789 
> session lost, hunting for new mon
> Mar 28 19:09:01 neon kernel: [1232513.321176] libceph: mon5 172.22.0.20:6789 
> session established
> Mar 28 19:09:32 neon kernel: [1232543.977003] libceph: mon5 172.22.0.20:6789 
> session lost, hunting for new mon
> Mar 28 19:09:32 neon kernel: [1232543.979567] libceph: mon2 172.21.0.20:6789 
> session established
> Mar 28 19:09:39 neon kernel: [1232551.435001] ceph: mds0 caps renewed
> Mar 28 19:10:02 neon kernel: [1232574.697885] libceph: mon2 172.21.0.20:6789 
> session lost, hunting for new mon
> Mar 28 19:10:02 neon kernel: [1232574.763614] libceph: mon4 172.22.0.18:6789 
> session established
> Mar 28 19:10:33 neon kernel: [1232605.418776] libceph: mon4 172.22.0.18:6789 
> session lost, hunting for new mon
> Mar 28 19:10:33 neon kernel: [1232605.420896] libceph: mon0 172.21.0.16:6789 
> session established
> Mar 28 19:11:04 neon kernel: [1232636.139720] libceph: mon0 172.21.0.16:6789 
> session lost, hunting for new mon
> Mar 28 19:11:04 neon kernel: [1232636.205717] libceph: mon3 172.22.0.16:6789 
> session established
> 
> Mar 28 19:07:40 sodium kernel: [1211268.708716] libceph: mon0 
> 172.21.0.16:6789 session lost, hunting for new mon
> Mar 28 19:07:44 sodium kernel: [1211272.208735] libceph: mon5 
> 172.22.0.20:6789 socket closed (con state OPEN)
> Mar 28 19:07:53 sodium kernel: [1211281.683700] libceph: mon2 
> 172.21.0.20:6789 socket closed (con state OPEN)
> Mar 28 19:08:18 sodium kernel: [1211306.856489] libceph: mon5 
> 172.22.0.20:6789 session established
> Mar 28 19:08:49 sodium kernel: [1211337.575101] libceph: mon5 
> 172.22.0.20:6789 session lost, hunting for new mon
> Mar 28 19:08:49 sodium kernel: [1211337.640884] libceph: mon0 
> 172.21.0.16:6789 session established
> Mar 28 19:09:20 sodium kernel: [1211368.296187] libceph: mon0 
> 172.21.0.16:6789 session lost, hunting for new mon
> Mar 28 19:09:20 sodium kernel: [1211368.299194] libceph: mon4 
> 172.22.0.18:6789 session established
> Mar 28 19:09:50 sodium kernel: [1211399.017229] libceph: mon4 
> 172.22.0.18:6789 session lost, hunting for new mon
> Mar 28 19:09:50 sodium kernel: [1211399.019655] libceph: mon5 
> 172.22.0.20:6789 session established
> 
> * the active MDS happens to be on sodium's side (172.22.*), whereas the
> primary MON happens to be on neon's side (172.21.*), which explains t

[ceph-users] session lost, hunting for new mon / session established : every 30s until unmount/remount

[Nicolas Huillard]

Hi all,

I didn't find much information regarding this kernel client loop in the
ML. Here are my observation, around which I'll try to investigate.

My setup:
* 2 datacenters connected using an IPsec tunnel configured for routing
(2 subnets)
* connection to the WAN using PPPoE and the pppd kernel module
* the PPP connection lasts exactly 7 days, after which the provider
kills it, and my PPP client restarts it (the WAN/inter-cluster
communication is thus disconnected during ~30s)
* 3 MON+n×OSD+MGR+MDS on each datacenter
* 2 client servers using cephfs/kernel module; one of them on each
datacenter runs the pppd client and the IPSec endpoint (Pacemaker
manages this front-end aspect of the cluster)
* a single cephfs mount which is not managed by Pacemaker

Observations:
* when the ppp0 connection stops, the pppd restores the default route
from "using the PPP tunnel" to "using a virtual IP which happens to be
on the same host" (but could move to the other peer)

Mar 28 19:07:09 neon pppd[5543]: restoring old default route to eth0 
[172.21.0.254]

* IPsec et al. react cleanly (remove the tunnel, recreate it when PPP
is up again)

Mar 28 19:07:43 neon pppd[5543]: Connected to 02::42 via interface eth1.835
Mar 28 19:07:43 neon pppd[5543]: CHAP authentication succeeded
Mar 28 19:07:43 neon pppd[5543]: peer from calling number 02::42 authorized
Mar 28 19:07:43 neon pppd[5543]: replacing old default route to eth0 
[172.21.0.254]

* 20s after the PPP link is up and IPsec is restored, libceph starts to
complain (neon is the client/gateway on 172.21.0.0/16 which lost its
PPP, sodium is the remote side of the same IPsec tunnel) :

Mar 28 19:08:03 neon kernel: [1232455.656828] libceph: mon1 172.21.0.18:6789 
socket closed (con state OPEN)
Mar 28 19:08:12 neon kernel: [1232463.846633] ceph: mds0 caps stale
Mar 28 19:08:16 neon kernel: [1232468.128577] ceph: mds0 caps went stale, 
renewing
Mar 28 19:08:16 neon kernel: [1232468.128581] ceph: mds0 caps stale
Mar 28 19:08:30 neon kernel: [1232482.601183] libceph: mon3 172.22.0.16:6789 
session established
Mar 28 19:09:01 neon kernel: [1232513.256059] libceph: mon3 172.22.0.16:6789 
session lost, hunting for new mon
Mar 28 19:09:01 neon kernel: [1232513.321176] libceph: mon5 172.22.0.20:6789 
session established
Mar 28 19:09:32 neon kernel: [1232543.977003] libceph: mon5 172.22.0.20:6789 
session lost, hunting for new mon
Mar 28 19:09:32 neon kernel: [1232543.979567] libceph: mon2 172.21.0.20:6789 
session established
Mar 28 19:09:39 neon kernel: [1232551.435001] ceph: mds0 caps renewed
Mar 28 19:10:02 neon kernel: [1232574.697885] libceph: mon2 172.21.0.20:6789 
session lost, hunting for new mon
Mar 28 19:10:02 neon kernel: [1232574.763614] libceph: mon4 172.22.0.18:6789 
session established
Mar 28 19:10:33 neon kernel: [1232605.418776] libceph: mon4 172.22.0.18:6789 
session lost, hunting for new mon
Mar 28 19:10:33 neon kernel: [1232605.420896] libceph: mon0 172.21.0.16:6789 
session established
Mar 28 19:11:04 neon kernel: [1232636.139720] libceph: mon0 172.21.0.16:6789 
session lost, hunting for new mon
Mar 28 19:11:04 neon kernel: [1232636.205717] libceph: mon3 172.22.0.16:6789 
session established

Mar 28 19:07:40 sodium kernel: [1211268.708716] libceph: mon0 172.21.0.16:6789 
session lost, hunting for new mon
Mar 28 19:07:44 sodium kernel: [1211272.208735] libceph: mon5 172.22.0.20:6789 
socket closed (con state OPEN)
Mar 28 19:07:53 sodium kernel: [1211281.683700] libceph: mon2 172.21.0.20:6789 
socket closed (con state OPEN)
Mar 28 19:08:18 sodium kernel: [1211306.856489] libceph: mon5 172.22.0.20:6789 
session established
Mar 28 19:08:49 sodium kernel: [1211337.575101] libceph: mon5 172.22.0.20:6789 
session lost, hunting for new mon
Mar 28 19:08:49 sodium kernel: [1211337.640884] libceph: mon0 172.21.0.16:6789 
session established
Mar 28 19:09:20 sodium kernel: [1211368.296187] libceph: mon0 172.21.0.16:6789 
session lost, hunting for new mon
Mar 28 19:09:20 sodium kernel: [1211368.299194] libceph: mon4 172.22.0.18:6789 
session established
Mar 28 19:09:50 sodium kernel: [1211399.017229] libceph: mon4 172.22.0.18:6789 
session lost, hunting for new mon
Mar 28 19:09:50 sodium kernel: [1211399.019655] libceph: mon5 172.22.0.20:6789 
session established

* the active MDS happens to be on sodium's side (172.22.*), whereas the
primary MON happens to be on neon's side (172.21.*), which explains the
dissimilar messages

* the remote peer for sodium also had a cephfs mount, and also
complained (lithium just experienced a lack of response from its
gateway, a virtual IP on sodium, which IPsec tunnel to neon was down) :

Mar 28 19:07:35 lithium kernel: [603537.458858] libceph: mon2 172.21.0.20:6789 
session lost, hunting for new mon
Mar 28 19:07:44 lithium kernel: [603545.702369] libceph: mon5 172.22.0.20:6789 
socket closed (con state OPEN)
Mar 28 19:07:58 lithium kernel: [603560.177718] libceph: mon2 172.21.0.20:6789 
socket closed (con state OPEN)
Mar 28 19:08:16 li

Re: [ceph-users] Random individual OSD failures with "connection refused reported by" another OSD?

[Andre Goree]

On 2018/03/28 1:39 pm, Subhachandra Chandra wrote:

We have seen similar behavior when there are network issues. AFAIK, the 
OSD is being reported down by an OSD that cannot reach it. But either 
another OSD that can reach it or the heartbeat between the OSD and the 
monitor declares it up. The OSD "boot" message does not seem to 
indicate an actual OSD restart.


Subhachandra

On Wed, Mar 28, 2018 at 10:30 AM, Andre Goree  wrote:


Hello,

I've recently had a minor issue come up where random individual OSDs 
are failed due to a connection refused on another OSD.  I say minor, 
bc it's not a node-wide issue, and appears to be random nodes -- and 
besides that, the OSD comes up within less than a second, as if the 
OSD is sent a "restart," or something.



...


Great!  Thank you!  Yes I found it funny that it "restarted" so quickly, 
and from my readings I remember that it takes more than a single OSD 
heartbeat failing to produce and _actual_ failure, so as to prevent 
false positives.  Thanks for the insight!



--
Andre Goree
-=-=-=-=-=-
Email - andre at drenet.net
Website   - http://blog.drenet.net
PGP key   - http://www.drenet.net/pubkey.html
-=-=-=-=-=-
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Random individual OSD failures with "connection refused reported by" another OSD?

[Subhachandra Chandra]

We have seen similar behavior when there are network issues. AFAIK, the OSD
is being reported down by an OSD that cannot reach it. But either another
OSD that can reach it or the heartbeat between the OSD and the monitor
declares it up. The OSD "boot" message does not seem to indicate an actual
OSD restart.

Subhachandra

On Wed, Mar 28, 2018 at 10:30 AM, Andre Goree  wrote:

> Hello,
>
> I've recently had a minor issue come up where random individual OSDs are
> failed due to a connection refused on another OSD.  I say minor, bc it's
> not a node-wide issue, and appears to be random nodes -- and besides that,
> the OSD comes up within less than a second, as if the OSD is sent a
> "restart," or something.
>
> On the MON I see this (notice the entire time is ~0.77s):
>
> 2018-03-26 22:39:36.821247 mon.mon-01 [INF] osd.77 failed
> (root=default,host=osd-05) (connection refused reported by osd.55)
> 2018-03-26 22:39:36.935445 mon.mon-01 [WRN] Health check failed: 1 osds
> down (OSD_DOWN)
> 2018-03-26 22:39:39.959496 mon.mon-01 [WRN] Health check failed: Reduced
> data availability: 6 pgs peering (PG_AVAILABILITY)
> 2018-03-26 22:39:41.969578 mon.mon-01 [WRN] Health check failed: Degraded
> data redundancy: 6528/1742880 objects degraded (0.375%), 46 pgs degraded
> (PG_DEGRADED)
> 2018-03-26 22:39:43.978429 mon.mon-01 [INF] Health check cleared:
> PG_AVAILABILITY (was: Reduced data availability: 6 pgs peering)
> 2018-03-26 22:39:48.913112 mon.mon-01 [WRN] Health check update: Degraded
> data redundancy: 19411/1742880 objects degraded (1.114%), 136 pgs degraded
> (PG_DEGRADED)
> 2018-03-26 22:40:06.288138 mon.mon-01 [INF] Health check cleared: OSD_DOWN
> (was: 1 osds down)
> 2018-03-26 22:40:06.301955 mon.mon-01 [INF] osd.77
> 172.16.238.18:6818/57264 boot
> 2018-03-26 22:40:07.298884 mon.mon-01 [WRN] Health check update: Degraded
> data redundancy: 17109/1742880 objects degraded (0.982%), 120 pgs degraded
> (PG_DEGRADED)
> 2018-03-26 22:40:13.330362 mon.mon-01 [INF] Health check cleared:
> PG_DEGRADED (was: Degraded data redundancy: 5605/1742880 objects degraded
> (0.322%), 39 pgs degraded)
> 2018-03-26 22:40:13.330409 mon.mon-01 [INF] Cluster is now healthy
>
>
> On host osd-05 (which hosts osd.77) there appears to be normal heartbeat
> traffic before the OSD spontaneously reboots (truncated for brevity):
>
> 2018-03-26 22:33:00.773897 7efcaf20f700  0 -- 172.16.239.18:6818/7788 >>
> 172.16.239.21:6818/8675 conn(0x55c5ea2d9000 :6818
> s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=0).handle_connect_msg accept connect_seq 73 vs existing csq=73
> existing_state=STATE_STANDBY
> 2018-03-26 22:33:00.774124 7efcaf20f700  0 -- 172.16.239.18:6818/7788 >>
> 172.16.239.21:6818/8675 conn(0x55c5ea2d9000 :6818
> s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=0).handle_connect_msg accept connect_seq 74 vs existing csq=73
> existing_state=STATE_STANDBY
> 2018-03-26 22:39:56.832556 7f80ceea8e00  0 set uid:gid to 64045:64045
> (ceph:ceph)
> 2018-03-26 22:39:56.832576 7f80ceea8e00  0 ceph version 12.2.4
> (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable), process
> (unknown), pid 57264
> 2018-03-26 22:39:56.849487 7f80ceea8e00  0 pidfile_write: ignore empty
> --pid-file
> 2018-03-26 22:39:56.859045 7f80ceea8e00  0 load: jerasure load: lrc load:
> isa
> 2018-03-26 22:39:56.859122 7f80ceea8e00  1 bdev create path
> /var/lib/ceph/osd/ceph-77/block type kernel
> 2018-03-26 22:39:56.859135 7f80ceea8e00  1 bdev(0x5568ccae4d80
> /var/lib/ceph/osd/ceph-77/block) open path /var/lib/ceph/osd/ceph-77/block
> 2018-03-26 22:39:56.859398 7f80ceea8e00  1 bdev(0x5568ccae4d80
> /var/lib/ceph/osd/ceph-77/block) open size 8001559724032 (0x7470220,
> 7452 GB) block_size 4096 (4096 B) rotational
> 2018-03-26 22:39:56.859711 7f80ceea8e00  1 
> bluestore(/var/lib/ceph/osd/ceph-77)
> _set_cache_sizes max 0.5 < ratio 0.99
> 2018-03-26 22:39:56.859733 7f80ceea8e00  1 
> bluestore(/var/lib/ceph/osd/ceph-77)
> _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0
> 2018-03-26 22:39:56.859738 7f80ceea8e00  1 bdev(0x5568ccae4d80
> /var/lib/ceph/osd/ceph-77/block) close
> 2018-03-26 22:39:57.132071 7f80ceea8e00  1 
> bluestore(/var/lib/ceph/osd/ceph-77)
> _mount path /var/lib/ceph/osd/ceph-77
> 2018-03-26 22:39:57.132534 7f80ceea8e00  1 bdev create path
> /var/lib/ceph/osd/ceph-77/block type kernel
> ...truncated...
>
>
> Same on host osd-07 (which hosts osd.55, the one that reported connection
> refused), you'll see normal heartbeat traffic, followed by something
> interesting, before normal heartbeat traffic returns:
>
> 2018-03-26 22:33:20.598576 7f495c52e700  0 -- 172.16.239.20:6810/7206 >>
> 172.16.239.21:6816/8668 conn(0x5619e70c9800 :6810
> s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> l=0).handle_connect_msg accept connect_seq 42 vs existing csq=41
> existing_state=STATE_STANDBY
> 2018-03-26 22:39:36.974204 7f495cd2f700  0 -- 172.16.239.20:6810/7206 >>
> 172.16.239.17:6805/6991 conn(0x5619f476b000 :6810
> s

Re: [ceph-users] ceph mds memory usage 20GB : is it normal ?

[Alexandre DERUMIER]

>>Can you also share `ceph daemon mds.2 cache status`, the full `ceph 
>>daemon mds.2 perf dump`, and `ceph status`? 

Sorry, too late, I needed to restart the mds daemon because I was out of memory 
:(

Seem stable for now. (around 500mb)

Not sure It was related, but I had a ganesha-nfs ->cephfs daemon running on 
this cluster. (but no client connected to it)


>>Note [1] will be in 12.2.5 and may help with your issue. 
>>[1] https://github.com/ceph/ceph/pull/20527 

ok thanks !



- Mail original -
De: "Patrick Donnelly" 
À: "Alexandre Derumier" 
Cc: "ceph-users" 
Envoyé: Mardi 27 Mars 2018 20:35:08
Objet: Re: [ceph-users] ceph mds memory usage 20GB : is it normal ?

Hello Alexandre, 

On Thu, Mar 22, 2018 at 2:29 AM, Alexandre DERUMIER  
wrote: 
> Hi, 
> 
> I'm running cephfs since 2 months now, 
> 
> and my active msd memory usage is around 20G now (still growing). 
> 
> ceph 1521539 10.8 31.2 20929836 20534868 ? Ssl janv.26 8573:34 
> /usr/bin/ceph-mds -f --cluster ceph --id 2 --setuser ceph --setgroup ceph 
> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 
> 
> 
> this is on luminous 12.2.2 
> 
> only tuning done is: 
> 
> mds_cache_memory_limit = 5368709120 
> 
> 
> (5GB). I known it's a soft limit, but 20G seem quite huge vs 5GB  
> 
> 
> Is it normal ? 

No, that's definitely not normal! 


> # ceph daemon mds.2 perf dump mds 
> { 
> "mds": { 
> "request": 1444009197, 
> "reply": 1443999870, 
> "reply_latency": { 
> "avgcount": 1443999870, 
> "sum": 1657849.656122933, 
> "avgtime": 0.001148095 
> }, 
> "forward": 0, 
> "dir_fetch": 51740910, 
> "dir_commit": 9069568, 
> "dir_split": 64367, 
> "dir_merge": 58016, 
> "inode_max": 2147483647, 
> "inodes": 2042975, 
> "inodes_top": 152783, 
> "inodes_bottom": 138781, 
> "inodes_pin_tail": 1751411, 
> "inodes_pinned": 1824714, 
> "inodes_expired": 7258145573, 
> "inodes_with_caps": 1812018, 
> "caps": 2538233, 
> "subtrees": 2, 
> "traverse": 1591668547, 
> "traverse_hit": 1259482170, 
> "traverse_forward": 0, 
> "traverse_discover": 0, 
> "traverse_dir_fetch": 30827836, 
> "traverse_remote_ino": 7510, 
> "traverse_lock": 86236, 
> "load_cent": 144401980319, 
> "q": 49, 
> "exported": 0, 
> "exported_inodes": 0, 
> "imported": 0, 
> "imported_inodes": 0 
> } 
> } 

Can you also share `ceph daemon mds.2 cache status`, the full `ceph 
daemon mds.2 perf dump`, and `ceph status`? 

Note [1] will be in 12.2.5 and may help with your issue. 

[1] https://github.com/ceph/ceph/pull/20527 

-- 
Patrick Donnelly 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Random individual OSD failures with "connection refused reported by" another OSD?

[Andre Goree]

Hello,

I've recently had a minor issue come up where random individual OSDs are 
failed due to a connection refused on another OSD.  I say minor, bc it's 
not a node-wide issue, and appears to be random nodes -- and besides 
that, the OSD comes up within less than a second, as if the OSD is sent 
a "restart," or something.


On the MON I see this (notice the entire time is ~0.77s):

2018-03-26 22:39:36.821247 mon.mon-01 [INF] osd.77 failed 
(root=default,host=osd-05) (connection refused reported by osd.55)
2018-03-26 22:39:36.935445 mon.mon-01 [WRN] Health check failed: 1 osds 
down (OSD_DOWN)
2018-03-26 22:39:39.959496 mon.mon-01 [WRN] Health check failed: Reduced 
data availability: 6 pgs peering (PG_AVAILABILITY)
2018-03-26 22:39:41.969578 mon.mon-01 [WRN] Health check failed: 
Degraded data redundancy: 6528/1742880 objects degraded (0.375%), 46 pgs 
degraded (PG_DEGRADED)
2018-03-26 22:39:43.978429 mon.mon-01 [INF] Health check cleared: 
PG_AVAILABILITY (was: Reduced data availability: 6 pgs peering)
2018-03-26 22:39:48.913112 mon.mon-01 [WRN] Health check update: 
Degraded data redundancy: 19411/1742880 objects degraded (1.114%), 136 
pgs degraded (PG_DEGRADED)
2018-03-26 22:40:06.288138 mon.mon-01 [INF] Health check cleared: 
OSD_DOWN (was: 1 osds down)
2018-03-26 22:40:06.301955 mon.mon-01 [INF] osd.77 
172.16.238.18:6818/57264 boot
2018-03-26 22:40:07.298884 mon.mon-01 [WRN] Health check update: 
Degraded data redundancy: 17109/1742880 objects degraded (0.982%), 120 
pgs degraded (PG_DEGRADED)
2018-03-26 22:40:13.330362 mon.mon-01 [INF] Health check cleared: 
PG_DEGRADED (was: Degraded data redundancy: 5605/1742880 objects 
degraded (0.322%), 39 pgs degraded)

2018-03-26 22:40:13.330409 mon.mon-01 [INF] Cluster is now healthy


On host osd-05 (which hosts osd.77) there appears to be normal heartbeat 
traffic before the OSD spontaneously reboots (truncated for brevity):


2018-03-26 22:33:00.773897 7efcaf20f700  0 -- 172.16.239.18:6818/7788 >> 
172.16.239.21:6818/8675 conn(0x55c5ea2d9000 :6818 
s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 
l=0).handle_connect_msg accept connect_seq 73 vs existing csq=73 
existing_state=STATE_STANDBY
2018-03-26 22:33:00.774124 7efcaf20f700  0 -- 172.16.239.18:6818/7788 >> 
172.16.239.21:6818/8675 conn(0x55c5ea2d9000 :6818 
s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 
l=0).handle_connect_msg accept connect_seq 74 vs existing csq=73 
existing_state=STATE_STANDBY
2018-03-26 22:39:56.832556 7f80ceea8e00  0 set uid:gid to 64045:64045 
(ceph:ceph)
2018-03-26 22:39:56.832576 7f80ceea8e00  0 ceph version 12.2.4 
(52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable), process 
(unknown), pid 57264
2018-03-26 22:39:56.849487 7f80ceea8e00  0 pidfile_write: ignore empty 
--pid-file
2018-03-26 22:39:56.859045 7f80ceea8e00  0 load: jerasure load: lrc 
load: isa
2018-03-26 22:39:56.859122 7f80ceea8e00  1 bdev create path 
/var/lib/ceph/osd/ceph-77/block type kernel
2018-03-26 22:39:56.859135 7f80ceea8e00  1 bdev(0x5568ccae4d80 
/var/lib/ceph/osd/ceph-77/block) open path 
/var/lib/ceph/osd/ceph-77/block
2018-03-26 22:39:56.859398 7f80ceea8e00  1 bdev(0x5568ccae4d80 
/var/lib/ceph/osd/ceph-77/block) open size 8001559724032 (0x7470220, 
7452 GB) block_size 4096 (4096 B) rotational
2018-03-26 22:39:56.859711 7f80ceea8e00  1 
bluestore(/var/lib/ceph/osd/ceph-77) _set_cache_sizes max 0.5 < ratio 
0.99
2018-03-26 22:39:56.859733 7f80ceea8e00  1 
bluestore(/var/lib/ceph/osd/ceph-77) _set_cache_sizes cache_size 
1073741824 meta 0.5 kv 0.5 data 0
2018-03-26 22:39:56.859738 7f80ceea8e00  1 bdev(0x5568ccae4d80 
/var/lib/ceph/osd/ceph-77/block) close
2018-03-26 22:39:57.132071 7f80ceea8e00  1 
bluestore(/var/lib/ceph/osd/ceph-77) _mount path 
/var/lib/ceph/osd/ceph-77
2018-03-26 22:39:57.132534 7f80ceea8e00  1 bdev create path 
/var/lib/ceph/osd/ceph-77/block type kernel

...truncated...


Same on host osd-07 (which hosts osd.55, the one that reported 
connection refused), you'll see normal heartbeat traffic, followed by 
something interesting, before normal heartbeat traffic returns:


2018-03-26 22:33:20.598576 7f495c52e700  0 -- 172.16.239.20:6810/7206 >> 
172.16.239.21:6816/8668 conn(0x5619e70c9800 :6810 
s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 
l=0).handle_connect_msg accept connect_seq 42 vs existing csq=41 
existing_state=STATE_STANDBY
2018-03-26 22:39:36.974204 7f495cd2f700  0 -- 172.16.239.20:6810/7206 >> 
172.16.239.17:6805/6991 conn(0x5619f476b000 :6810 
s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 
l=0).handle_connect_msg accept connect_seq 57 vs existing csq=57 
existing_state=STATE_STANDBY
2018-03-26 22:39:36.974457 7f495cd2f700  0 -- 172.16.239.20:6810/7206 >> 
172.16.239.17:6805/6991 conn(0x5619f476b000 :6810 
s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 
l=0).handle_connect_msg accept connect_seq 58 vs existing csq=57 
existing_state=STATE_STANDBY
2018-03-26 22:39:36.978674 7f494851d700  1 osd.55 pg_epoch: 21688 
pg[1.714( v 21687

Re: [ceph-users] where is it possible download CentOS 7.5

[Max Cuttins]

Hi Jason,

i really don't want to stress this much than I already did.
But I need to have a clear answer.


Il 28/03/2018 13:36, Jason Dillaman ha scritto:

But I don't think that CentOS7.5 will use the kernel 4.16 ... so you are
telling me that new feature will be backported to the kernel 3.* ?

Nope. I'm not part of the Red hat kernel team and don't have the
influence to shape what they do.

The RHEL/CentOS 7.5 3.x-based kernel will have all the necessary bug fixes.


You wrote about bug fixes... ok, but this is a new feature. (aka: before 
was not possible, not it is).

So, RHEL/CentOS 7.5 will run iSCSI LIO out of the box?
Yes or no?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

[adrien.geor...@cc.in2p3.fr]

Hmm looks like I restarted everything except MDS...
So it's the same issue! That's why the MDS kill themselves during the 
reboot of one of the monitors with MDS in 12.2.2.


Thanks Dan!

Adrien

Le 28/03/2018 à 16:43, Dan van der Ster a écrit :

Do you have the startup banners for mds.cccephadm14 and 15? It sure
looks like they were running 12.2.2 with the "not writeable with
daemon features" error.

-- dan

On Wed, Mar 28, 2018 at 3:12 PM, adrien.geor...@cc.in2p3.fr
 wrote:

Hi,

All Ceph services were in 12.2.4 version.

Adrien


Le 28/03/2018 à 14:47, Dan van der Ster a écrit :

Hi,

Which versions were those MDS's before and after the restarted standby
MDS?

Cheers, Dan



On Wed, Mar 28, 2018 at 11:11 AM, adrien.geor...@cc.in2p3.fr
 wrote:

Hi,

I just had the same issue with our 12.2.4 cluster but not during the
upgrade.
One of our 3 monitors restarted (the one with a standby MDS) and the 2
others active MDS killed themselves :

2018-03-28 09:36:24.376888 7f910bc0f700  0 mds.cccephadm14 handle_mds_map
mdsmap compatset compat={},rocompat={},incompat={1=base v0.20,2=client
writeable ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor
table,9=file layout v2} not writeable with daemon features
compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds
uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline
data,8=file layout v2}, killing myself
2018-03-28 09:36:24.376903 7f910bc0f700  1 mds.cccephadm14 suicide.
wanted
state up:active
2018-03-28 09:36:25.379607 7f910bc0f700  1 mds.1.62 shutdown: shutting
down
rank 1


2018-03-28 09:36:24.375867 7fad455bf700  0 mds.cccephadm15 handle_mds_map
mdsmap compatset compat={},rocompat={},incompat={1=base v0.20,2=client
writeable ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor
table,9=file layout v2} not writeable with daemon features
compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds
uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline
data,8=file layout v2}, killing myself
2018-03-28 09:36:24.375883 7fad455bf700  1 mds.cccephadm15 suicide.
wanted
state up:active
2018-03-28 09:36:25.377633 7fad455bf700  1 mds.0.50 shutdown: shutting
down
rank 0

I had to restart manually the MDS services to get it works.

Adrien


Le 21/03/2018 à 11:37, Martin Palma a écrit :

Just run into this problem on our production cluster

It would have been nice if the release notes of 12.2.4 had been
adapted to inform user about this.

Best,
Martin

On Wed, Mar 14, 2018 at 9:53 PM, Gregory Farnum 
wrote:

On Wed, Mar 14, 2018 at 12:41 PM, Lars Marowsky-Bree 
wrote:

On 2018-03-14T06:57:08, Patrick Donnelly  wrote:


Yes. But the real outcome is not "no MDS [is] active" but "some or
all
metadata I/O will pause" -- and there is no avoiding that. During an
MDS upgrade, a standby must take over the MDS being shutdown (and
upgraded).  During takeover, metadata I/O will briefly pause as the
rank is unavailable. (Specifically, no other rank can obtains locks
or
communicate with the "failed" rank; so metadata I/O will necessarily
pause until a standby takes over.) Single active vs. multiple active
upgrade makes little difference in this outcome.

Fair, except that there's no standby MDS at this time in case the
update
goes wrong.


Is another approach theoretically feasible? Have the updated MDS
only
go
into the incompatible mode once there's a quorum of new ones
available,
or something?

I believe so, yes. That option wasn't explored for this patch because
it was just disambiguating the compatibility flags and the full
side-effects weren't realized.

Would such a patch be accepted if we ended up pursuing this? Any
suggestions on how to best go about this?

It'd be ugly, but you'd have to set it up so that
* new MDSes advertise the old set of required values
* but can identify when all the MDSes are new
* then mark somewhere that they can use the correct values
* then switch to the proper requirements

I don't remember the details of this CompatSet code any more, and it's
definitely made trickier by the MDS having no permanent local state.
Since we do luckily have both the IDs and the strings, you might be
able to do something in the MDSMonitor to identify whether booting
MDSes have "too-old", "old-featureset-but-support-new-feature", or
"new, correct feature advertising" and then either massage that
incoming message down to the "old-featureset-but-support-new-feature"
(if not all the MDSes are new) or do an auto-upgrade of the required
features in the map. And you might also need compatibility code in the
MDS to make sure it sends out the appropriate bits on connection, but
I *think* the CompatSet checks are only done o

Re: [ceph-users] What do you use to benchmark your rgw?

[Janne Johansson]

2018-03-28 16:21 GMT+02:00 David Byte :

> I use cosbench (the last rc works well enough). I can get multiple GB/s
> from my 6 node cluster with 2 RGWs.
>
>
> To add info to this, it's not unexpectedly low for us, we know the
S3+https layer added latencies,
and it is EC pools on cheap+large spin disks optimized for cheap
almost-passive storage.

We get something close to 50% of what rados-bench gives us at the back end
of the RGWs,
that might be a more interesting value, since that bench doesn't have to do
any http chunking
nor MD5 sum the incoming files and so on.


> s3cmd and cli version of cyberduck to test it end-to-end using parallelism
> if possible.
>
> Getting some 100MB/s at most, from 500km distance over https against
> 5*radosgw behind HAProxy.
>
>
> 2018-03-28 11:17 GMT+02:00 Matthew Vernon :
>
>> Hi,
>>
>> What are people here using to benchmark their S3 service (i.e. the rgw)?
>> rados bench is great for some things, but doesn't tell me about what
>> performance I can get from my rgws.
>
>
-- 
May the most significant bit of your life be positive.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] What do you use to benchmark your rgw?

[Mark Nelson]

Personally I usually use a modified version of Mark Seger's getput tool 
here:


https://github.com/markhpc/getput/tree/wip-fix-timing

The difference between this version and upstream is primarily to make 
getput more accurate/useful when using something like CBT for 
orchestration instead of the included orchestration wrapper (gpsuite).


CBT can use this version of getput and run relatively accurate 
mutli-client tests without requiring quite as much setup as cosbench.  
Having said that, many folks have used cosbench effectively and I 
suspect that might be a good option for many people.  I'm not sure how 
much development is happening these days, I think the primary author may 
no longer be working on the project.


Mark

On 03/28/2018 09:21 AM, David Byte wrote:
I use cosbench (the last rc works well enough). I can get multiple 
GB/s from my 6 node cluster with 2 RGWs.


David Byte
Sr. Technical Strategist
IHV Alliances and Embedded
SUSE

Sent from my iPhone. Typos are Apple's fault.

On Mar 28, 2018, at 5:26 AM, Janne Johansson > wrote:


s3cmd and cli version of cyberduck to test it end-to-end using 
parallelism if possible.


Getting some 100MB/s at most, from 500km distance over https against 
5*radosgw behind HAProxy.



2018-03-28 11:17 GMT+02:00 Matthew Vernon >:


Hi,

What are people here using to benchmark their S3 service (i.e.
the rgw)?
rados bench is great for some things, but doesn't tell me about what
performance I can get from my rgws.

It seems that there used to be rest-bench, but that isn't in Jewel
AFAICT; I had a bit of a look at cosbench but it looks fiddly to
set up
and a bit under-maintained (the most recent version doesn't work
out of
the box, and the PR to fix that has been languishing for a while).

This doesn't seem like an unusual thing to want to do, so I'd like to
know what other ceph folk are using (and, if you like, the
numbers you
get from the benchmarkers)...?

Thanks,

Matthew


--
 The Wellcome Sanger Institute is operated by Genome Research
 Limited, a charity registered in England with number 1021457 and a
 company registered in England with number 2742969, whose registered
 office is 215 Euston Road, London, NW1 2BE.
___
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com





--
May the most significant bit of your life be positive.
___
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

[Dan van der Ster]

Do you have the startup banners for mds.cccephadm14 and 15? It sure
looks like they were running 12.2.2 with the "not writeable with
daemon features" error.

-- dan

On Wed, Mar 28, 2018 at 3:12 PM, adrien.geor...@cc.in2p3.fr
 wrote:
> Hi,
>
> All Ceph services were in 12.2.4 version.
>
> Adrien
>
>
> Le 28/03/2018 à 14:47, Dan van der Ster a écrit :
>>
>> Hi,
>>
>> Which versions were those MDS's before and after the restarted standby
>> MDS?
>>
>> Cheers, Dan
>>
>>
>>
>> On Wed, Mar 28, 2018 at 11:11 AM, adrien.geor...@cc.in2p3.fr
>>  wrote:
>>>
>>> Hi,
>>>
>>> I just had the same issue with our 12.2.4 cluster but not during the
>>> upgrade.
>>> One of our 3 monitors restarted (the one with a standby MDS) and the 2
>>> others active MDS killed themselves :
>>>
>>> 2018-03-28 09:36:24.376888 7f910bc0f700  0 mds.cccephadm14 handle_mds_map
>>> mdsmap compatset compat={},rocompat={},incompat={1=base v0.20,2=client
>>> writeable ranges,3=default file layouts on dirs,4=dir inode in separate
>>> object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
>>> anchor
>>> table,9=file layout v2} not writeable with daemon features
>>> compat={},rocompat={},incompat={1=base v0.20,2=client writeable
>>> ranges,3=default file layouts on dirs,4=dir inode in separate
>>> object,5=mds
>>> uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline
>>> data,8=file layout v2}, killing myself
>>> 2018-03-28 09:36:24.376903 7f910bc0f700  1 mds.cccephadm14 suicide.
>>> wanted
>>> state up:active
>>> 2018-03-28 09:36:25.379607 7f910bc0f700  1 mds.1.62 shutdown: shutting
>>> down
>>> rank 1
>>>
>>>
>>> 2018-03-28 09:36:24.375867 7fad455bf700  0 mds.cccephadm15 handle_mds_map
>>> mdsmap compatset compat={},rocompat={},incompat={1=base v0.20,2=client
>>> writeable ranges,3=default file layouts on dirs,4=dir inode in separate
>>> object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
>>> anchor
>>> table,9=file layout v2} not writeable with daemon features
>>> compat={},rocompat={},incompat={1=base v0.20,2=client writeable
>>> ranges,3=default file layouts on dirs,4=dir inode in separate
>>> object,5=mds
>>> uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline
>>> data,8=file layout v2}, killing myself
>>> 2018-03-28 09:36:24.375883 7fad455bf700  1 mds.cccephadm15 suicide.
>>> wanted
>>> state up:active
>>> 2018-03-28 09:36:25.377633 7fad455bf700  1 mds.0.50 shutdown: shutting
>>> down
>>> rank 0
>>>
>>> I had to restart manually the MDS services to get it works.
>>>
>>> Adrien
>>>
>>>
>>> Le 21/03/2018 à 11:37, Martin Palma a écrit :

 Just run into this problem on our production cluster

 It would have been nice if the release notes of 12.2.4 had been
 adapted to inform user about this.

 Best,
 Martin

 On Wed, Mar 14, 2018 at 9:53 PM, Gregory Farnum 
 wrote:
>
> On Wed, Mar 14, 2018 at 12:41 PM, Lars Marowsky-Bree 
> wrote:
>>
>> On 2018-03-14T06:57:08, Patrick Donnelly  wrote:
>>
>>> Yes. But the real outcome is not "no MDS [is] active" but "some or
>>> all
>>> metadata I/O will pause" -- and there is no avoiding that. During an
>>> MDS upgrade, a standby must take over the MDS being shutdown (and
>>> upgraded).  During takeover, metadata I/O will briefly pause as the
>>> rank is unavailable. (Specifically, no other rank can obtains locks
>>> or
>>> communicate with the "failed" rank; so metadata I/O will necessarily
>>> pause until a standby takes over.) Single active vs. multiple active
>>> upgrade makes little difference in this outcome.
>>
>> Fair, except that there's no standby MDS at this time in case the
>> update
>> goes wrong.
>>
 Is another approach theoretically feasible? Have the updated MDS
 only
 go
 into the incompatible mode once there's a quorum of new ones
 available,
 or something?
>>>
>>> I believe so, yes. That option wasn't explored for this patch because
>>> it was just disambiguating the compatibility flags and the full
>>> side-effects weren't realized.
>>
>> Would such a patch be accepted if we ended up pursuing this? Any
>> suggestions on how to best go about this?
>
> It'd be ugly, but you'd have to set it up so that
> * new MDSes advertise the old set of required values
> * but can identify when all the MDSes are new
> * then mark somewhere that they can use the correct values
> * then switch to the proper requirements
>
> I don't remember the details of this CompatSet code any more, and it's
> definitely made trickier by the MDS having no permanent local state.
> Since we do luckily have both the IDs and the strings, you might be
> able to do something in the MDSMonitor to identify whether booting
> MDSes have "too-old", "old-featureset-but-support-new-feature", or
> "new, correct feature a

Re: [ceph-users] What do you use to benchmark your rgw?

[David Byte]

I use cosbench (the last rc works well enough). I can get multiple GB/s from my 
6 node cluster with 2 RGWs.

David Byte
Sr. Technical Strategist
IHV Alliances and Embedded
SUSE

Sent from my iPhone. Typos are Apple's fault.

On Mar 28, 2018, at 5:26 AM, Janne Johansson 
mailto:icepic...@gmail.com>> wrote:

s3cmd and cli version of cyberduck to test it end-to-end using parallelism if 
possible.

Getting some 100MB/s at most, from 500km distance over https against 5*radosgw 
behind HAProxy.


2018-03-28 11:17 GMT+02:00 Matthew Vernon 
mailto:m...@sanger.ac.uk>>:
Hi,

What are people here using to benchmark their S3 service (i.e. the rgw)?
rados bench is great for some things, but doesn't tell me about what
performance I can get from my rgws.

It seems that there used to be rest-bench, but that isn't in Jewel
AFAICT; I had a bit of a look at cosbench but it looks fiddly to set up
and a bit under-maintained (the most recent version doesn't work out of
the box, and the PR to fix that has been languishing for a while).

This doesn't seem like an unusual thing to want to do, so I'd like to
know what other ceph folk are using (and, if you like, the numbers you
get from the benchmarkers)...?

Thanks,

Matthew


--
 The Wellcome Sanger Institute is operated by Genome Research
 Limited, a charity registered in England with number 1021457 and a
 company registered in England with number 2742969, whose registered
 office is 215 Euston Road, London, NW1 2BE.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



--
May the most significant bit of your life be positive.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

[adrien.geor...@cc.in2p3.fr]

Hi,

All Ceph services were in 12.2.4 version.

Adrien

Le 28/03/2018 à 14:47, Dan van der Ster a écrit :

Hi,

Which versions were those MDS's before and after the restarted standby MDS?

Cheers, Dan



On Wed, Mar 28, 2018 at 11:11 AM, adrien.geor...@cc.in2p3.fr
 wrote:

Hi,

I just had the same issue with our 12.2.4 cluster but not during the
upgrade.
One of our 3 monitors restarted (the one with a standby MDS) and the 2
others active MDS killed themselves :

2018-03-28 09:36:24.376888 7f910bc0f700  0 mds.cccephadm14 handle_mds_map
mdsmap compatset compat={},rocompat={},incompat={1=base v0.20,2=client
writeable ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor
table,9=file layout v2} not writeable with daemon features
compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds
uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline
data,8=file layout v2}, killing myself
2018-03-28 09:36:24.376903 7f910bc0f700  1 mds.cccephadm14 suicide. wanted
state up:active
2018-03-28 09:36:25.379607 7f910bc0f700  1 mds.1.62 shutdown: shutting down
rank 1


2018-03-28 09:36:24.375867 7fad455bf700  0 mds.cccephadm15 handle_mds_map
mdsmap compatset compat={},rocompat={},incompat={1=base v0.20,2=client
writeable ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor
table,9=file layout v2} not writeable with daemon features
compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds
uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline
data,8=file layout v2}, killing myself
2018-03-28 09:36:24.375883 7fad455bf700  1 mds.cccephadm15 suicide. wanted
state up:active
2018-03-28 09:36:25.377633 7fad455bf700  1 mds.0.50 shutdown: shutting down
rank 0

I had to restart manually the MDS services to get it works.

Adrien


Le 21/03/2018 à 11:37, Martin Palma a écrit :

Just run into this problem on our production cluster

It would have been nice if the release notes of 12.2.4 had been
adapted to inform user about this.

Best,
Martin

On Wed, Mar 14, 2018 at 9:53 PM, Gregory Farnum 
wrote:

On Wed, Mar 14, 2018 at 12:41 PM, Lars Marowsky-Bree 
wrote:

On 2018-03-14T06:57:08, Patrick Donnelly  wrote:


Yes. But the real outcome is not "no MDS [is] active" but "some or all
metadata I/O will pause" -- and there is no avoiding that. During an
MDS upgrade, a standby must take over the MDS being shutdown (and
upgraded).  During takeover, metadata I/O will briefly pause as the
rank is unavailable. (Specifically, no other rank can obtains locks or
communicate with the "failed" rank; so metadata I/O will necessarily
pause until a standby takes over.) Single active vs. multiple active
upgrade makes little difference in this outcome.

Fair, except that there's no standby MDS at this time in case the update
goes wrong.


Is another approach theoretically feasible? Have the updated MDS only
go
into the incompatible mode once there's a quorum of new ones
available,
or something?

I believe so, yes. That option wasn't explored for this patch because
it was just disambiguating the compatibility flags and the full
side-effects weren't realized.

Would such a patch be accepted if we ended up pursuing this? Any
suggestions on how to best go about this?

It'd be ugly, but you'd have to set it up so that
* new MDSes advertise the old set of required values
* but can identify when all the MDSes are new
* then mark somewhere that they can use the correct values
* then switch to the proper requirements

I don't remember the details of this CompatSet code any more, and it's
definitely made trickier by the MDS having no permanent local state.
Since we do luckily have both the IDs and the strings, you might be
able to do something in the MDSMonitor to identify whether booting
MDSes have "too-old", "old-featureset-but-support-new-feature", or
"new, correct feature advertising" and then either massage that
incoming message down to the "old-featureset-but-support-new-feature"
(if not all the MDSes are new) or do an auto-upgrade of the required
features in the map. And you might also need compatibility code in the
MDS to make sure it sends out the appropriate bits on connection, but
I *think* the CompatSet checks are only done on the monitor and when
an MDS receives an MDSMap.
-Greg
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.co

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

[Dan van der Ster]

Hi,

Which versions were those MDS's before and after the restarted standby MDS?

Cheers, Dan



On Wed, Mar 28, 2018 at 11:11 AM, adrien.geor...@cc.in2p3.fr
 wrote:
> Hi,
>
> I just had the same issue with our 12.2.4 cluster but not during the
> upgrade.
> One of our 3 monitors restarted (the one with a standby MDS) and the 2
> others active MDS killed themselves :
>
> 2018-03-28 09:36:24.376888 7f910bc0f700  0 mds.cccephadm14 handle_mds_map
> mdsmap compatset compat={},rocompat={},incompat={1=base v0.20,2=client
> writeable ranges,3=default file layouts on dirs,4=dir inode in separate
> object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor
> table,9=file layout v2} not writeable with daemon features
> compat={},rocompat={},incompat={1=base v0.20,2=client writeable
> ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds
> uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline
> data,8=file layout v2}, killing myself
> 2018-03-28 09:36:24.376903 7f910bc0f700  1 mds.cccephadm14 suicide. wanted
> state up:active
> 2018-03-28 09:36:25.379607 7f910bc0f700  1 mds.1.62 shutdown: shutting down
> rank 1
>
>
> 2018-03-28 09:36:24.375867 7fad455bf700  0 mds.cccephadm15 handle_mds_map
> mdsmap compatset compat={},rocompat={},incompat={1=base v0.20,2=client
> writeable ranges,3=default file layouts on dirs,4=dir inode in separate
> object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor
> table,9=file layout v2} not writeable with daemon features
> compat={},rocompat={},incompat={1=base v0.20,2=client writeable
> ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds
> uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline
> data,8=file layout v2}, killing myself
> 2018-03-28 09:36:24.375883 7fad455bf700  1 mds.cccephadm15 suicide. wanted
> state up:active
> 2018-03-28 09:36:25.377633 7fad455bf700  1 mds.0.50 shutdown: shutting down
> rank 0
>
> I had to restart manually the MDS services to get it works.
>
> Adrien
>
>
> Le 21/03/2018 à 11:37, Martin Palma a écrit :
>>
>> Just run into this problem on our production cluster
>>
>> It would have been nice if the release notes of 12.2.4 had been
>> adapted to inform user about this.
>>
>> Best,
>> Martin
>>
>> On Wed, Mar 14, 2018 at 9:53 PM, Gregory Farnum 
>> wrote:
>>>
>>> On Wed, Mar 14, 2018 at 12:41 PM, Lars Marowsky-Bree 
>>> wrote:

 On 2018-03-14T06:57:08, Patrick Donnelly  wrote:

> Yes. But the real outcome is not "no MDS [is] active" but "some or all
> metadata I/O will pause" -- and there is no avoiding that. During an
> MDS upgrade, a standby must take over the MDS being shutdown (and
> upgraded).  During takeover, metadata I/O will briefly pause as the
> rank is unavailable. (Specifically, no other rank can obtains locks or
> communicate with the "failed" rank; so metadata I/O will necessarily
> pause until a standby takes over.) Single active vs. multiple active
> upgrade makes little difference in this outcome.

 Fair, except that there's no standby MDS at this time in case the update
 goes wrong.

>> Is another approach theoretically feasible? Have the updated MDS only
>> go
>> into the incompatible mode once there's a quorum of new ones
>> available,
>> or something?
>
> I believe so, yes. That option wasn't explored for this patch because
> it was just disambiguating the compatibility flags and the full
> side-effects weren't realized.

 Would such a patch be accepted if we ended up pursuing this? Any
 suggestions on how to best go about this?
>>>
>>> It'd be ugly, but you'd have to set it up so that
>>> * new MDSes advertise the old set of required values
>>> * but can identify when all the MDSes are new
>>> * then mark somewhere that they can use the correct values
>>> * then switch to the proper requirements
>>>
>>> I don't remember the details of this CompatSet code any more, and it's
>>> definitely made trickier by the MDS having no permanent local state.
>>> Since we do luckily have both the IDs and the strings, you might be
>>> able to do something in the MDSMonitor to identify whether booting
>>> MDSes have "too-old", "old-featureset-but-support-new-feature", or
>>> "new, correct feature advertising" and then either massage that
>>> incoming message down to the "old-featureset-but-support-new-feature"
>>> (if not all the MDSes are new) or do an auto-upgrade of the required
>>> features in the map. And you might also need compatibility code in the
>>> MDS to make sure it sends out the appropriate bits on connection, but
>>> I *think* the CompatSet checks are only done on the monitor and when
>>> an MDS receives an MDSMap.
>>> -Greg
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>> __

Re: [ceph-users] Fwd: High IOWait Issue

[Alex Gorbachev]

On Mon, Mar 26, 2018 at 11:40 PM, Sam Huracan 
wrote:

> Hi,
>
> We are using Raid cache mode Writeback for SSD journal, I consider this is
> reason of utilization of SSD journal is so low.
> Is it  true? Anybody has experience with this matter, plz confirm.
>


I turn the writeback mode off for the SSDs, as this seems to make the
controller cache a bottleneck.

--
Alex Gorbachev
Storcium




>
> Thanks
>
> 2018-03-26 23:00 GMT+07:00 Sam Huracan :
>
>> Thanks for your information.
>> Here is result when I run atop on 1 Ceph HDD host:
>> http://prntscr.com/iwmc86
>>
>> There is some disk busy with over 100%, but the SSD journal (SSD) use
>> only 3%, is it normal? Is there any way to optimize using of SSD journal?
>> Could you give me some keyword?
>>
>> Here is configuration of Ceph HDD Host:
>> Dell PowerEdge R730xd Server Quantity
>> PE R730/xd Motherboard 1
>> Intel Xeon E5-2620 v4 2.1GHz,20M Cache,8.0GT/s QPI,Turbo,HT,8C/16T (85W)
>> Max Mem 2133MHz 1
>> 16GB RDIMM, 2400MT/s, Dual Rank, x8 Data Width 2
>> 300GB 15K RPM SAS 12Gbps 2.5in Flex Bay Hard Drive - OS Drive (RAID 1) 2
>> 4TB 7.2K RPM NLSAS 12Gbps 512n 3.5in Hot-plug Hard Drive - OSD Drive 7
>> 200GB Solid State Drive SATA Mix Use MLC 6Gbps 2.5in Hot-plug Drive -
>> Journal Drive (RAID 1) 2
>> PERC H730 Integrated RAID Controller, 1GB Cache *(we are using Writeback
>> mode)* 1
>> Dual, Hot-plug, Redundant Power Supply (1+1), 750W 1
>> Broadcom 5720 QP 1Gb Network Daughter Card 1
>> QLogic 57810 Dual Port 10Gb Direct Attach/SFP+ Network Adapter 1
>>
>> For some reasons, we can't configure Jumbo Frame in this cluster. We'll
>> refer your suggest about scrub.
>>
>>
>> 2018-03-26 7:41 GMT+07:00 Christian Balzer :
>>
>>>
>>> Hello,
>>>
>>> in general and as reminder for others, the more information you supply,
>>> the more likely are people to answer and answer with actually pertinent
>>> information.
>>> Since you haven't mentioned the hardware (actual HDD/SSD models, CPU/RAM,
>>> controllers, etc) we're still missing a piece of the puzzle that could be
>>> relevant.
>>>
>>> But given what we have some things are more likely than others.
>>> Also, an inline 90KB screenshot of a TEXT iostat output is a bit of a
>>> no-no, never mind that atop instead of top from the start would have
>>> given
>>> you and us much more insight.
>>>
>>> On Sun, 25 Mar 2018 14:35:57 +0700 Sam Huracan wrote:
>>>
>>> > Thank you all.
>>> >
>>> > 1. Here is my ceph.conf file:
>>> > https://pastebin.com/xpF2LUHs
>>> >
>>> As Lazlo noted (and it matches your iostat output beautifully), tuning
>>> down scrubs is likely going to have an immediate beneficial impact, as
>>> deep-scrubs in particular are VERY disruptive and I/O intense operations.
>>>
>>> However the "osd scrub sleep = 0.1" may make things worse in certain
>>> Jewel
>>> versions, as they all went through the unified queue and this would cause
>>> a sleep for ALL operations, not just the scrub ones.
>>> I can't remember when this was fixed and the changelog is of no help, so
>>> hopefully somebody who knows will pipe up.
>>> If in doubt of course, experiment.
>>>
>>> In addition to that, if you have low usage times, set
>>> your osd_scrub_(start|end)_hour accordingly and also check the ML
>>> archives
>>> for other scrub scheduling tips.
>>>
>>> I'd also leave these:
>>> filestore max sync interval = 100
>>> filestore min sync interval = 50
>>> filestore queue max ops  = 5000
>>> filestore queue committing max ops  = 5000
>>> journal max write entries  = 1000
>>> journal queue max ops  = 5000
>>>
>>> at their defaults, playing with those parameters requires a good
>>> understanding of how Ceph filestore works AND usually only makes sense
>>> with SSD/NVMe setups.
>>> Especially the first 2 could lead to quite the IO pileup.
>>>
>>>
>>> > 2. Here is result from ceph -s:
>>> > root@ceph1:/etc/ceph# ceph -s
>>> > cluster 31154d30-b0d3-4411-9178-0bbe367a5578
>>> >  health HEALTH_OK
>>> >  monmap e3: 3 mons at {ceph1=
>>> > 10.0.30.51:6789/0,ceph2=10.0.30.52:6789/0,ceph3=10.0.30.53:6789/0}
>>> > election epoch 18, quorum 0,1,2 ceph1,ceph2,ceph3
>>> >  osdmap e2473: 63 osds: 63 up, 63 in
>>> > flags sortbitwise,require_jewel_osds
>>> >   pgmap v34069952: 4096 pgs, 6 pools, 21534 GB data, 5696 kobjects
>>> > 59762 GB used, 135 TB / 194 TB avail
>>> > 4092 active+clean
>>> >2 active+clean+scrubbing
>>> >2 active+clean+scrubbing+deep
>>> >   client io 36096 kB/s rd, 41611 kB/s wr, 1643 op/s rd, 1634 op/s wr
>>> >
>>> See above about deep-scrub, which will read ALL the objects of the PG
>>> being scrubbed and thus not only saturates the OSDs involved with reads
>>> but ALSO dirties the pagecache with cold objects, making other reads on
>>> the nodes slow by requiring them to hit the disks, too.
>>>
>>> It would be interesting to see a "ceph -s" when your cluster is bu

Re: [ceph-users] What do you use to benchmark your rgw?

[Janne Johansson]

s3cmd and cli version of cyberduck to test it end-to-end using parallelism
if possible.

Getting some 100MB/s at most, from 500km distance over https against
5*radosgw behind HAProxy.


2018-03-28 11:17 GMT+02:00 Matthew Vernon :

> Hi,
>
> What are people here using to benchmark their S3 service (i.e. the rgw)?
> rados bench is great for some things, but doesn't tell me about what
> performance I can get from my rgws.
>
> It seems that there used to be rest-bench, but that isn't in Jewel
> AFAICT; I had a bit of a look at cosbench but it looks fiddly to set up
> and a bit under-maintained (the most recent version doesn't work out of
> the box, and the PR to fix that has been languishing for a while).
>
> This doesn't seem like an unusual thing to want to do, so I'd like to
> know what other ceph folk are using (and, if you like, the numbers you
> get from the benchmarkers)...?
>
> Thanks,
>
> Matthew
>
>
> --
>  The Wellcome Sanger Institute is operated by Genome Research
>  Limited, a charity registered in England with number 1021457 and a
>  company registered in England with number 2742969, whose registered
>  office is 215 Euston Road, London, NW1 2BE.
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 
May the most significant bit of your life be positive.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Getting a public file from radosgw

[Marc Roos]

 
Yes great!!! Thanks

https://192.168.1.114:7480/test:test/test.txt




-Original Message-
From: Matt Benjamin [mailto:mbenj...@redhat.com] 
Sent: woensdag 28 maart 2018 14:23
To: Marc Roos
Cc: ceph-users
Subject: Re: [ceph-users] Getting a public file from radosgw

Hi Marc,

It looks to me, from the naming of the test users here, as if you are 
being guided by the information here:

  http://docs.ceph.com/docs/master/radosgw/multitenancy/

which I think is the right starting point.  The distinction is that the 
"test" in "test$tester1" and "test2" in "test2$tester3" are distinct 
tenants, and the bit I -think- you're looking for is the syntax for 
identifying the tenant explicitly in a URL.

"In case of S3, a colon character is used to separate tenant and bucket"

Matt

On Wed, Mar 28, 2018 at 8:14 AM, Marc Roos  
wrote:
>
> I created these users with radosgw-admin [
> "test2$tester3",
> "test$tester1",
> "test$tester2"
> ]
>
> test$tester1 has created bucket test, and in this bucket file test.txt
>
> test$tester2 cannot create a bucket test
>
> test2$tester3 has created bucket test, and in this bucket file 
> test.txt
>
> There must be something in this http://objects.local/test/test.txt 
> that identifies if I would like to have the test.txt file of $tester1 
> or $tester3? Maybe something like 
> http://objects.local/test/test.txt?ACCESSKEY=tester3key ?
>
> This is what I get now
> 
> NoSuchBucket
> test
> tx00101-005abb84b5-2d8dc8-default d> 2d8dc8-default-default
> 
>
>
>
> -Original Message-
> From: Wido den Hollander [mailto:w...@42on.com]
> Sent: woensdag 28 maart 2018 13:45
> To: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] Getting a public file from radosgw
>
>
>
> On 03/28/2018 12:53 PM, Marc Roos wrote:
>> There must be in the wget/curl url some userid/unique identification 
>> not? Otherwise it could be anybodies test bucket/file?
>
> Let's say you have 'objects.local' as hostname.
>
> You can then fetch the object:
>
> - http://test.objects.local/test.txt
> - http://objects.local/test/test.txt
>
> Both should work.
>
> Wido
>
>>
>>
>> [marc@os0 ~]$ s3cmd info s3://test/test.txt s3://test/test.txt
>> (object):
>>File size: 15
>>Last mod:  Wed, 28 Mar 2018 10:49:00 GMT
>>MIME type: text/plain
>>Storage:   STANDARD
>>MD5 sum:   120ea8a25e5d487bf68b5f7096440019
>>SSE:   none
>>policy:none
>>cors:  none
>>ACL:   *anon*: READ
>>ACL:   Test1 User: FULL_CONTROL
>>URL:   http://192.168.1.111:7480/test/test.txt
>> [marc@os0 ~]$ s3cmd info s3://test/
>> s3://test/ (bucket):
>>Location:  us-east-1
>>Payer: BucketOwner
>>Expiration Rule: none
>>policy:none
>>cors:  none
>>ACL:   *anon*: READ
>>ACL:   Test1 User: FULL_CONTROL
>>URL:   http://192.168.1.111:7480/test/
>>
>>
>>
>>
>>
>> -Original Message-
>> From: Wido den Hollander [mailto:w...@42on.com]
>> Sent: woensdag 28 maart 2018 12:05
>> To: ceph-users@lists.ceph.com
>> Subject: Re: [ceph-users] Getting a public file from radosgw
>>
>>
>>
>> On 03/28/2018 11:59 AM, Marc Roos wrote:
>>>
>>>
>>>
>>>
>>>
>>> Do you have maybe some pointers, or example? ;)
>>>
>>
>> When you upload using s3cmd try using the -P flag, that will set the 
>> public-read ACL.
>>
>> Wido
>>
>>> This XML file does not appear to have any style information 
>>> associated
>>
>>> with it. The document tree is shown below.
>>>
>>>
>> NoSuchBuckettest> I
>> d>
>>>
>> tx000bb-005abb6720-2d8dc8-default
>> 2
>> d8
>>> dc8-default-default
>>>
>>>
>>> -Original Message-
>>> From: Jaroslaw Owsiewski [mailto:jaroslaw.owsiew...@allegro.pl]
>>> Sent: woensdag 28 maart 2018 11:49
>>> To: Marc Roos
>>> Cc: ceph-users
>>> Subject: Re: [ceph-users] Getting a public file from radosgw
>>>
>>> Yes. File should have "public-read" permissions.
>>>
>>> Regards
>>> --
>>> Jarek
>>>
>>> 2018-03-28 11:41 GMT+02:00 Marc Roos :
>>>
>>>
>>>
>>>
>>>  Is it possible to get a file directly from a bucket without
>>>  authenticating
>>>
>>>  With something like wget
>>> https://radosgw.example.com/user/bucket/file
>>>   or
>>>  https://radosgw.example.com/uniqueid
>>> 
>>>  ___
>>>  ceph-users mailing list
>>>  ceph-users@lists.ceph.com
>>>  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> 
>>>
>>>
>>>
>>>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>> __

Re: [ceph-users] Getting a public file from radosgw

[Matt Benjamin]

Hi Marc,

It looks to me, from the naming of the test users here, as if you are
being guided by the information here:

  http://docs.ceph.com/docs/master/radosgw/multitenancy/

which I think is the right starting point.  The distinction is that
the "test" in "test$tester1" and "test2" in "test2$tester3" are
distinct tenants, and the bit I -think- you're looking for is the
syntax for identifying the tenant explicitly in a URL.

"In case of S3, a colon character is used to separate tenant and bucket"

Matt

On Wed, Mar 28, 2018 at 8:14 AM, Marc Roos  wrote:
>
> I created these users with radosgw-admin
> [
> "test2$tester3",
> "test$tester1",
> "test$tester2"
> ]
>
> test$tester1 has created bucket test, and in this bucket file test.txt
>
> test$tester2 cannot create a bucket test
>
> test2$tester3 has created bucket test, and in this bucket file test.txt
>
> There must be something in this http://objects.local/test/test.txt that
> identifies if I would like to have the test.txt file of $tester1 or
> $tester3? Maybe something like
> http://objects.local/test/test.txt?ACCESSKEY=tester3key ?
>
> This is what I get now
> 
> NoSuchBucket
> test
> tx00101-005abb84b5-2d8dc8-default
> 2d8dc8-default-default
> 
>
>
>
> -Original Message-
> From: Wido den Hollander [mailto:w...@42on.com]
> Sent: woensdag 28 maart 2018 13:45
> To: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] Getting a public file from radosgw
>
>
>
> On 03/28/2018 12:53 PM, Marc Roos wrote:
>> There must be in the wget/curl url some userid/unique identification
>> not? Otherwise it could be anybodies test bucket/file?
>
> Let's say you have 'objects.local' as hostname.
>
> You can then fetch the object:
>
> - http://test.objects.local/test.txt
> - http://objects.local/test/test.txt
>
> Both should work.
>
> Wido
>
>>
>>
>> [marc@os0 ~]$ s3cmd info s3://test/test.txt s3://test/test.txt
>> (object):
>>File size: 15
>>Last mod:  Wed, 28 Mar 2018 10:49:00 GMT
>>MIME type: text/plain
>>Storage:   STANDARD
>>MD5 sum:   120ea8a25e5d487bf68b5f7096440019
>>SSE:   none
>>policy:none
>>cors:  none
>>ACL:   *anon*: READ
>>ACL:   Test1 User: FULL_CONTROL
>>URL:   http://192.168.1.111:7480/test/test.txt
>> [marc@os0 ~]$ s3cmd info s3://test/
>> s3://test/ (bucket):
>>Location:  us-east-1
>>Payer: BucketOwner
>>Expiration Rule: none
>>policy:none
>>cors:  none
>>ACL:   *anon*: READ
>>ACL:   Test1 User: FULL_CONTROL
>>URL:   http://192.168.1.111:7480/test/
>>
>>
>>
>>
>>
>> -Original Message-
>> From: Wido den Hollander [mailto:w...@42on.com]
>> Sent: woensdag 28 maart 2018 12:05
>> To: ceph-users@lists.ceph.com
>> Subject: Re: [ceph-users] Getting a public file from radosgw
>>
>>
>>
>> On 03/28/2018 11:59 AM, Marc Roos wrote:
>>>
>>>
>>>
>>>
>>>
>>> Do you have maybe some pointers, or example? ;)
>>>
>>
>> When you upload using s3cmd try using the -P flag, that will set the
>> public-read ACL.
>>
>> Wido
>>
>>> This XML file does not appear to have any style information
>>> associated
>>
>>> with it. The document tree is shown below.
>>>
>>>
>> NoSuchBuckettest> d>
>>>
>> tx000bb-005abb6720-2d8dc8-default2
>> d8
>>> dc8-default-default
>>>
>>>
>>> -Original Message-
>>> From: Jaroslaw Owsiewski [mailto:jaroslaw.owsiew...@allegro.pl]
>>> Sent: woensdag 28 maart 2018 11:49
>>> To: Marc Roos
>>> Cc: ceph-users
>>> Subject: Re: [ceph-users] Getting a public file from radosgw
>>>
>>> Yes. File should have "public-read" permissions.
>>>
>>> Regards
>>> --
>>> Jarek
>>>
>>> 2018-03-28 11:41 GMT+02:00 Marc Roos :
>>>
>>>
>>>
>>>
>>>  Is it possible to get a file directly from a bucket without
>>>  authenticating
>>>
>>>  With something like wget
>>> https://radosgw.example.com/user/bucket/file
>>>   or
>>>  https://radosgw.example.com/uniqueid
>>> 
>>>  ___
>>>  ceph-users mailing list
>>>  ceph-users@lists.ceph.com
>>>  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> 
>>>
>>>
>>>
>>>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
> 

Re: [ceph-users] Getting a public file from radosgw

[Marc Roos]

 
I created these users with radosgw-admin
[
"test2$tester3",
"test$tester1",
"test$tester2"
]

test$tester1 has created bucket test, and in this bucket file test.txt

test$tester2 cannot create a bucket test

test2$tester3 has created bucket test, and in this bucket file test.txt

There must be something in this http://objects.local/test/test.txt that 
identifies if I would like to have the test.txt file of $tester1 or 
$tester3? Maybe something like 
http://objects.local/test/test.txt?ACCESSKEY=tester3key ?

This is what I get now

NoSuchBucket
test
tx00101-005abb84b5-2d8dc8-default
2d8dc8-default-default




-Original Message-
From: Wido den Hollander [mailto:w...@42on.com] 
Sent: woensdag 28 maart 2018 13:45
To: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Getting a public file from radosgw



On 03/28/2018 12:53 PM, Marc Roos wrote:
> There must be in the wget/curl url some userid/unique identification 
> not? Otherwise it could be anybodies test bucket/file?

Let's say you have 'objects.local' as hostname.

You can then fetch the object:

- http://test.objects.local/test.txt
- http://objects.local/test/test.txt

Both should work.

Wido

>  
> 
> [marc@os0 ~]$ s3cmd info s3://test/test.txt s3://test/test.txt 
> (object):
>File size: 15
>Last mod:  Wed, 28 Mar 2018 10:49:00 GMT
>MIME type: text/plain
>Storage:   STANDARD
>MD5 sum:   120ea8a25e5d487bf68b5f7096440019
>SSE:   none
>policy:none
>cors:  none
>ACL:   *anon*: READ
>ACL:   Test1 User: FULL_CONTROL
>URL:   http://192.168.1.111:7480/test/test.txt
> [marc@os0 ~]$ s3cmd info s3://test/
> s3://test/ (bucket):
>Location:  us-east-1
>Payer: BucketOwner
>Expiration Rule: none
>policy:none
>cors:  none
>ACL:   *anon*: READ
>ACL:   Test1 User: FULL_CONTROL
>URL:   http://192.168.1.111:7480/test/
> 
> 
> 
> 
> 
> -Original Message-
> From: Wido den Hollander [mailto:w...@42on.com]
> Sent: woensdag 28 maart 2018 12:05
> To: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] Getting a public file from radosgw
> 
> 
> 
> On 03/28/2018 11:59 AM, Marc Roos wrote:
>>  
>>
>>
>>
>>
>> Do you have maybe some pointers, or example? ;)
>>
> 
> When you upload using s3cmd try using the -P flag, that will set the 
> public-read ACL.
> 
> Wido
> 
>> This XML file does not appear to have any style information 
>> associated
> 
>> with it. The document tree is shown below.
>>   
>>
> NoSuchBuckettest d>
>>
> tx000bb-005abb6720-2d8dc8-default2
> d8
>> dc8-default-default
>>
>>
>> -Original Message-
>> From: Jaroslaw Owsiewski [mailto:jaroslaw.owsiew...@allegro.pl]
>> Sent: woensdag 28 maart 2018 11:49
>> To: Marc Roos
>> Cc: ceph-users
>> Subject: Re: [ceph-users] Getting a public file from radosgw
>>
>> Yes. File should have "public-read" permissions.
>>
>> Regards
>> --
>> Jarek
>>
>> 2018-03-28 11:41 GMT+02:00 Marc Roos :
>>
>>
>>
>>
>>  Is it possible to get a file directly from a bucket without
>>  authenticating
>>  
>>  With something like wget
>> https://radosgw.example.com/user/bucket/file
>>   or
>>  https://radosgw.example.com/uniqueid
>>  
>>  ___
>>  ceph-users mailing list
>>  ceph-users@lists.ceph.com
>>  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> 
>>  
>>
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Getting a public file from radosgw

[Wido den Hollander]

On 03/28/2018 12:53 PM, Marc Roos wrote:
> There must be in the wget/curl url some userid/unique identification 
> not? Otherwise it could be anybodies test bucket/file?

Let's say you have 'objects.local' as hostname.

You can then fetch the object:

- http://test.objects.local/test.txt
- http://objects.local/test/test.txt

Both should work.

Wido

>  
> 
> [marc@os0 ~]$ s3cmd info s3://test/test.txt
> s3://test/test.txt (object):
>File size: 15
>Last mod:  Wed, 28 Mar 2018 10:49:00 GMT
>MIME type: text/plain
>Storage:   STANDARD
>MD5 sum:   120ea8a25e5d487bf68b5f7096440019
>SSE:   none
>policy:none
>cors:  none
>ACL:   *anon*: READ
>ACL:   Test1 User: FULL_CONTROL
>URL:   http://192.168.1.111:7480/test/test.txt
> [marc@os0 ~]$ s3cmd info s3://test/
> s3://test/ (bucket):
>Location:  us-east-1
>Payer: BucketOwner
>Expiration Rule: none
>policy:none
>cors:  none
>ACL:   *anon*: READ
>ACL:   Test1 User: FULL_CONTROL
>URL:   http://192.168.1.111:7480/test/
> 
> 
> 
> 
> 
> -Original Message-
> From: Wido den Hollander [mailto:w...@42on.com] 
> Sent: woensdag 28 maart 2018 12:05
> To: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] Getting a public file from radosgw
> 
> 
> 
> On 03/28/2018 11:59 AM, Marc Roos wrote:
>>  
>>
>>
>>
>>
>> Do you have maybe some pointers, or example? ;)
>>
> 
> When you upload using s3cmd try using the -P flag, that will set the
> public-read ACL.
> 
> Wido
> 
>> This XML file does not appear to have any style information associated 
> 
>> with it. The document tree is shown below.
>>   
>>
> NoSuchBuckettest
>>
> tx000bb-005abb6720-2d8dc8-default2d8
>> dc8-default-default
>>
>>
>> -Original Message-
>> From: Jaroslaw Owsiewski [mailto:jaroslaw.owsiew...@allegro.pl] 
>> Sent: woensdag 28 maart 2018 11:49
>> To: Marc Roos
>> Cc: ceph-users
>> Subject: Re: [ceph-users] Getting a public file from radosgw
>>
>> Yes. File should have "public-read" permissions.
>>
>> Regards
>> --
>> Jarek
>>
>> 2018-03-28 11:41 GMT+02:00 Marc Roos :
>>
>>
>>
>>
>>  Is it possible to get a file directly from a bucket without
>>  authenticating
>>  
>>  With something like wget 
>> https://radosgw.example.com/user/bucket/file 
>>   or
>>  https://radosgw.example.com/uniqueid 
>>  
>>  ___
>>  ceph-users mailing list
>>  ceph-users@lists.ceph.com
>>  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
>>  
>>  
>>
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] where is it possible download CentOS 7.5

[Jason Dillaman]

On Wed, Mar 28, 2018 at 7:33 AM, Brad Hubbard  wrote:
> On Wed, Mar 28, 2018 at 6:53 PM, Max Cuttins  wrote:
>> Il 27/03/2018 13:46, Brad Hubbard ha scritto:
>>
>>
>>
>> On Tue, Mar 27, 2018 at 9:12 PM, Max Cuttins  wrote:
>>>
>>> Hi Brad,
>>>
>>> that post was mine. I knew it quite well.
>>>
>>> That Post was about confirm the fact that minimum requirements written in
>>> the documentation really didn't exists.
>>>
>>> However I never asked if there is somewhere a place where is possible to
>>> download the DEV or the RC of Centos7.5.
>>> I was thinking about to join the community of tester and developers that
>>> are already testing Ceph on that "not ready" environment.
>>>
>>> In that POST these questions were not really made, so no answer where
>>> given.
>>
>>
>> From that thread.
>>
>> "The necessary kernel changes actually are included as part of 4.16-rc1
>> which is available now. We also offer a pre-built test kernel with the
>> necessary fixes here [1].
>>
>> [1] https://shaman.ceph.com/repos/kernel/ceph-iscsi-test/";
>>
>> I notice that URL is unavailable so maybe the real question should be why is
>> that kernel no longer available?
>>
>>
>> Yes, the link was broken and it seemed to me a misprint of old docs.
>> As all other stuffs described didn't exists already I thought that event
>> this Kernel test was not available (already or anymore).
>
> The link is fixed as of 12-18 hours ago and the kernel is available again.
>
>>
>>
>> There are plenty more available at
>> https://shaman.ceph.com/repos/kernel/testing/ but *I* can't tell you which
>> is relevant but perhaps someone else can.
>>
>>
>> However the 4.16 is almost ready to be released (shoulded had been already).
>> At this moment is just a double work use that kernel and after upgrade it to
>> the final one.
>
> OK, I guess you just need to wait (by your choice) then.
>
>>
>>
>>> I see that you talked also about other distribution. Well, I read around
>>> that Suse already implement iSCSI.
>>> However as far as I know (which is not so much), this distribution use
>>> modified kernel in order to let this work.
>>> And in order to use it it's needed  a dashboard that can handle these kind
>>> of differences (OpenAttic).
>>> I knew already OpenAttic is contributing in developing the next generation
>>> of the Ceph Dashboard (and this sound damn good!).
>>> However this also means to me that the official dashboard should not be
>>> talking about ISCSI at all (as every implementation of iSCSI are running on
>>> mod version).
>>>
>>> So these are the things I cannot figure out:
>>> Why is the iSCSI board on the CEPH official dashboard? (I could understand
>>> on OpenAttic which run on SUSE but not on the official one).
>>
>> Why do you believe it should not be?
>>
>>
>> Maybe I'm in wrong, but I guess that the dashboard manager expects to get
>> data/details/stats/config from a particular set of paths, components and
>> daemons which cannot be the same for all the ad-hoc implementation.
>> So there is a dashboard that show values for a component which is not
>> there (instead could be there something else but written in another way).
>> Every ad-hoc implementation (like OpenAttic) of course know where to find
>> data/details/stats/config for work with their implementation (so it's
>> understandable that they have board for iSCSI).
>> Right?
>
> Not as far as I'm concerned. See John's email on the subject in this thread.
>
>>
>>
>>> And why, in the official documentation, the minimu requirements to let
>>> iSCSI work, is to install CentOS7.5? Which doesn't exist? Is there a RC
>>> candidate which I can start to use?
>>
>>
>> But it doesn't say that, it says " RHEL/CentOS 7.5; Linux kernel v4.16 or
>> newer; or the Ceph iSCSI client test kernel". You seem to be ignoring the
>> "Ceph iSCSI client test kernel" part?
>>
>> Yes, the link was broken and it seemed to me a misprint of old docs.
>>
>> Moreover at first read I figure out that I needed both centOS7.5 AND kernel
>> 4.16. OR the kernel test.
>> Now you are telling me that all requirements are alternative. Which explain
>> to me why the documentation suggest just CentOS and not all others
>> distribution.
>> Also this sounds good.
>>
>> But I don't think that CentOS7.5 will use the kernel 4.16 ... so you are
>> telling me that new feature will be backported to the kernel 3.* ?
>
> Nope. I'm not part of the Red hat kernel team and don't have the
> influence to shape what they do.

The RHEL/CentOS 7.5 3.x-based kernel will have all the necessary bug fixes.

>> Is it right? So i don't need to upgrade the kernel If I'll use
>> RHEL/CentOS7.5 ?
>> This sound even better. I was a bit worried to don't use the mainstream
>> kernel of the distribution.
>
> We'll have to wait and see what ships.

See above.

>>
>>> And... if SUSE or even other distribution works already with iSCSI... why
>>> the documentation just doesn't reccomend these ones instead of RHEL or
>>> CENTOS?
>>
>> Becau

Re: [ceph-users] where is it possible download CentOS 7.5

[Brad Hubbard]

On Wed, Mar 28, 2018 at 6:53 PM, Max Cuttins  wrote:
> Il 27/03/2018 13:46, Brad Hubbard ha scritto:
>
>
>
> On Tue, Mar 27, 2018 at 9:12 PM, Max Cuttins  wrote:
>>
>> Hi Brad,
>>
>> that post was mine. I knew it quite well.
>>
>> That Post was about confirm the fact that minimum requirements written in
>> the documentation really didn't exists.
>>
>> However I never asked if there is somewhere a place where is possible to
>> download the DEV or the RC of Centos7.5.
>> I was thinking about to join the community of tester and developers that
>> are already testing Ceph on that "not ready" environment.
>>
>> In that POST these questions were not really made, so no answer where
>> given.
>
>
> From that thread.
>
> "The necessary kernel changes actually are included as part of 4.16-rc1
> which is available now. We also offer a pre-built test kernel with the
> necessary fixes here [1].
>
> [1] https://shaman.ceph.com/repos/kernel/ceph-iscsi-test/";
>
> I notice that URL is unavailable so maybe the real question should be why is
> that kernel no longer available?
>
>
> Yes, the link was broken and it seemed to me a misprint of old docs.
> As all other stuffs described didn't exists already I thought that event
> this Kernel test was not available (already or anymore).

The link is fixed as of 12-18 hours ago and the kernel is available again.

>
>
> There are plenty more available at
> https://shaman.ceph.com/repos/kernel/testing/ but *I* can't tell you which
> is relevant but perhaps someone else can.
>
>
> However the 4.16 is almost ready to be released (shoulded had been already).
> At this moment is just a double work use that kernel and after upgrade it to
> the final one.

OK, I guess you just need to wait (by your choice) then.

>
>
>> I see that you talked also about other distribution. Well, I read around
>> that Suse already implement iSCSI.
>> However as far as I know (which is not so much), this distribution use
>> modified kernel in order to let this work.
>> And in order to use it it's needed  a dashboard that can handle these kind
>> of differences (OpenAttic).
>> I knew already OpenAttic is contributing in developing the next generation
>> of the Ceph Dashboard (and this sound damn good!).
>> However this also means to me that the official dashboard should not be
>> talking about ISCSI at all (as every implementation of iSCSI are running on
>> mod version).
>>
>> So these are the things I cannot figure out:
>> Why is the iSCSI board on the CEPH official dashboard? (I could understand
>> on OpenAttic which run on SUSE but not on the official one).
>
> Why do you believe it should not be?
>
>
> Maybe I'm in wrong, but I guess that the dashboard manager expects to get
> data/details/stats/config from a particular set of paths, components and
> daemons which cannot be the same for all the ad-hoc implementation.
> So there is a dashboard that show values for a component which is not
> there (instead could be there something else but written in another way).
> Every ad-hoc implementation (like OpenAttic) of course know where to find
> data/details/stats/config for work with their implementation (so it's
> understandable that they have board for iSCSI).
> Right?

Not as far as I'm concerned. See John's email on the subject in this thread.

>
>
>> And why, in the official documentation, the minimu requirements to let
>> iSCSI work, is to install CentOS7.5? Which doesn't exist? Is there a RC
>> candidate which I can start to use?
>
>
> But it doesn't say that, it says " RHEL/CentOS 7.5; Linux kernel v4.16 or
> newer; or the Ceph iSCSI client test kernel". You seem to be ignoring the
> "Ceph iSCSI client test kernel" part?
>
> Yes, the link was broken and it seemed to me a misprint of old docs.
>
> Moreover at first read I figure out that I needed both centOS7.5 AND kernel
> 4.16. OR the kernel test.
> Now you are telling me that all requirements are alternative. Which explain
> to me why the documentation suggest just CentOS and not all others
> distribution.
> Also this sounds good.
>
> But I don't think that CentOS7.5 will use the kernel 4.16 ... so you are
> telling me that new feature will be backported to the kernel 3.* ?

Nope. I'm not part of the Red hat kernel team and don't have the
influence to shape what they do.

> Is it right? So i don't need to upgrade the kernel If I'll use
> RHEL/CentOS7.5 ?
> This sound even better. I was a bit worried to don't use the mainstream
> kernel of the distribution.

We'll have to wait and see what ships.

>
>> And... if SUSE or even other distribution works already with iSCSI... why
>> the documentation just doesn't reccomend these ones instead of RHEL or
>> CENTOS?
>
> Because that would be odd, to say the least. If the documentation is
> incorrect for CentOS then it was, at least at some point, thought to be
> correct and it probably will be correct again in the near future and, if
> not, we can review and correct it as necessary.
>
>
> Of co

Re: [ceph-users] Getting a public file from radosgw

[Marc Roos]

There must be in the wget/curl url some userid/unique identification 
not? Otherwise it could be anybodies test bucket/file?
 

[marc@os0 ~]$ s3cmd info s3://test/test.txt
s3://test/test.txt (object):
   File size: 15
   Last mod:  Wed, 28 Mar 2018 10:49:00 GMT
   MIME type: text/plain
   Storage:   STANDARD
   MD5 sum:   120ea8a25e5d487bf68b5f7096440019
   SSE:   none
   policy:none
   cors:  none
   ACL:   *anon*: READ
   ACL:   Test1 User: FULL_CONTROL
   URL:   http://192.168.1.111:7480/test/test.txt
[marc@os0 ~]$ s3cmd info s3://test/
s3://test/ (bucket):
   Location:  us-east-1
   Payer: BucketOwner
   Expiration Rule: none
   policy:none
   cors:  none
   ACL:   *anon*: READ
   ACL:   Test1 User: FULL_CONTROL
   URL:   http://192.168.1.111:7480/test/





-Original Message-
From: Wido den Hollander [mailto:w...@42on.com] 
Sent: woensdag 28 maart 2018 12:05
To: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Getting a public file from radosgw



On 03/28/2018 11:59 AM, Marc Roos wrote:
>  
> 
> 
> 
> 
> Do you have maybe some pointers, or example? ;)
> 

When you upload using s3cmd try using the -P flag, that will set the
public-read ACL.

Wido

> This XML file does not appear to have any style information associated 

> with it. The document tree is shown below.
>   
> 
NoSuchBuckettest
> 
tx000bb-005abb6720-2d8dc8-default2d8
> dc8-default-default
> 
> 
> -Original Message-
> From: Jaroslaw Owsiewski [mailto:jaroslaw.owsiew...@allegro.pl] 
> Sent: woensdag 28 maart 2018 11:49
> To: Marc Roos
> Cc: ceph-users
> Subject: Re: [ceph-users] Getting a public file from radosgw
> 
> Yes. File should have "public-read" permissions.
> 
> Regards
> --
> Jarek
> 
> 2018-03-28 11:41 GMT+02:00 Marc Roos :
> 
> 
> 
> 
>   Is it possible to get a file directly from a bucket without
>   authenticating
>   
>   With something like wget 
> https://radosgw.example.com/user/bucket/file 
>   or
>   https://radosgw.example.com/uniqueid 
>  
>   ___
>   ceph-users mailing list
>   ceph-users@lists.ceph.com
>   http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
>  
>   
> 
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Getting a public file from radosgw

[Wido den Hollander]

On 03/28/2018 11:59 AM, Marc Roos wrote:
>  
> 
> 
> 
> 
> Do you have maybe some pointers, or example? ;) 
> 

When you upload using s3cmd try using the -P flag, that will set the
public-read ACL.

Wido

> This XML file does not appear to have any style information associated 
> with it. The document tree is shown below.
>   
> NoSuchBuckettest
> tx000bb-005abb6720-2d8dc8-default2d8
> dc8-default-default
> 
> 
> -Original Message-
> From: Jaroslaw Owsiewski [mailto:jaroslaw.owsiew...@allegro.pl] 
> Sent: woensdag 28 maart 2018 11:49
> To: Marc Roos
> Cc: ceph-users
> Subject: Re: [ceph-users] Getting a public file from radosgw
> 
> Yes. File should have "public-read" permissions.
> 
> Regards
> --
> Jarek
> 
> 2018-03-28 11:41 GMT+02:00 Marc Roos :
> 
> 
> 
> 
>   Is it possible to get a file directly from a bucket without
>   authenticating
>   
>   With something like wget 
> https://radosgw.example.com/user/bucket/file 
>   or
>   https://radosgw.example.com/uniqueid 
>  
>   ___
>   ceph-users mailing list
>   ceph-users@lists.ceph.com
>   http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
>  
>   
> 
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Getting a public file from radosgw

[Marc Roos]

 




Do you have maybe some pointers, or example? ;) 

This XML file does not appear to have any style information associated 
with it. The document tree is shown below.
  
NoSuchBuckettest
tx000bb-005abb6720-2d8dc8-default2d8
dc8-default-default


-Original Message-
From: Jaroslaw Owsiewski [mailto:jaroslaw.owsiew...@allegro.pl] 
Sent: woensdag 28 maart 2018 11:49
To: Marc Roos
Cc: ceph-users
Subject: Re: [ceph-users] Getting a public file from radosgw

Yes. File should have "public-read" permissions.

Regards
--
Jarek

2018-03-28 11:41 GMT+02:00 Marc Roos :




Is it possible to get a file directly from a bucket without
authenticating

With something like wget 
https://radosgw.example.com/user/bucket/file 
  or
https://radosgw.example.com/uniqueid 
 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
 




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Multipart Failure SOLVED - Missing Pool not created automatically

[Ingo Reimann]

Hi all,

i was able to track down the problem:

our zone config contains a default-placement with a value for
"data_extra_pool". This pool, e.g. "dev.rgw.buckets.non-ec" did not exists
and could not be created automatically.

Logs show:
2018-03-28 11:26:33.151533 7f569b30e700  1 -- 10.197.115.31:0/388946122
--> 10.197.115.21:6789/0 -- pool_op(create pool 0 auid 0 tid 33 name
dev.rgw.buckets.non-ec v0) v4 -- 0x564e68394780 con 0
2018-03-28 11:26:33.151775 7f56f75cd700  1 -- 10.197.115.31:0/388946122
<== mon.0 10.197.115.21:6789/0 11  pool_op_reply(tid 33 (1) Operation
not permitted v1083) v1  43+0+0 (967791429 0 0) 0x564e68394780 con
0x564e66898000

I created the pool and that solved the problem.

Could anybody tell me, which caps are needed, that this could happen
automatically?

Best regards,

Ingo


-Ursprüngliche Nachricht-
Von: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] Im Auftrag von
Ingo Reimann
Gesendet: Freitag, 2. März 2018 14:15
An: ceph-users
Betreff: [ceph-users] Multipart Upload - POST fails

Hi,

we discovered some problem with our installation - Multipart upload is not
working.

What we did:
* tried upload with cyberduck as well as with script from
http://tracker.ceph.com/issues/12790
* tried against jewel gateways and luminous gateways from old cluster
* tried against 12.2.4 gateway with jewel-era cluster

Surprisingly this is no signature problem as in the issue above, instead I
get the following in the logs:

2018-03-02 13:59:04.927353 7fe2053ca700  1 == starting new request
req=0x7fe2053c42c0 =
2018-03-02 13:59:04.927383 7fe2053ca700  2 req 61:0.30::POST
/luminous-12-2-4/Data128MB::initializing for trans_id =
tx0003d-005a994a98-10c84997-default
2018-03-02 13:59:04.927396 7fe2053ca700 10 rgw api priority: s3=5
s3website=4
2018-03-02 13:59:04.927399 7fe2053ca700 10 host=cephrgw01.dunkel.de
2018-03-02 13:59:04.927422 7fe2053ca700 20 subdomain=
domain=cephrgw01.dunkel.de in_hosted_domain=1 in_hosted_domain_s3website=0
2018-03-02 13:59:04.927427 7fe2053ca700 20 final domain/bucket subdomain=
domain=cephrgw01.dunkel.de in_hosted_domain=1 in_hosted_domain_s3website=0
s->info.domain=cephrgw01.dunkel.de
s->info.request_uri=/luminous-12-2-4/Data128MB
2018-03-02 13:59:04.927447 7fe2053ca700 10 meta>>
HTTP_X_AMZ_CONTENT_SHA256
2018-03-02 13:59:04.927454 7fe2053ca700 10 meta>> HTTP_X_AMZ_DATE
2018-03-02 13:59:04.927459 7fe2053ca700 10 x>>
x-amz-content-sha256:254bcc3fc4f27172636df4bf32de9f107f620d559b20d760197e4
52b97453917
2018-03-02 13:59:04.927464 7fe2053ca700 10 x>> x-amz-date:20180302T125904Z
2018-03-02 13:59:04.927493 7fe2053ca700 20 get_handler
handler=22RGWHandler_REST_Obj_S3
2018-03-02 13:59:04.927500 7fe2053ca700 10
handler=22RGWHandler_REST_Obj_S3
2018-03-02 13:59:04.927505 7fe2053ca700  2 req 61:0.000152:s3:POST
/luminous-12-2-4/Data128MB::getting op 4
2018-03-02 13:59:04.927512 7fe2053ca700 10
op=28RGWInitMultipart_ObjStore_S3
2018-03-02 13:59:04.927514 7fe2053ca700  2 req 61:0.000161:s3:POST
/luminous-12-2-4/Data128MB:init_multipart:verifying requester
2018-03-02 13:59:04.927519 7fe2053ca700 20
rgw::auth::StrategyRegistry::s3_main_strategy_t: trying
rgw::auth::s3::AWSAuthStrategy
2018-03-02 13:59:04.927524 7fe2053ca700 20 rgw::auth::s3::AWSAuthStrategy:
trying rgw::auth::s3::S3AnonymousEngine
2018-03-02 13:59:04.927531 7fe2053ca700 20
rgw::auth::s3::S3AnonymousEngine denied with reason=-1
2018-03-02 13:59:04.927533 7fe2053ca700 20 rgw::auth::s3::AWSAuthStrategy:
trying rgw::auth::s3::LocalEngine
2018-03-02 13:59:04.927569 7fe2053ca700 10 v4 signature format =
48cc8c61a70dde17932d925f65f843116199c1ca10094db83e7de05bfbd57dc4
2018-03-02 13:59:04.927584 7fe2053ca700 10 v4 credential format =
8DGDGA57XL9YPM8DGEQQ/20180302/us-east-1/s3/aws4_request
2018-03-02 13:59:04.927587 7fe2053ca700 10 access key id =
8DGDGA57XL9YPM8DGEQQ
2018-03-02 13:59:04.927589 7fe2053ca700 10 credential scope =
20180302/us-east-1/s3/aws4_request
2018-03-02 13:59:04.927620 7fe2053ca700 10 canonical headers format =
content-type:application/octet-stream
date:Fri, 02 Mar 2018 12:59:04 GMT
host:cephrgw01.dunkel.de
x-amz-content-sha256:254bcc3fc4f27172636df4bf32de9f107f620d559b20d760197e4
52b97453917
x-amz-date:20180302T125904Z

2018-03-02 13:59:04.927634 7fe2053ca700 10 payload request hash =
254bcc3fc4f27172636df4bf32de9f107f620d559b20d760197e452b97453917
2018-03-02 13:59:04.927690 7fe2053ca700 10 canonical request = POST
/luminous-12-2-4/Data128MB
uploads=
content-type:application/octet-stream
date:Fri, 02 Mar 2018 12:59:04 GMT
host:cephrgw01.dunkel.de
x-amz-content-sha256:254bcc3fc4f27172636df4bf32de9f107f620d559b20d760197e4
52b97453917
x-amz-date:20180302T125904Z

content-type;date;host;x-amz-content-sha256;x-amz-date
254bcc3fc4f27172636df4bf32de9f107f620d559b20d760197e452b97453917
2018-03-02 13:59:04.927696 7fe2053ca700 10 canonical request hash =
54e9858263535b46a3c4e51b2ae5c1d0bf5e7a7690c5bba722eea749e7b936c4
2018-03-02 13:59:04.927716 7fe2053ca700 10 string 

[ceph-users] Getting a public file from radosgw

[Marc Roos]

Is it possible to get a file directly from a bucket without 
authenticating

With something like wget https://radosgw.example.com/user/bucket/file or 
https://radosgw.example.com/uniqueid
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Upgrading ceph and mapped rbds

[Götz Reinicke]

Hi, I bet I did read it somewhere already, but can’t remember where….

Our ceph 10.2. cluster is fin and healthy and I have a couple of rbds exported 
to some fileserver and a nfs server.

The upgrade to V 12.2 documentation is clear regarding upgrading/restarting all 
MONs first, after that, the OSDs.

My question is: How to proceed with the serves which map the rbds?

Upgrading these after the ceph MONs/OSDs?

May I notice some interruption to the RBDs resp. the SMB and NFS Services? The 
NFS hosts some VMs for Xen …


Thanks for feedback dun comments . Regards . Götz


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] What is in the mon leveldb?

[Wido den Hollander]

On 03/28/2018 01:34 AM, Tracy Reed wrote:
>>   health: HEALTH_WARN
>>   recovery 1230/13361271 objects misplaced (0.009%)
>>
>> and no recovery is happening. I'm not sure why. This hasn't happened
>> before. But the mon db had been growing since long before this
>> circumstance.
> 
> Hmmok, the recent trouble started a few days ago when we removed a
> node containing 4 OSDs from the cluster. The OSDs on that node were shut
> down but were not removed from the crush map. So apparently this has
> caused some issues. I just removed the OSDs properly and now there is
> recovery happening. Unfortunately it now says 30% of my objects are
> misplaced so I'm looking at 24 hours of recovery. Maybe the store.db
> will be smaller when it finally finishes.
> 

When all PGs are active+clean your store.db will start to shrink.

Which Ceph version are you on? And with which version was this cluster
started?

It could be that you have sub-optimal CRUSH tunables coming from a old
version.

There could be a lot of things happening here, but hard to tell.

But your MONs will shrink after the PGs are all active+clean.

Wido



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] What do you use to benchmark your rgw?

[Matthew Vernon]

Hi,

What are people here using to benchmark their S3 service (i.e. the rgw)?
rados bench is great for some things, but doesn't tell me about what
performance I can get from my rgws.

It seems that there used to be rest-bench, but that isn't in Jewel
AFAICT; I had a bit of a look at cosbench but it looks fiddly to set up
and a bit under-maintained (the most recent version doesn't work out of
the box, and the PR to fix that has been languishing for a while).

This doesn't seem like an unusual thing to want to do, so I'd like to
know what other ceph folk are using (and, if you like, the numbers you
get from the benchmarkers)...?

Thanks,

Matthew


-- 
 The Wellcome Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE. 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

[adrien.geor...@cc.in2p3.fr]

Hi,

I just had the same issue with our 12.2.4 cluster but not during the 
upgrade.
One of our 3 monitors restarted (the one with a standby MDS) and the 2 
others active MDS killed themselves :


2018-03-28 09:36:24.376888 7f910bc0f700  0 mds.cccephadm14 
handle_mds_map mdsmap compatset compat={},rocompat={},incompat={1=base 
v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir 
inode in separate object,5=mds uses versioned encoding,6=dirfrag is 
stored in omap,8=no anchor table,9=file layout v2} not writeable with 
daemon features compat={},rocompat={},incompat={1=base v0.20,2=client 
writeable ranges,3=default file layouts on dirs,4=dir inode in separate 
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds 
uses inline data,8=file layout v2}, killing myself
2018-03-28 09:36:24.376903 7f910bc0f700  1 mds.cccephadm14 suicide. 
wanted state up:active
2018-03-28 09:36:25.379607 7f910bc0f700  1 mds.1.62 shutdown: shutting 
down rank 1



2018-03-28 09:36:24.375867 7fad455bf700  0 mds.cccephadm15 
handle_mds_map mdsmap compatset compat={},rocompat={},incompat={1=base 
v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir 
inode in separate object,5=mds uses versioned encoding,6=dirfrag is 
stored in omap,8=no anchor table,9=file layout v2} not writeable with 
daemon features compat={},rocompat={},incompat={1=base v0.20,2=client 
writeable ranges,3=default file layouts on dirs,4=dir inode in separate 
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds 
uses inline data,8=file layout v2}, killing myself
2018-03-28 09:36:24.375883 7fad455bf700  1 mds.cccephadm15 suicide. 
wanted state up:active
2018-03-28 09:36:25.377633 7fad455bf700  1 mds.0.50 shutdown: shutting 
down rank 0


I had to restart manually the MDS services to get it works.

Adrien

Le 21/03/2018 à 11:37, Martin Palma a écrit :

Just run into this problem on our production cluster

It would have been nice if the release notes of 12.2.4 had been
adapted to inform user about this.

Best,
Martin

On Wed, Mar 14, 2018 at 9:53 PM, Gregory Farnum  wrote:

On Wed, Mar 14, 2018 at 12:41 PM, Lars Marowsky-Bree  wrote:

On 2018-03-14T06:57:08, Patrick Donnelly  wrote:


Yes. But the real outcome is not "no MDS [is] active" but "some or all
metadata I/O will pause" -- and there is no avoiding that. During an
MDS upgrade, a standby must take over the MDS being shutdown (and
upgraded).  During takeover, metadata I/O will briefly pause as the
rank is unavailable. (Specifically, no other rank can obtains locks or
communicate with the "failed" rank; so metadata I/O will necessarily
pause until a standby takes over.) Single active vs. multiple active
upgrade makes little difference in this outcome.

Fair, except that there's no standby MDS at this time in case the update
goes wrong.


Is another approach theoretically feasible? Have the updated MDS only go
into the incompatible mode once there's a quorum of new ones available,
or something?

I believe so, yes. That option wasn't explored for this patch because
it was just disambiguating the compatibility flags and the full
side-effects weren't realized.

Would such a patch be accepted if we ended up pursuing this? Any
suggestions on how to best go about this?

It'd be ugly, but you'd have to set it up so that
* new MDSes advertise the old set of required values
* but can identify when all the MDSes are new
* then mark somewhere that they can use the correct values
* then switch to the proper requirements

I don't remember the details of this CompatSet code any more, and it's
definitely made trickier by the MDS having no permanent local state.
Since we do luckily have both the IDs and the strings, you might be
able to do something in the MDSMonitor to identify whether booting
MDSes have "too-old", "old-featureset-but-support-new-feature", or
"new, correct feature advertising" and then either massage that
incoming message down to the "old-featureset-but-support-new-feature"
(if not all the MDSes are new) or do an auto-upgrade of the required
features in the map. And you might also need compatibility code in the
MDS to make sure it sends out the appropriate bits on connection, but
I *think* the CompatSet checks are only done on the monitor and when
an MDS receives an MDSMap.
-Greg
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] where is it possible download CentOS 7.5

[Max Cuttins]

Il 27/03/2018 13:46, Brad Hubbard ha scritto:



On Tue, Mar 27, 2018 at 9:12 PM, Max Cuttins > wrote:


Hi Brad,

    that post was mine. I knew it quite well.

That Post was about confirm the fact that minimum requirements
written in the documentation really didn't exists.

However I never asked if there is somewhere a place where is
possible to download the DEV or the RC of Centos7.5.
I was thinking about to join the community of tester and
developers that are already testing Ceph on that "/not ready/"
environment.

In that POST these questions were not really made, so no answer
where given.


From that thread.

"The necessary kernel changes actually are included as part of 4.16-rc1
which is available now. We also offer a pre-built test kernel with the
necessary fixes here [1].

[1] https://shaman.ceph.com/repos/kernel/ceph-iscsi-test/"; 



I notice that URL is unavailable so maybe the real question should be 
why is that kernel no longer available?


Yes, the link was broken and it seemed to me a misprint of old docs.
As all other stuffs described didn't exists already I thought that event 
this Kernel test was not available (already or anymore).



There are plenty more available at 
https://shaman.ceph.com/repos/kernel/testing/ but *I* can't tell you 
which is relevant but perhaps someone else can.


However the 4.16 is almost ready to be released (shoulded had been already).
At this moment is just a double work use that kernel and after upgrade 
it to the final one.




I see that you talked also about other distribution. Well, I read
around that Suse already implement iSCSI.
However as far as I know (which is not so much), this distribution
use modified kernel in order to let this work.
And in order to use it it's needed  a dashboard that can handle
these kind of differences (OpenAttic).
I knew already OpenAttic is contributing in developing the next
generation of the Ceph Dashboard (and this sound damn good!).
However this also means to me that the *official dashboard* should
not be talking about ISCSI at all (as every implementation of
iSCSI are running on mod version).

So these are the things I cannot figure out:
Why is the iSCSI board on the CEPH official dashboard? (I could
understand on OpenAttic which run on SUSE but not on the official
one).

Why do you believe it should not be?


Maybe I'm in wrong, but I guess that the dashboard manager expects to 
get data/details/stats/config from a particular set of paths, components 
and daemons which cannot be the same for all the ad-hoc implementation.
So there is a dashboard that show values for a component which is 
not there (instead could be there something else but written in another 
way).
Every ad-hoc implementation (like OpenAttic) of course know where to 
find data/details/stats/config for work with their implementation (so 
it's understandable that they have board for iSCSI).

Right?



And why, in the official documentation, the minimu requirements to
let iSCSI work, is to install CentOS7.5? Which doesn't exist? Is
there a RC candidate which I can start to use?


But it doesn't say that, it says " RHEL/CentOS 7.5; Linux kernel v4.16 
or newer; or the Ceph iSCSI client test kernel 
". You seem to 
be ignoring the "Ceph iSCSI client test kernel 
" part?



Yes, the link was broken and it seemed to me a misprint of old docs.

Moreover at first read I figure out that I needed both centOS7.5 *AND 
*kernel 4.16. *OR *the kernel test.
Now you are telling me that all requirements are alternative. Which 
explain to me why the documentation suggest just CentOS and not all 
others distribution.

Also this sounds good.

But I don't think that CentOS7.5 will use the kernel 4.16 ... so you are 
telling me that new feature will be backported to the kernel 3.* ?
Is it right? So i don't need to upgrade the kernel If I'll use 
RHEL/CentOS7.5 ?
This sound even better. I was a bit worried to don't use the mainstream 
kernel of the distribution.



And... if SUSE or even other distribution works already with
iSCSI... why the documentation just doesn't reccomend these ones
instead of RHEL or CENTOS?

Because that would be odd, to say the least. If the documentation is 
incorrect for CentOS then it was, at least at some point, thought to 
be correct and it probably will be correct again in the near future 
and, if not, we can review and correct it as necessary.


Of course the best way to predict the future is to make it happen. ;)
But this is odd for a documentation (at least should be a warning box 
saying that all components will be ready in the near future).


Thank you Brad to answer me.

_

Re: [ceph-users] MDS Bug/Problem

[Perrin, Christopher (zimkop1)]

Hi

It is Possible that I have extracted the wrong log message. I will look into 
that. What happened is that out of the blue
all MDSs started failing. Only after many failed stating attempts with the OSDs 
blocking "old" messages I reset the
journal. After the MDSs where running again we had several files the were empty 
or filled with 0x0, that were previously
fine. I guess that was to be expected after resetting the journal but I had no 
idea how to get them running otherwise.

Regards

Chris

Am Samstag, den 24.03.2018, 05:13 +0800 schrieb John Spray:
> On Fri, Mar 23, 2018 at 7:45 PM, Perrin, Christopher (zimkop1)
>  wrote:
> > Hi,
> > 
> > Last week out MDSs started failing one after another, and could not be 
> > started anymore. After a lot of tinkering I found out that MDSs crashed 
> > after trying to rejoin the Cluster. The only Solution I found that, let 
> > them start again was resetting the journal vie cephfs-journal-tool. Now I 
> > have broken files all over the Cluster.
> 
> Can you clarify:
>  - is the backtrace below is from before you used cephfs-journal-tool
> or after?
>  - what do you mean by "broken file"
> 
> The backtrace you're seeing is in some damage-handling code -- the
> crash is a bug (http://tracker.ceph.com/issues/23452), but you'd only
> be hitting it if your metadata was already inconsistent.  One way it
> could get inconsistent would be from use of cephfs-journal-tool
> (wiping the journal but not the session table, resulting in session
> state inconsistent with the rest of the metadata), but if the crash
> was from before you did that then something else would be going on.
> 
> It would also help if you could be more specific about the original
> issue that drove you to using cephfs-journal-tool, if that was
> something other than the backtrace you included.
> 
> Cheers,
> John
> 
> > Before the crash the OSDs blocked tens of thousands of slow requests.
> > 
> > Can I somehow restore the broken files (I still have a backup of the 
> > journal) and how can I make sure that this doesn't happen agian. I am still 
> > not sure why this even happened.
> > 
> > This happened on ceph version 12.2.3.
> > 
> > This is the log of one MDS:
> >   -224> 2018-03-22 15:52:47.310437 7fd5798fd700  1 -- 
> > x.x.1.17:6803/122963511 <== mon.0 x.x.1.17:6789/0 2  auth_reply(proto 2 
> > 0 (0) Success) v1  33+0+0 (3611581813 0 0) 0x555883df2780 con 
> > 0x555883eb5000
> >   -223> 2018-03-22 15:52:47.310482 7fd5798fd700 10 monclient(hunting): my 
> > global_id is 745317
> >   -222> 2018-03-22 15:52:47.310634 7fd5798fd700  1 -- 
> > x.x.1.17:6803/122963511 --> x.x.1.17:6789/0 -- auth(proto 2 32 bytes epoch 
> > 0) v1 -- 0x555883df2f00 con 0
> >   -221> 2018-03-22 15:52:47.311096 7fd57c09f700  5 -- 
> > x.x.1.17:6803/122963511 >> x.x.1.17:6789/0 conn(0x555883eb5000 :-1 
> > s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=793748 cs=1 l=1). rx 
> > mon.0 seq 3 0x555883df2f00 auth_reply(proto 2 0 (0) Success) v1
> >   -220> 2018-03-22 15:52:47.311178 7fd5798fd700  1 -- 
> > x.x.1.17:6803/122963511 <== mon.0 x.x.1.17:6789/0 3  auth_reply(proto 2 
> > 0 (0) Success) v1  222+0+0 (1789869469 0 0) 0x555883df2f00 con 
> > 0x555883eb5000
> >   -219> 2018-03-22 15:52:47.311319 7fd5798fd700  1 -- 
> > x.x.1.17:6803/122963511 --> x.x.1.17:6789/0 -- auth(proto 2 181 bytes epoch 
> > 0) v1 -- 0x555883df2780 con 0
> >   -218> 2018-03-22 15:52:47.312122 7fd57c09f700  5 -- 
> > x.x.1.17:6803/122963511 >> x.x.1.17:6789/0 conn(0x555883eb5000 :-1 
> > s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=793748 cs=1 l=1). rx 
> > mon.0 seq 4 0x555883df2780 auth_reply(proto 2 0 (0) Success) v1
> >   -217> 2018-03-22 15:52:47.312208 7fd5798fd700  1 -- 
> > x.x.1.17:6803/122963511 <== mon.0 x.x.1.17:6789/0 4  auth_reply(proto 2 
> > 0 (0) Success) v1  799+0+0 (4156877078 0 0) 0x555883df2780 con 
> > 0x555883eb5000
> >   -216> 2018-03-22 15:52:47.312393 7fd5798fd700  1 monclient: found 
> > mon.filer1
> >   -215> 2018-03-22 15:52:47.312416 7fd5798fd700 10 monclient: 
> > _send_mon_message to mon.filer1 at x.x.1.17:6789/0
> >   -214> 2018-03-22 15:52:47.312427 7fd5798fd700  1 -- 
> > x.x.1.17:6803/122963511 --> x.x.1.17:6789/0 -- mon_subscribe({monmap=0+}) 
> > v2 -- 0x555883c8ed80 con 0
> >   -213> 2018-03-22 15:52:47.312461 7fd5798fd700 10 monclient: 
> > _check_auth_rotating renewing rotating keys (they expired before 2018-03-22 
> > 15:52:17.312460)
> >   -212> 2018-03-22 15:52:47.312477 7fd5798fd700 10 monclient: 
> > _send_mon_message to mon.filer1 at x.x.1.17:6789/0
> >   -211> 2018-03-22 15:52:47.312482 7fd5798fd700  1 -- 
> > x.x.1.17:6803/122963511 --> x.x.1.17:6789/0 -- auth(proto 2 2 bytes epoch 
> > 0) v1 -- 0x555883df2f00 con 0
> >   -210> 2018-03-22 15:52:47.312552 7fd580637200  5 monclient: authenticate 
> > success, global_id 745317
> >   -209> 2018-03-22 15:52:47.312570 7fd580637200 10 monclient: 
> > wait_auth_rotating waiting (until 2018-03-22 15:53:17.312568)
> >   -208> 2018

Re: [ceph-users] Radosgw ldap info

[Marc Roos]

 
 
I think I have something wrong in my ldap setup, I cannot radosgw-admin 
user info --uid ldap users. So I have to fix this first.


-Original Message-
From: Benjeman Meekhof [mailto:bmeek...@umich.edu] 
Sent: maandag 26 maart 2018 18:17
To: ceph-users
Subject: Re: [ceph-users] Radosgw ldap info

Hi Marc,

I can't speak to your other questions but as far as the user auth caps 
those are still kept in the radosgw metadata outside of ldap.  As far as 
I know all that LDAP gives you is a way to authenticate users with a 
user/password combination.

So, for example, if you create a user 'ldapuser' in your ldap directory, 
generate a token for that user,  and then use the LDAP token to 
authenticate to RGW as that user you would then find this info in the 
radosgw metadata where it can be altered to set quotas, caps, etc.  You 
could perhaps even add an access key so the conventional auth also works 
for that user identity (I have never tried that, we only do one or the 
other for any given user).

$ radosgw-admin user info --uid ldapuser

{
"user_id": "ldapuser",
"caps": [],
... etc ...
"type": "ldap"
}

thanks,
Ben

On Sat, Mar 24, 2018 at 10:30 AM, Marc Roos  
wrote:
>
>
> To clarify if I understand correctly:
>
> It is NOT POSSIBLE to use an s3 client like eg. cyberduck/mountainduck 

> and supply a user with an 'Access key' and a 'Password' regardless if 
> the user is defined in ldap or local?
>
> I honestly cannot see how this ldap integration should even work, 
> without a proper ldap scheme for auth caps being available. Nor do I 
> understand where you set currently these auth caps, nor do I 
> understand what use the current ldap functionality has.
>
> Would be nice to update this on these pages
>
> https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2/h
> tml-single/ceph_object_gateway_with_ldapad_guide/index
> http://docs.ceph.com/docs/master/radosgw/ldap-auth/
>
>
> Maybe it is good to give some 'beginners' access to the docs pages.
> Because as they are learning ceph (and maybe missing info in the docs) 

> they can add this then. Because I have the impression that many things 

> asked here could be added to the docs.
>
>
>
>
>
> -Original Message-
> From: Konstantin Shalygin [mailto:k0...@k0ste.ru]
> Sent: zondag 18 maart 2018 5:04
> To: ceph-users@lists.ceph.com
> Cc: Marc Roos; Yehuda Sadeh-Weinraub
> Subject: Re: [ceph-users] Radosgw ldap user authentication issues
>
> Hi Marc
>
>
>> looks like no search is being done there.
>
>> rgw::auth::s3::AWSAuthStrategy denied with reason=-13
>
>
> The same for me, http://tracker.ceph.com/issues/23091
>
>
> But Yehuda closed this.
>
>
>
>
> k
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Group-based permissions issue when using ACLs on CephFS

[Yan, Zheng]

On Tue, Mar 27, 2018 at 12:16 AM, Josh Haft  wrote:
> Here's what I'm seeing using basic owner/group permissions. Both
> directories are mounted on my NFS client with the same options. Only
> difference is underneath, from the NFS server, 'aclsupport' is mounted
> via ceph-fuse with fuse_default_permissions=0 (acls enabled), and
> 'noaclsupport' is mounted via ceph-fuse with
> fuse_default_permissions=1.
>
> 'user2' is part of 'group1' and should have r/w access to 'dir', but
> does not when trying to access the filesystem mounted with ACL
> support.
>
> [user2@test01 ]$ groups
> user2 group1
>
> [user2@test01 ]$ stat -c "%i" /mnt/cephfs/aclsupport/dir/
> 1099511790134
> [user2@test01 ]$ stat -c "%i" /mnt/cephfs/noaclsupport/dir/
> 1099511790134
>
> [user2@test01 ]$ ls -lh /mnt/cephfs/aclsupport
> total 1.5K
> drwxrws--- 1 user1   group1  0 Mar 22 15:32 dir
>
> [user2@test01 ]$ ls /mnt/cephfs/aclsupport/dir/
> ls: reading directory /mnt/cephfs/aclsupport/dir/: Permission denied
>
> [user2@test01 ]$ ls /mnt/cephfs/noaclsupport/dir/
> foo
>

This is expected behaviour. When fuse_default_permissions=0, all
permission checks are done in ceph-fuse. In your case, ceph-fuse can't
find which groups request initiator are in. This is due to limitation
of fuse API. I don't have idea how to fix it.

Regards
Yan, Zheng


> On Sat, Mar 24, 2018 at 3:26 AM, Yan, Zheng  wrote:
>> On Sat, Mar 24, 2018 at 11:34 AM, Josh Haft  wrote:
>>>
>>>
>>> On Fri, Mar 23, 2018 at 8:49 PM, Yan, Zheng  wrote:

 On Fri, Mar 23, 2018 at 9:50 PM, Josh Haft  wrote:
 > On Fri, Mar 23, 2018 at 12:14 AM, Yan, Zheng  wrote:
 >>
 >> On Fri, Mar 23, 2018 at 5:14 AM, Josh Haft  wrote:
 >> > Hello!
 >> >
 >> > I'm running Ceph 12.2.2 with one primary and one standby MDS.
 >> > Mounting
 >> > CephFS via ceph-fuse (to leverage quotas), and enabled ACLs by adding
 >> > fuse_default_permissions=0 and client_acl_type=posix_acl to the mount
 >> > options. I then export this mount via NFS and the clients mount
 >> > NFS4.1.
 >> >
 >> does fuse_default_permissions=0 work?
 >
 > Yes, ACLs work as expected when I set fuse_default_permissions=0.
 >
 >> > After doing some in-depth testing it seems I'm unable to allow access
 >> > from
 >> > the NFS clients to a directory/file based on group membership when
 >> > the
 >> > underlying CephFS was mounted with ACL support. This issue appears
 >> > using
 >> > both filesystem permissions (e.g. chgrp) and NFSv4 ACLs. However,
 >> > ACLs do
 >> > work if the principal is a user instead of a group. If I disable ACL
 >> > support
 >> > on the ceph-fuse mount, things work as expected using fs permissions;
 >> > obviously I don't get ACL support.
 >> >
 >> > As an intermediate step I did check whether this works directly on
 >> > the
 >> > CephFS filesystem - on the NFS server - and it does. So it appears to
 >> > be an
 >> > issue re-exporting it via NFS.
 >> >
 >> > I do not see this issue when mounting CephFS via the kernel,
 >> > exporting via
 >> > NFS, and re-running these tests.
 >> >
 >> > I searched the ML and bug reports but only found this -
 >> > http://tracker.ceph.com/issues/12617 - which seems close to the issue
 >> > I'm
 >> > running into, but was closed as resolved 2+ years ago.
 >> >
 >> > Has anyone else run into this? Am I missing something obvious?
 >> >
 >>
 >> ceph-fuse does permission check according to localhost's config of
 >> supplement group. that's why you see this behavior.
 >
 > You're saying both the NFS client and server (where ceph-fuse is
 > running) need to use the same directory backend? (they are)
 > I should have mentioned I'm using LDAP/AD on client and server, so I
 > don't think that is the problem.
 >
 > Either way, I would not expect the behavior to change simply by
 > enabling ACLs, especially when I'm using filesystem permissions, and
 > ACLs aren't part of the equation.

 More specifically, ceph-fuse find which groups request initiator are
 in by function fuse_req_getgroups(). this function does tricks on
 "/proc/%lu/task/%lu/status".  It only works  when nfs client and
 ceph-fuse are running on the same machine.

>>> So why does this work when I'm using ceph-fuse but ACLs are disabled?

>>
>> Really?
>>
>> Please check if supplement groups work for inodes without ACL (mount
>> fuse with config option fuse_default_permissions=0)
>>
>>

 >> Yan, Zheng
 >>
 >> > Thanks!
 >> > Josh
 >> >
 >> >
 >> > ___
 >> > ceph-users mailing list
 >> > ceph-users@lists.ceph.com
 >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 >> >
>>>
>>>
___
ceph-users mailing list
ceph-users@lists