date:20161214

Re: [ceph-users] cephfs quota

2016-12-14 Thread Shinobu Kinjo

Would you give us some outputs?

 # getfattr -n ceph.quota.max_bytes /some/dir

and

 # ls -l /some/dir

On Thu, Dec 15, 2016 at 4:41 PM, gjprabu  wrote:
>
> Hi Team,
>
>   We are using ceph version 10.2.4 (Jewel) and data's are mounted
> with cephfs file system in linux. We are trying to set quota for directory
> and files but its don't worked with below document. I have set 100mb for
> directory quota but after reaching keep me allowing to put the data in that
> location. Highly appreciate any one help on this issue.
>
> http://docs.ceph.com/docs/jewel/cephfs/quota/
>
>
> Regards
> Prabu GJ
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] cephfs quota

2016-12-14 Thread gjprabu



Hi Team,



  We are using ceph version 10.2.4 (Jewel) and data's are mounted with 
cephfs file system in linux. We are trying to set quota for directory and files 
but its don't worked with below document. I have set 100mb for directory quota 
but after reaching keep me allowing to put the data in that location. Highly 
appreciate any one help on this issue.



http://docs.ceph.com/docs/jewel/cephfs/quota/





Regards

Prabu GJ

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [Fixed] OS-Prober In Ubuntu Xenial causes journal errors

2016-12-14 Thread Christian Balzer


Hello,

On Wed, 14 Dec 2016 16:23:18 - Nick Fisk wrote:

> Hi All,
> 
> For all those who have been hit by the bug in Ubuntu where update-grub causes 
> your OSD's to crash out when it probes the partition,
> a fixed package has been released to xenial-proposed which fixes the problem. 
> I have installed and confirm it fixes the problem.
> 
> https://launchpad.net/ubuntu/+source/os-prober/1.70ubuntu3.1
>
Just for the record and since a major pet peeve of mine has been triggered:

OS-prober is the FIRST thing thing I un-install on every last Debian
machine of mine, as it has a long history of doing unwanted and outright
bad things.

Christian
-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Global OnLine Japan/Rakuten Communications
http://www.gol.com/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] What happens if all replica OSDs journals are broken?

2016-12-14 Thread Kevin Olbrich

2016-12-14 2:37 GMT+01:00 Christian Balzer :

>
> Hello,
>

Hi!

>
> On Wed, 14 Dec 2016 00:06:14 +0100 Kevin Olbrich wrote:
>
> > Ok, thanks for your explanation!
> > I read those warnings about size 2 + min_size 1 (we are using ZFS as
> RAID6,
> > called zraid2) as OSDs.
> >
> This is similar to my RAID6 or RAID10 backed OSDs with regards to having
> very resilient, extremely unlikely to fail OSDs.
>

This was our intention (unlikely to fail, data security > performance).
We use Ceph for OpenStack (Cinder RBD).

> As such a Ceph replication of 2 with min_size is a calculated risk,
> acceptable for me on others in certain use cases.
> This is also with very few (2-3) journals per SSD.
>

We are running 14x 500G RAID6 ZFS-RAID per Host (1x journal, 1x OSD, 32GB
RAM).
The ZFS pools use L2ARC-Cache on Samsung 850 PRO's 128GB.
Hint: Was a bad idea, would have better split the ZFS pools. (ZFS
performance was very good but double parity with 4k random on sync with
ceph takes very long, resulting in XXX requests blocked more than 32
seconds).
Currently I am waiting for a lab cluster to test "osd op threads" for these
single OSD hosts.

> If:
>
> 1. Your journal SSDs are well trusted and monitored (Intel DC S36xx, 37xx)
>

Indeed Intel DC P3700 400GB for Ceph. We had Samsung 850 PRO before I leard
4k random while DSYNC is a very bad idea... ;-)

2. Your failure domain represented by a journal SSD is small enough
> (meaning that replicating the lost OSDs can be done quickly)
>

OSDs are rather large but we are "just" using 8 TB (size 2) in the whole
cluster (OSD is 24% full).
Before we moved from infernalis to jewel, a recovery from an OSD which was
offline for 8 hours took approx. one hour to be back in sync.

it may be an acceptable risk for you as well.

We got reliable backups in the past but downtime is a greater problem.

>
>
> Time to raise replication!
> >
> If you can afford that (money, space, latency), definitely go for it.
>

It's more the double journal failure which scares me compared to the OSD
itself (as ZFS was very reliable in the past).

Kevin

> Christian
> > Kevin
> >
> > 2016-12-13 0:00 GMT+01:00 Christian Balzer :
> >
> > > On Mon, 12 Dec 2016 22:41:41 +0100 Kevin Olbrich wrote:
> > >
> > > > Hi,
> > > >
> > > > just in case: What happens when all replica journal SSDs are broken
> at
> > > once?
> > > >
> > > That would be bad, as in BAD.
> > >
> > > In theory you just "lost" all the associated OSDs and their data.
> > >
> > > In practice everything but in the in-flight data at the time is still
> on
> > > the actual OSDs (HDDs), but it's inconsistent and inaccessible as far
> as
> > > Ceph is concerned.
> > >
> > > So with some trickery and an experienced data-recovery Ceph consultant
> you
> > > _may_ get things running with limited data loss/corruption, but that's
> > > speculation and may be wishful thinking on my part.
> > >
> > > Another data point to deploy only well known/monitored/trusted SSDs and
> > > have a 3x replication.
> > >
> > > > The PGs most likely will be stuck inactive but as I read, the
> journals
> > > just
> > > > need to be replaced (http://ceph.com/planet/ceph-
> recover-osds-after-ssd-
> > > > journal-failure/).
> > > >
> > > > Does this also work in this case?
> > > >
> > > Not really, no.
> > >
> > > The above works by having still a valid state and operational OSDs from
> > > which the "broken" one can recover.
> > >
> > > Christian
> > > --
> > > Christian BalzerNetwork/Systems Engineer
> > > ch...@gol.com   Global OnLine Japan/Rakuten Communications
> > > http://www.gol.com/
> > >
>
>
> --
> Christian BalzerNetwork/Systems Engineer
> ch...@gol.com   Global OnLine Japan/Rakuten Communications
> http://www.gol.com/
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] 10.2.3: Howto disable cephx_sign_messages and preventing a LogFlood

2016-12-14 Thread Ilya Dryomov

On Wed, Dec 14, 2016 at 5:10 PM, Bjoern Laessig
 wrote:
> Hi,
>
> i triggered a Kernel bug in the ceph-krbd code
>  * http://www.spinics.net/lists/ceph-devel/msg33802.html

The fix is ready and is set to be merged into 4.10-rc1.

How often can you hit it?

>
> Ilya Dryomov wrote in a reply: The way to go is to \u201cdisabling cephx
> message signing\u201d

That's one option.  Lightening the load on the client machine is
another - this bug should only surface under high memory pressure.

>
> I have a lot of systems that use librbd and only 2 Hosts that user krbd.
> I do not want to disable message-signing completely, so after trying to:
>
>   rbd map mypool/myrbd --id dev --keyring /etc/ceph/ceph.client.dev.keyring 
> --options nocephx_sign_messages
>
> which lead to endless waiting. Strace says:
>
> [pid 10845] open("/sys/bus/rbd/add_single_major", O_WRONLY) = 4
> [pid 10845] write(4, 
> "[2001:67c:670:100:5054:ff:fe78:2a86]:6789,[2001:67c:670:100:5054:ff:fecf:22f6]:6789,[2001:67c:670:100:5054:ff:feae:42ee]:6789
>  name=dev,key=client.dev,nocephx_sign_messages ptxdev WORK_CEPH_BLA -", 194 
> 
> [pid 10899] <... futex resumed> )   = -1 ETIMEDOUT (Connection timed out)
> [pid 10899] futex(0x55f663895508, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 10899] futex(0x55f66389555c, 
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 3, {1481729207, 622660011}, 
> ) = -1 ETIMEDOUT (Connection timed out)
> [pid 10899] futex(0x55f663895508, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 10899] futex(0x55f66389555c, 
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 5, {1481729212, 622818887}, 
> ) = -1 ETIMEDOUT (Connection timed out)
> [pid 10899] futex(0x55f663895508, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 10899] futex(0x55f66389555c, 
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 7, {1481729217, 622973709}, 
> ) = -1 ETIMEDOUT (Connection timed out)
> [pid 10899] futex(0x55f663895508, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 10899] futex(0x55f66389555c, 
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 9, {1481729222, 623078200}, 
> ) = -1 ETIMEDOUT (Connection timed out)
> [pid 10899] futex(0x55f663895508, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 10899] futex(0x55f66389555c, 
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 11, {1481729227, 623190181}, 
> ) = -1 ETIMEDOUT (Connection timed out)
> [pid 10899] futex(0x55f663895508, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 10899] futex(0x55f66389555c, 
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 13, {1481729232, 623298560}, 
> ) = -1 ETIMEDOUT (Connection timed out)
> [pid 10899] futex(0x55f663895508, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 10899] futex(0x55f66389555c, 
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 15, {1481729237, 623504923}, 
> ) = -1 ETIMEDOUT (Connection timed out)
> [pid 10899] futex(0x55f663895508, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 10899] futex(0x55f66389555c, 
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 17, {1481729242, 623709707}, 
> ) = -1 ETIMEDOUT (Connection timed out)
> [pid 10899] futex(0x55f663895508, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 10899] futex(0x55f66389555c, 
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 19, {1481729247, 623816985}, 
> ) = -1 ETIMEDOUT (Connection timed out)
> [pid 10899] futex(0x55f663895508, FUTEX_WAKE_PRIVATE, 1) = 0
> (... cycle endless )
>
> After failing with rbd-map-options, i set in /etc/ceph/ceph.conf:
>> [global]
>> cephx_sign_messages = false
>
> Now i have 1 (one) ceph-osd-process which starts logging an enormous
> (4GBytes/hour) count of lines like:
>
>
>> 2016-12-14 07:49:40.111415 7f25f0c0a700  0 SIGN: MSG 1 Sender did not set 
>> CEPH_MSG_FOOTER_SIGNED.
>> 2016-12-14 07:49:40.111422 7f25f0c0a700  0 SIGN: MSG 1 Message signature 
>> does not match contents.
>> 2016-12-14 07:49:40.111423 7f25f0c0a700  0 SIGN: MSG 1Signature on message:
>> 2016-12-14 07:49:40.111425 7f25f0c0a700  0 SIGN: MSG 1sig: 0
>> 2016-12-14 07:49:40.111427 7f25f0c0a700  0 SIGN: MSG 1Locally calculated 
>> signature:
>> 2016-12-14 07:49:40.111428 7f25f0c0a700  0 SIGN: MSG 1
>> sig_check:1990181984681779795
>> 2016-12-14 07:49:40.111429 7f25f0c0a700  0 Signature failed.
>> 2016-12-14 07:49:40.111430 7f25f0c0a700  0 -- 
>> [myOsdNode1.ipv6.address]:6802/29624 >> 
>> [my_kRBD_client.ipv6.address]:0/943150662 pipe(0x5603813bd400 sd=121 :6802 
>> s=2 pgs=353544735 cs=1 l=1 c=0x560381182900).Signature check failed
>
> Then i read on:
>   http://docs.ceph.com/docs/jewel/rados/troubleshooting/log-and-debug/
> that \u201cCeph\u2019s logging levels operate on a scale of 1 to 20, where 1 
> is
> terse and 20 is verbose.\u201d The Messages i got, had a loglevel of 0 so i
> set the loglevel to -1.
>
>> ceph tell osd.3 injectargs --debug-auth -1
>> ceph tell osd.3 injectargs --debug-ms -1
>
> to get rid of them.
>
> Actually i do not have to delete the logfiles every 12 hours, so my pain
> has gone but its a workaround for a workaround. That is painful. What
>

[ceph-users] [Fixed] OS-Prober In Ubuntu Xenial causes journal errors

2016-12-14 Thread Nick Fisk

Hi All,

For all those who have been hit by the bug in Ubuntu where update-grub causes 
your OSD's to crash out when it probes the partition,
a fixed package has been released to xenial-proposed which fixes the problem. I 
have installed and confirm it fixes the problem.

https://launchpad.net/ubuntu/+source/os-prober/1.70ubuntu3.1

Nick

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] 10.2.3: Howto disable cephx_sign_messages and preventing a LogFlood

2016-12-14 Thread Bjoern Laessig

Hi,

i triggered a Kernel bug in the ceph-krbd code
 * http://www.spinics.net/lists/ceph-devel/msg33802.html

Ilya Dryomov wrote in a reply: The way to go is to “disabling cephx
message signing”

I have a lot of systems that use librbd and only 2 Hosts that user krbd.
I do not want to disable message-signing completely, so after trying to:

  rbd map mypool/myrbd --id dev --keyring /etc/ceph/ceph.client.dev.keyring 
--options nocephx_sign_messages 

which lead to endless waiting. Strace says:

[pid 10845] open("/sys/bus/rbd/add_single_major", O_WRONLY) = 4
[pid 10845] write(4, 
"[2001:67c:670:100:5054:ff:fe78:2a86]:6789,[2001:67c:670:100:5054:ff:fecf:22f6]:6789,[2001:67c:670:100:5054:ff:feae:42ee]:6789
 name=dev,key=client.dev,nocephx_sign_messages ptxdev WORK_CEPH_BLA -", 194 

[pid 10899] <... futex resumed> )   = -1 ETIMEDOUT (Connection timed out)
[pid 10899] futex(0x55f663895508, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 10899] futex(0x55f66389555c, 
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 3, {1481729207, 622660011}, 
) = -1 ETIMEDOUT (Connection timed out)
[pid 10899] futex(0x55f663895508, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 10899] futex(0x55f66389555c, 
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 5, {1481729212, 622818887}, 
) = -1 ETIMEDOUT (Connection timed out)
[pid 10899] futex(0x55f663895508, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 10899] futex(0x55f66389555c, 
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 7, {1481729217, 622973709}, 
) = -1 ETIMEDOUT (Connection timed out)
[pid 10899] futex(0x55f663895508, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 10899] futex(0x55f66389555c, 
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 9, {1481729222, 623078200}, 
) = -1 ETIMEDOUT (Connection timed out)
[pid 10899] futex(0x55f663895508, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 10899] futex(0x55f66389555c, 
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 11, {1481729227, 623190181}, 
) = -1 ETIMEDOUT (Connection timed out)
[pid 10899] futex(0x55f663895508, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 10899] futex(0x55f66389555c, 
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 13, {1481729232, 623298560}, 
) = -1 ETIMEDOUT (Connection timed out)
[pid 10899] futex(0x55f663895508, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 10899] futex(0x55f66389555c, 
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 15, {1481729237, 623504923}, 
) = -1 ETIMEDOUT (Connection timed out)
[pid 10899] futex(0x55f663895508, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 10899] futex(0x55f66389555c, 
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 17, {1481729242, 623709707}, 
) = -1 ETIMEDOUT (Connection timed out)
[pid 10899] futex(0x55f663895508, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 10899] futex(0x55f66389555c, 
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 19, {1481729247, 623816985}, 
) = -1 ETIMEDOUT (Connection timed out)
[pid 10899] futex(0x55f663895508, FUTEX_WAKE_PRIVATE, 1) = 0
(... cycle endless )

After failing with rbd-map-options, i set in /etc/ceph/ceph.conf:
> [global]
> cephx_sign_messages = false

Now i have 1 (one) ceph-osd-process which starts logging an enormous
(4GBytes/hour) count of lines like:


> 2016-12-14 07:49:40.111415 7f25f0c0a700  0 SIGN: MSG 1 Sender did not set 
> CEPH_MSG_FOOTER_SIGNED.
> 2016-12-14 07:49:40.111422 7f25f0c0a700  0 SIGN: MSG 1 Message signature does 
> not match contents.
> 2016-12-14 07:49:40.111423 7f25f0c0a700  0 SIGN: MSG 1Signature on message:
> 2016-12-14 07:49:40.111425 7f25f0c0a700  0 SIGN: MSG 1sig: 0
> 2016-12-14 07:49:40.111427 7f25f0c0a700  0 SIGN: MSG 1Locally calculated 
> signature:
> 2016-12-14 07:49:40.111428 7f25f0c0a700  0 SIGN: MSG 1
> sig_check:1990181984681779795
> 2016-12-14 07:49:40.111429 7f25f0c0a700  0 Signature failed.
> 2016-12-14 07:49:40.111430 7f25f0c0a700  0 -- 
> [myOsdNode1.ipv6.address]:6802/29624 >> 
> [my_kRBD_client.ipv6.address]:0/943150662 pipe(0x5603813bd400 sd=121 :6802 
> s=2 pgs=353544735 cs=1 l=1 c=0x560381182900).Signature check failed

Then i read on:
  http://docs.ceph.com/docs/jewel/rados/troubleshooting/log-and-debug/
that “Ceph’s logging levels operate on a scale of 1 to 20, where 1 is
terse and 20 is verbose.” The Messages i got, had a loglevel of 0 so i
set the loglevel to -1.

> ceph tell osd.3 injectargs --debug-auth -1
> ceph tell osd.3 injectargs --debug-ms -1

to get rid of them. 

Actually i do not have to delete the logfiles every 12 hours, so my pain
has gone but its a workaround for a workaround. That is painful. What
could i do to disable cephx-message-signing only for the krbd clients?

Björn Lässig


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] radosgw fastcgi problem

2016-12-14 Thread Z Will

Hi:
Very sorry for the last email, it was an accident. Recently, I
trid to configure radosgw (0.94.7) with nginx as frontend, benchmark
it with cosbench. And I found a strange thing. My os-related
 configuration are,
net.core.netdev_max_backlog = 1000
net.core.somaxconn = 1024
 In rgw_main.cc, I see #define SOCKET_BACKLOG 1024. so here is the problem.

1、When I configure cosbench with 1000 workers , everything is OK. But
when it was 2000workers,there will be a lot of CLOSE_WAIT state
connections on rgw side, and rgw will no longer response to any
request.Use netstat command , I can see there are 1024 CLOSE_WAIT
connections.

2、I change  net.core.somaxconn to 128, and run cosbench with 2000
workers again, very soon, there will be no response from radosgw, and
use netstat command , I can see there are 128 CLOSE_WAIT connections.


I tried a lot things, googled it , but the problem still exist. I
thought if there is some bug on fastcgi lib. Have anyone met this
problem?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] radosgw fastcgi problem

2016-12-14 Thread Z Will

Hi:
Recently, I trid to configure radosgw with nginx as frontend,
benchmark it with cosbench. And I found a strange thing. My os-related
 configuration are,
net.core.netdev_max_backlog = 1000
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs quota

[ceph-users] cephfs quota

Re: [ceph-users] [Fixed] OS-Prober In Ubuntu Xenial causes journal errors

Re: [ceph-users] What happens if all replica OSDs journals are broken?

Re: [ceph-users] 10.2.3: Howto disable cephx_sign_messages and preventing a LogFlood

[ceph-users] [Fixed] OS-Prober In Ubuntu Xenial causes journal errors

[ceph-users] 10.2.3: Howto disable cephx_sign_messages and preventing a LogFlood

[ceph-users] radosgw fastcgi problem

[ceph-users] radosgw fastcgi problem

9 matches

Site Navigation

Mail list logo

Footer information