[ceph-users] Improvement Request: Honor -j for rocksdb

[Dyweni - Ceph-Users]

Hi,

I noticed while compiling Ceph Jewel (10.2.0), that the compiling 
process does not fully honor make's -j switch.  In the ps output I've 
attached, you will see that I've requested only 3 concurrent jobs.


Make assigned 2 jobs to ceph and 1 job to rocksdb.  The rocksdb then 
took 6 additional jobs (I have 6 cores in this computer), for a total of 
8 (or 9 depending how you look at it) concurrent processes.


Can this be corrected?

Thanks,
Dyweni



# ps  fau
USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
root 27232  0.1  0.5  58344 43408 pts/3Ss   23:18   0:00 bash
root 27951  0.3  0.6 143856 56988 pts/3S+   23:18   0:00  \_ 
/usr/bin/python3.3 -bO /usr/lib/python-exec/python3.3/ebuild ceph-10.2.0.ebuild 
digest clean compile
root 13128  0.0  0.0   4204  1404 pts/3S+   23:22   0:00  \_ 
[sys-cluster/ceph-10.2.0] sandbox /usr/lib/portage/python3.3/ebuild.sh compile
root 13129  0.0  0.0  27060  7480 pts/3S+   23:22   0:00  \_ 
/bin/bash /usr/lib/portage/python3.3/ebuild.sh compile
root 13150  0.0  0.0  27192  6952 pts/3S+   23:22   0:00  
\_ /bin/bash /usr/lib/portage/python3.3/ebuild.sh compile
root 13155  0.0  0.0  14836  3236 pts/3S+   23:22   0:00
  \_ /bin/bash /usr/lib/portage/python3.3/ebuild-helpers/emake -j3
root 13157  0.0  0.0   9432  2324 pts/3S+   23:22   0:00
  \_ make -j3 -j3
root 13158  0.0  0.0  14580  2920 pts/3S+   23:22   0:00
  \_ /bin/sh -c fail=; \ if (target_option=k; case ${target_option-} in 
?) ;; *) echo "am__make_running_with_option: internal error: invalid" "target 
option '${target_option-}' specified" >&2; exit 1;; esac; has_opt
root 13165  0.0  0.0  14580  2356 pts/3S+   23:22   0:00
  \_ /bin/sh -c fail=; \ if (target_option=k; case 
${target_option-} in ?) ;; *) echo "am__make_running_with_option: internal 
error: invalid" "target option '${target_option-}' specified" >&2; exit 1;; 
esac; has
root 13166  0.1  0.0  13744  6612 pts/3S+   23:22   0:00
  \_ make all
root 13178  0.1  0.0  13744  6544 pts/3S+   23:22   0:00
  \_ make all-recursive
root 13182  0.0  0.0  14580  2984 pts/3S+   23:22   0:00
  \_ /bin/sh -c fail=; \ if (target_option=k; case 
${target_option-} in ?) ;; *) echo "am__make_running_with_option: internal 
error: invalid" "target option '${target_option-}' specified" >&2; exit 1
root 13192  0.1  0.0  13740  6672 pts/3S+   23:22   0:00
  \_ make all-am
root 13206  0.0  0.0  14440  2776 pts/3S+   23:22   0:00
  \_ /bin/sh -c cd rocksdb && 
CC="x86_64-pc-linux-gnu-gcc" CXX="x86_64-pc-linux-gnu-g++" EXTRA_CXXFLAGS=-fPIC 
PORTABLE=1 make -j6 static_lib
root 13208  0.4  0.0  12288  5256 pts/3S+   23:22   0:00
  |   \_ make -j6 static_lib
root 14760  0.0  0.0  14576  2704 pts/3S+   23:22   0:00
  |   \_ /bin/sh -c echo "  CC  " 
db/db_impl.o;x86_64-pc-linux-gnu-g++ -march=native -O2 -pipe  -std=gnu++11 
-fPIC -g -W -Wextra -Wall -Wsign-compare -Wshadow -Wno-unused-parameter -I
root 14761  0.0  0.0  12912  2636 pts/3S+   23:22   0:00
  |   |   \_ 
/usr/x86_64-pc-linux-gnu/gcc-bin/4.9.3/x86_64-pc-linux-gnu-g++ -march=native 
-O2 -pipe -std=gnu++11 -fPIC -g -W -Wextra -Wall -Wsign-compare -Wshadow 
-Wno-unused-parameter -I
root 14762 74.3  6.6 570972 541112 pts/3   R+   23:22   0:11
  |   |   \_ 
/usr/libexec/gcc/x86_64-pc-linux-gnu/4.9.3/cc1plus -quiet -I . -I ./include 
-D_GNU_SOURCE -D ROCKSDB_PLATFORM_POSIX -D ROCKSDB_LIB_IO_POSIX -D 
_LARGEFILE64_SOURCE -D OS_L
root 14763  0.3  0.1  29548 15552 pts/3S+   23:22   0:00
  |   |   \_ 
/usr/lib/gcc/x86_64-pc-linux-gnu/4.9.3/../../../../x86_64-pc-linux-gnu/bin/as 
-I . -I ./include --64 -o db/db_impl.o
root 15077  0.0  0.0  14576  2692 pts/3S+   23:23   0:00
  |   \_ /bin/sh -c echo "  CC  " 
db/managed_iterator.o;x86_64-pc-linux-gnu-g++ -march=native -O2 -pipe  
-std=gnu++11 -fPIC -g -W -Wextra -Wall -Wsign-compare -Wshadow -Wno-unused-par
root 15084  0.0  0.0  12912  2724 pts/3S+   23:23   0:00
  |   |   \_ 
/usr/x86_64-pc-linux-gnu/gcc-bin/4.9.3/x86_64-pc-linux-gnu-g++ -march=native 
-O2 -pipe -std=gnu++11 -fPIC -g -W -Wextra -Wall -Wsign-compare -Wshadow 
-Wno-unused-parameter -I
root 15086 20.0  0.9 106368 79064 pts/3R+   23:23   0:00  

[ceph-users] Web based S3 client

[张灿]

Hi,

In order to help new users to get hands on S3, we developed a web based S3 
client called “Sree”, and hope to see if it could become part of Ceph. 
Currently we host the project at:

https://github.com/cannium/Sree

Users could use Sree to manage their files in browser, through Ceph’s S3 
interface. I think it’s more friendly for new users than s3cmd, and would help 
Ceph to hit more users.

Any suggestions are welcomed. Hope to see your replies.


Cheers,
Can ZHANG
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] about slides on VAULT of 2016

[席智勇]

got~
thank you~

regards~

2016-04-28 20:59 GMT+08:00 Sage Weil :

> Hi,
>
> Here are the slides:
>
>
> http://www.slideshare.net/sageweil1/bluestore-a-new-faster-storage-backend-for-ceph
>
> sage
>
> On Thu, 28 Apr 2016, 席智勇 wrote:
>
> > hi sage:
> >I find the slides of VAULT of 2016 on this
> > page(http://events.linuxfoundation.org/events/vault/program/slides),
> > it seems not the whole accoding to the schedule info, and I didn't find
> > yours. Can you share your slides or any things usefull on VAULT about
> > BlueStore.
> >
> >
> > regards~
> > zhiyong
> >
> >
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] help troubleshooting some osd communication problems

[Samuel Just]

I'd guess that to make any progress we'll need debug ms = 20 on both
sides of the connection when a message is lost.
-Sam

On Thu, Apr 28, 2016 at 2:38 PM, Mike Lovell  wrote:
> there was a problem on one of the clusters i manage a couple weeks ago where
> pairs of OSDs would wait indefinitely on subops from the other OSD in the
> pair. we used a liberal dose of "ceph osd down ##" on the osds and
> eventually things just sorted them out a couple days later.
>
> it seems to have come back today and co-workers and i are stuck on trying to
> figure out why this is happening. here are the details that i know.
> currently 2 OSDs, 41 and 148, keep waiting on subops from each other
> resulting in lines such as the following in ceph.log.
>
> 2016-04-28 13:29:26.875797 osd.41 10.217.72.22:6802/3769 56283 : cluster
> [WRN] slow request 30.642736 seconds old, received at 2016-04-28
> 13:28:56.233001: osd_op(client.11172360.0:516946146
> rbd_data.36bfe359c4998.0d08 [set-alloc-hint object_size 4194304
> write_size 4194304,write 1835008~143360] 17.3df49873 RETRY=1
> ack+ondisk+retry+write+redirected+known_if_redirected e159001) currently
> waiting for subops from 5,140,148
>
> 2016-04-28 13:29:28.031452 osd.148 10.217.72.11:6820/6487 25324 : cluster
> [WRN] slow request 30.960922 seconds old, received at 2016-04-28
> 13:28:57.070471: osd_op(client.24127500.0:2960618
> rbd_data.38178d8adeb4d.10f8 [set-alloc-hint object_size 8388608
> write_size 8388608,write 3194880~4096] 17.fb41a37c RETRY=1
> ack+ondisk+retry+write+redirected+known_if_redirected e159001) currently
> waiting for subops from 41,115
>
> from digging in the logs, it appears like some messages are being lost
> between the OSDs. this is what osd.41 sees:
> -
> 2016-04-28 13:28:56.233702 7f3b171e0700  1 -- 10.217.72.22:6802/3769 <==
> client.11172360 10.217.72.41:0/6031968 6 
> osd_op(client.11172360.0:516946146 rbd_data.36bfe359c4998.0d08
> [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~143360]
> 17.3df49873 RETRY=1 ack+ondisk+retry+write+redirected+known_if_redirected
> e159001) v5  236+0+143360 (781016428 0 3953649960) 0x1d551c00 con
> 0x1a78d9c0
> 2016-04-28 13:28:56.233983 7f3b49020700  1 -- 10.217.89.22:6825/313003769
> --> 10.217.89.18:6806/1010441 -- osd_repop(client.11172360.0:516946146 17.73
> 3df49873/rbd_data.36bfe359c4998.0d08/head//17 v 159001'26722799)
> v1 -- ?+46 0x1d6db200 con 0x21add440
> 2016-04-28 13:28:56.234017 7f3b49020700  1 -- 10.217.89.22:6825/313003769
> --> 10.217.89.11:6810/4543 -- osd_repop(client.11172360.0:516946146 17.73
> 3df49873/rbd_data.36bfe359c4998.0d08/head//17 v 159001'26722799)
> v1 -- ?+46 0x1d6dd000 con 0x21ada000
> 2016-04-28 13:28:56.234046 7f3b49020700  1 -- 10.217.89.22:6825/313003769
> --> 10.217.89.11:6812/43006487 -- osd_repop(client.11172360.0:516946146
> 17.73 3df49873/rbd_data.36bfe359c4998.0d08/head//17 v
> 159001'26722799) v1 -- ?+144137 0x14becc00 con 0xf2cd4a0
> 2016-04-28 13:28:56.243555 7f3b35976700  1 -- 10.217.89.22:6825/313003769
> <== osd.140 10.217.89.11:6810/4543 23 
> osd_repop_reply(client.11172360.0:516946146 17.73 ondisk, result = 0) v1
>  83+0+0 (494696391 0 0) 0x28ea7b00 con 0x21ada000
> 2016-04-28 13:28:56.257816 7f3b27d9b700  1 -- 10.217.89.22:6825/313003769
> <== osd.5 10.217.89.18:6806/1010441 35 
> osd_repop_reply(client.11172360.0:516946146 17.73 ondisk, result = 0) v1
>  83+0+0 (2393425574 0 0) 0xfe82fc0 con 0x21add440
>
>
> this, however is what osd.148 sees:
> -
> [ulhglive-root@ceph1 ~]# grep :516946146 /var/log/ceph/ceph-osd.148.log
> 2016-04-28 13:29:33.470156 7f195fcfc700  1 -- 10.217.72.11:6820/6487 <==
> client.11172360 10.217.72.41:0/6031968 460 
> osd_op(client.11172360.0:516946146 rbd_data.36bfe359c4998.0d08
> [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~143360]
> 17.3df49873 RETRY=2 ack+ondisk+retry+write+redirected+known_if_redirected
> e159002) v5  236+0+143360 (129493315 0 3953649960) 0x1edf2300 con
> 0x24dc0d60
>
> also, due to the ceph osd down commands, there is recovery that needs to
> happen for a PG shared between these OSDs that is never making any progress.
> its probably due to whatever is cause the repops to fail.
>
> i did some tcpdump on both sides limiting things to the ip addresses and
> ports being used by these two OSDs and see packets flowing between the two
> osds. i attempted to have wireshark decode the actual ceph traffic but it
> was only able to get bits and pieces of the ceph protocol bits but at least
> for the moment i'm blaming that on the ceph dissector for wireshark. there
> aren't any dropped or error packets on any of the network interfaces
> involved.
>
> does anyone have any ideas of where to look next or other tips for this?
> we've put debug_ms and debug_osd 

Re: [ceph-users] Troubleshoot blocked OSDs

[Tu Holmes]

I had this same sort of thing with Hammer. 
Looking forward to your results. 
Please post your configuration when done. 
I am contemplating doing a similar action to resolve my issues and it would be 
interesting in knowing your outcome first. 

//Tu




On Thu, Apr 28, 2016 at 1:18 PM -0700, "Andrus, Brian Contractor" 
 wrote:




















Load on all nodes is 1.04 to 1.07


I am updating now to Jewel 10.2 (from 9.2)


This is CephFS with SSD journals.


 


Hopefully the update to jewel fixes lots.


 


 


Brian Andrus


ITACS/Research Computing


Naval Postgraduate School


Monterey, California


voice: 831-656-6238


 


 


 




From: Lincoln Bryant [mailto:linco...@uchicago.edu]


Sent: Thursday, April 28, 2016 12:56 PM

To: Andrus, Brian Contractor

Cc: ceph-users@lists.ceph.com

Subject: Re: [ceph-users] Troubleshoot blocked OSDs




 


OK, a few more questions.



 




What does the load look like on the OSDs with ‘iostat’ during the rsync?



 




What version of Ceph? Are you using RBD, CephFS, something else? 




 




SSD journals or no?




 




—Lincoln




 





On Apr 28, 2016, at 2:53 PM, Andrus, Brian Contractor  wrote:



 




Lincoln,




 




That was the odd thing to me. Ceph health detail listed all 4 OSDs, so I 
checked all the systems.




I have since let it settle until it is OK again and started. Within a couple 
minutes, it started showing blocked requests and they are indeed on all 4 OSDs.




 




Brian Andrus




ITACS/Research Computing




Naval Postgraduate School




Monterey, California




voice: 831-656-6238




 




 




 






From: Lincoln
 Bryant [mailto:linco...@uchicago.edu] 

Sent: Thursday, April 28, 2016 12:31 PM

To: Andrus, Brian Contractor

Cc: ceph-users@lists.ceph.com

Subject: Re: [ceph-users] Troubleshoot blocked OSDs






 




Hi Brian,





 






The first thing you can do is “ceph health detail”, which should give you some 
more information about which OSD(s) have blocked requests.






 






If it’s isolated to one OSD in particular, perhaps use iostat to check 
utilization and/or smartctl to check health. 






 






—Lincoln






 









On Apr 28, 2016, at 2:26 PM, Andrus, Brian Contractor  wrote:





 






All,






 






I have a small ceph cluster with 4 OSDs and 3 MONs on 4 systems.






I was rsyncing about 50TB of files and things get very slow. To the point I 
stopped the rsync, but even with everything stopped, I see:






 






health HEALTH_WARN






    80 requests are blocked > 32 sec






 






The number was as high as 218, but they seem to be draining down.






I see no issues on any of the systems, CPU load is low, memory usage is low.






 






How do I go about finding why a request is blocked for so long? These have been 
hitting >500 seconds for block time.






 






Brian Andrus






ITACS/Research Computing






Naval Postgraduate School






Monterey, California






voice: 831-656-6238






 





___

ceph-users mailing list

ceph-users@lists.ceph.com

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com







 




___

ceph-users mailing list

ceph-users@lists.ceph.com

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com





 










___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] help troubleshooting some osd communication problems

[Mike Lovell]

there was a problem on one of the clusters i manage a couple weeks ago
where pairs of OSDs would wait indefinitely on subops from the other OSD in
the pair. we used a liberal dose of "ceph osd down ##" on the osds and
eventually things just sorted them out a couple days later.

it seems to have come back today and co-workers and i are stuck on trying
to figure out why this is happening. here are the details that i know.
currently 2 OSDs, 41 and 148, keep waiting on subops from each other
resulting in lines such as the following in ceph.log.

2016-04-28 13:29:26.875797 osd.41 10.217.72.22:6802/3769 56283 : cluster
[WRN] slow request 30.642736 seconds old, received at 2016-04-28
13:28:56.233001: osd_op(client.11172360.0:516946146
rbd_data.36bfe359c4998.0d08 [set-alloc-hint object_size 4194304
write_size 4194304,write 1835008~143360] 17.3df49873 RETRY=1
ack+ondisk+retry+write+redirected+known_if_redirected e159001) currently
waiting for subops from 5,140,148

2016-04-28 13:29:28.031452 osd.148 10.217.72.11:6820/6487 25324 : cluster
[WRN] slow request 30.960922 seconds old, received at 2016-04-28
13:28:57.070471: osd_op(client.24127500.0:2960618
rbd_data.38178d8adeb4d.10f8 [set-alloc-hint object_size 8388608
write_size 8388608,write 3194880~4096] 17.fb41a37c RETRY=1
ack+ondisk+retry+write+redirected+known_if_redirected e159001) currently
waiting for subops from 41,115

from digging in the logs, it appears like some messages are being lost
between the OSDs. this is what osd.41 sees:
-
2016-04-28 13:28:56.233702 7f3b171e0700  1 -- 10.217.72.22:6802/3769 <==
client.11172360 10.217.72.41:0/6031968 6 
osd_op(client.11172360.0:516946146 rbd_data.36bfe359c4998.0d08
[set-alloc-hint object_size 4194304 write_size 4194304,write
1835008~143360] 17.3df49873 RETRY=1
ack+ondisk+retry+write+redirected+known_if_redirected e159001) v5 
236+0+143360 (781016428 0 3953649960) 0x1d551c00 con 0x1a78d9c0
2016-04-28 13:28:56.233983 7f3b49020700  1 -- 10.217.89.22:6825/313003769
--> 10.217.89.18:6806/1010441 -- osd_repop(client.11172360.0:516946146
17.73 3df49873/rbd_data.36bfe359c4998.0d08/head//17 v
159001'26722799) v1 -- ?+46 0x1d6db200 con 0x21add440
2016-04-28 13:28:56.234017 7f3b49020700  1 -- 10.217.89.22:6825/313003769
--> 10.217.89.11:6810/4543 -- osd_repop(client.11172360.0:516946146 17.73
3df49873/rbd_data.36bfe359c4998.0d08/head//17 v
159001'26722799) v1 -- ?+46 0x1d6dd000 con 0x21ada000
2016-04-28 13:28:56.234046 7f3b49020700  1 -- 10.217.89.22:6825/313003769
--> 10.217.89.11:6812/43006487 -- osd_repop(client.11172360.0:516946146
17.73 3df49873/rbd_data.36bfe359c4998.0d08/head//17 v
159001'26722799) v1 -- ?+144137 0x14becc00 con 0xf2cd4a0
2016-04-28 13:28:56.243555 7f3b35976700  1 -- 10.217.89.22:6825/313003769
<== osd.140 10.217.89.11:6810/4543 23 
osd_repop_reply(client.11172360.0:516946146 17.73 ondisk, result = 0) v1
 83+0+0 (494696391 0 0) 0x28ea7b00 con 0x21ada000
2016-04-28 13:28:56.257816 7f3b27d9b700  1 -- 10.217.89.22:6825/313003769
<== osd.5 10.217.89.18:6806/1010441 35 
osd_repop_reply(client.11172360.0:516946146 17.73 ondisk, result = 0) v1
 83+0+0 (2393425574 0 0) 0xfe82fc0 con 0x21add440


this, however is what osd.148 sees:
-
[ulhglive-root@ceph1 ~]# grep :516946146 /var/log/ceph/ceph-osd.148.log
2016-04-28 13:29:33.470156 7f195fcfc700  1 -- 10.217.72.11:6820/6487 <==
client.11172360 10.217.72.41:0/6031968 460 
osd_op(client.11172360.0:516946146 rbd_data.36bfe359c4998.0d08
[set-alloc-hint object_size 4194304 write_size 4194304,write
1835008~143360] 17.3df49873 RETRY=2
ack+ondisk+retry+write+redirected+known_if_redirected e159002) v5 
236+0+143360 (129493315 0 3953649960) 0x1edf2300 con 0x24dc0d60

also, due to the ceph osd down commands, there is recovery that needs to
happen for a PG shared between these OSDs that is never making any
progress. its probably due to whatever is cause the repops to fail.

i did some tcpdump on both sides limiting things to the ip addresses and
ports being used by these two OSDs and see packets flowing between the two
osds. i attempted to have wireshark decode the actual ceph traffic but it
was only able to get bits and pieces of the ceph protocol bits but at least
for the moment i'm blaming that on the ceph dissector for wireshark. there
aren't any dropped or error packets on any of the network interfaces
involved.

does anyone have any ideas of where to look next or other tips for this?
we've put debug_ms and debug_osd at 1/1 to get the bits of info mentioned.
putting them at 20 probably isn't going to be helpful so anyone have a
suggestion on another level to put it at that might be useful? go figure
that this would happen while i'm at the openstack summit and it would keep
me from paying attention to some interesting presentations.

thanks in advance for any help.

mike

Re: [ceph-users] Troubleshoot blocked OSDs

[Andrus, Brian Contractor]

Load on all nodes is 1.04 to 1.07
I am updating now to Jewel 10.2 (from 9.2)
This is CephFS with SSD journals.

Hopefully the update to jewel fixes lots.


Brian Andrus
ITACS/Research Computing
Naval Postgraduate School
Monterey, California
voice: 831-656-6238



From: Lincoln Bryant [mailto:linco...@uchicago.edu]
Sent: Thursday, April 28, 2016 12:56 PM
To: Andrus, Brian Contractor
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Troubleshoot blocked OSDs

OK, a few more questions.

What does the load look like on the OSDs with ‘iostat’ during the rsync?

What version of Ceph? Are you using RBD, CephFS, something else?

SSD journals or no?

—Lincoln

On Apr 28, 2016, at 2:53 PM, Andrus, Brian Contractor 
> wrote:

Lincoln,

That was the odd thing to me. Ceph health detail listed all 4 OSDs, so I 
checked all the systems.
I have since let it settle until it is OK again and started. Within a couple 
minutes, it started showing blocked requests and they are indeed on all 4 OSDs.

Brian Andrus
ITACS/Research Computing
Naval Postgraduate School
Monterey, California
voice: 831-656-6238



From: Lincoln Bryant [mailto:linco...@uchicago.edu]
Sent: Thursday, April 28, 2016 12:31 PM
To: Andrus, Brian Contractor
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Troubleshoot blocked OSDs

Hi Brian,

The first thing you can do is “ceph health detail”, which should give you some 
more information about which OSD(s) have blocked requests.

If it’s isolated to one OSD in particular, perhaps use iostat to check 
utilization and/or smartctl to check health.

—Lincoln

On Apr 28, 2016, at 2:26 PM, Andrus, Brian Contractor 
> wrote:

All,

I have a small ceph cluster with 4 OSDs and 3 MONs on 4 systems.
I was rsyncing about 50TB of files and things get very slow. To the point I 
stopped the rsync, but even with everything stopped, I see:

health HEALTH_WARN
80 requests are blocked > 32 sec

The number was as high as 218, but they seem to be draining down.
I see no issues on any of the systems, CPU load is low, memory usage is low.

How do I go about finding why a request is blocked for so long? These have been 
hitting >500 seconds for block time.

Brian Andrus
ITACS/Research Computing
Naval Postgraduate School
Monterey, California
voice: 831-656-6238

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Troubleshoot blocked OSDs

[Lincoln Bryant]

OK, a few more questions.

What does the load look like on the OSDs with ‘iostat’ during the rsync?

What version of Ceph? Are you using RBD, CephFS, something else? 

SSD journals or no?

—Lincoln

> On Apr 28, 2016, at 2:53 PM, Andrus, Brian Contractor  
> wrote:
> 
> Lincoln,
>  
> That was the odd thing to me. Ceph health detail listed all 4 OSDs, so I 
> checked all the systems.
> I have since let it settle until it is OK again and started. Within a couple 
> minutes, it started showing blocked requests and they are indeed on all 4 
> OSDs.
>  
> Brian Andrus
> ITACS/Research Computing
> Naval Postgraduate School
> Monterey, California
> voice: 831-656-6238
>  
>  
>   <>
> From: Lincoln Bryant [mailto:linco...@uchicago.edu 
> ] 
> Sent: Thursday, April 28, 2016 12:31 PM
> To: Andrus, Brian Contractor
> Cc: ceph-users@lists.ceph.com 
> Subject: Re: [ceph-users] Troubleshoot blocked OSDs
>  
> Hi Brian,
>  
> The first thing you can do is “ceph health detail”, which should give you 
> some more information about which OSD(s) have blocked requests.
>  
> If it’s isolated to one OSD in particular, perhaps use iostat to check 
> utilization and/or smartctl to check health. 
>  
> —Lincoln
>  
> On Apr 28, 2016, at 2:26 PM, Andrus, Brian Contractor  > wrote:
>  
> All,
>  
> I have a small ceph cluster with 4 OSDs and 3 MONs on 4 systems.
> I was rsyncing about 50TB of files and things get very slow. To the point I 
> stopped the rsync, but even with everything stopped, I see:
>  
> health HEALTH_WARN
> 80 requests are blocked > 32 sec
>  
> The number was as high as 218, but they seem to be draining down.
> I see no issues on any of the systems, CPU load is low, memory usage is low.
>  
> How do I go about finding why a request is blocked for so long? These have 
> been hitting >500 seconds for block time.
>  
> Brian Andrus
> ITACS/Research Computing
> Naval Postgraduate School
> Monterey, California
> voice: 831-656-6238
>  
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
> 
>  
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Troubleshoot blocked OSDs

[Andrus, Brian Contractor]

Lincoln,

That was the odd thing to me. Ceph health detail listed all 4 OSDs, so I 
checked all the systems.
I have since let it settle until it is OK again and started. Within a couple 
minutes, it started showing blocked requests and they are indeed on all 4 OSDs.

Brian Andrus
ITACS/Research Computing
Naval Postgraduate School
Monterey, California
voice: 831-656-6238



From: Lincoln Bryant [mailto:linco...@uchicago.edu]
Sent: Thursday, April 28, 2016 12:31 PM
To: Andrus, Brian Contractor
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Troubleshoot blocked OSDs

Hi Brian,

The first thing you can do is “ceph health detail”, which should give you some 
more information about which OSD(s) have blocked requests.

If it’s isolated to one OSD in particular, perhaps use iostat to check 
utilization and/or smartctl to check health.

—Lincoln

On Apr 28, 2016, at 2:26 PM, Andrus, Brian Contractor 
> wrote:

All,

I have a small ceph cluster with 4 OSDs and 3 MONs on 4 systems.
I was rsyncing about 50TB of files and things get very slow. To the point I 
stopped the rsync, but even with everything stopped, I see:

health HEALTH_WARN
80 requests are blocked > 32 sec

The number was as high as 218, but they seem to be draining down.
I see no issues on any of the systems, CPU load is low, memory usage is low.

How do I go about finding why a request is blocked for so long? These have been 
hitting >500 seconds for block time.

Brian Andrus
ITACS/Research Computing
Naval Postgraduate School
Monterey, California
voice: 831-656-6238

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Troubleshoot blocked OSDs

[Lincoln Bryant]

Hi Brian,

The first thing you can do is “ceph health detail”, which should give you some 
more information about which OSD(s) have blocked requests.

If it’s isolated to one OSD in particular, perhaps use iostat to check 
utilization and/or smartctl to check health. 

—Lincoln

> On Apr 28, 2016, at 2:26 PM, Andrus, Brian Contractor  
> wrote:
> 
> All,
>  
> I have a small ceph cluster with 4 OSDs and 3 MONs on 4 systems.
> I was rsyncing about 50TB of files and things get very slow. To the point I 
> stopped the rsync, but even with everything stopped, I see:
>  
> health HEALTH_WARN
> 80 requests are blocked > 32 sec
>  
> The number was as high as 218, but they seem to be draining down.
> I see no issues on any of the systems, CPU load is low, memory usage is low.
>  
> How do I go about finding why a request is blocked for so long? These have 
> been hitting >500 seconds for block time.
>  
> Brian Andrus
> ITACS/Research Computing
> Naval Postgraduate School
> Monterey, California
> voice: 831-656-6238
>  
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Troubleshoot blocked OSDs

[Andrus, Brian Contractor]

All,

I have a small ceph cluster with 4 OSDs and 3 MONs on 4 systems.
I was rsyncing about 50TB of files and things get very slow. To the point I 
stopped the rsync, but even with everything stopped, I see:

health HEALTH_WARN
80 requests are blocked > 32 sec

The number was as high as 218, but they seem to be draining down.
I see no issues on any of the systems, CPU load is low, memory usage is low.

How do I go about finding why a request is blocked for so long? These have been 
hitting >500 seconds for block time.

Brian Andrus
ITACS/Research Computing
Naval Postgraduate School
Monterey, California
voice: 831-656-6238

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Predictive Device Failure

[Simon Murray]

Having just discussed this topic with Sage, it was pointed out that this
functionality should (possibly will) be a core component of Ceph.  I've put
something together with a custom Nagios plug-in, Icinga2, InfluxDB and
Grafana which appears to work rather reliably.  Blog post available at
http://www.datacentred.co.uk/blog/integrating-icinga2-with-influxdb-and-grafana/
for inspiration.

In the interests of sharing and caring it'd be nice if other operators
could offer some insight into their best practices for spotting when things
are about to go wrong so we all can be proactive about maintaining a stable
service.

Si

-- 
DataCentred Limited registered in England and Wales no. 05611763
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Hammer broke after adding 3rd osd server

[Andrei Mikhailovsky]

Hello guys,

Done a bit of digging over the last few days and collected a bunch of logs from 
the osd servers including ops in flight. Will be digging through the data later 
on today.

I've also done a bunch of network connectivity tests and can verify that I did 
not find any evidence of network issues. The ping and hping (tcp) tests were 
running over the past few days and did not show any errors or packet drops or 
similar issues. The network interface stats reflect that as well. I've ran the 
network tests between all osds servers and cloud host servers. The network 
interfaces are configured to use the same mtu of 65520 (ipoib interface).

What I did notice today is once again, the problem tend to start between 7:20 
and 7:40 in the morning, produce a tons of slow requests between the old two 
osd servers and the new osd server. The slow requests go away for about 20-30 
mins and return and keep returning after about 20-30 minutes. During the slow 
requests the ceph-osd processes go nuts on the new osd server only. They 
consume all cpu and the server load goes to 300+. A few hours into the slow 
requests cycle i've stopped the ceph-osd processes on one of the old osd 
servers, but the problem does not go away. The only thing that help the cluster 
is a full reboot of the new osd server. After the reboot the slow requests do 
not come back until the next morning.

If anyone has an idea what else I could try, please let me know.

Andrei

- Original Message -
> From: "Wido den Hollander" 
> To: "andrei" 
> Cc: "ceph-users" 
> Sent: Tuesday, 26 April, 2016 22:18:37
> Subject: Re: [ceph-users] Hammer broke after adding 3rd osd server

>> Op 26 april 2016 om 22:31 schreef Andrei Mikhailovsky :
>> 
>> 
>> Hi Wido,
>> 
>> Thanks for your reply. We have a very simple ceph network. A single 40gbit/s
>> infiniband switch where the osd servers and hosts are connected to. There are
>> no default gates on the storage network. The IB is used only for ceph;
>> everything else goes over the ethernet.
>> 
>> I've checked the stats on the IB interfaces of osd servers and there are no
>> errors. The ipoib interface has a very small number of dropped packets
>> (0.0003%).
>> 
>> What kind of network tests would you suggest that I run? What do you mean by 
>> "I
>> would suggest that you check if the network towards clients is also OK."? By
>> clients do you mean the host servers?
>> 
> 
> With clients I mean you verify if the hosts talking to the Ceph cluster can
> reach each machine running OSDs.
> 
> In my case there was packet loss from certain clients which caused the issues 
> to
> occur.
> 
> Wido
> 
>> Many thanks
>> 
>> Andrei
>> 
>> - Original Message -
>> > From: "Wido den Hollander" 
>> > To: "ceph-users" , "Andrei Mikhailovsky"
>> > 
>> > Sent: Tuesday, 26 April, 2016 21:17:59
>> > Subject: Re: [ceph-users] Hammer broke after adding 3rd osd server
>> 
>> >> Op 26 april 2016 om 17:52 schreef Andrei Mikhailovsky :
>> >> 
>> >> 
>> >> Hello everyone,
>> >> 
>> >> I've recently performed a hardware upgrade on our small two osd server 
>> >> ceph
>> >> cluster, which seems to have broke the ceph cluster. We are using ceph for
>> >> cloudstack rbd images for vms.All of our servers are Ubuntu 14.04 LTS with
>> >> latest updates and kernel 4.4.6 from ubuntu repo.
>> >> 
>> >> Previous hardware:
>> >> 
>> >> 2 x osd servers with 9 sas osds, 32gb ram and 12 core Intel cpu 2620 @ 
>> >> 2Ghz each
>> >> and 2 consumer SSDs for journal. Infiniband 40gbit/s networking using 
>> >> IPoIB.
>> >> 
>> >> The following things were upgraded:
>> >> 
>> >> 1. journal ssds were upgraded from consumer ssd to Intel 3710 200gb. We 
>> >> now have
>> >> 5 osds per single ssd.
>> >> 2. added additional osd server with 64gb ram, 10 osds, Intel 2670 cpu @ 
>> >> 2.6Ghz
>> >> 3. Upgraded ram on osd servers to become 64gb
>> >> 4. Installed additional osd disk to have 10 osds per server.
>> >> 
>> >> After adding the third osd server and finishing the initial sync, the 
>> >> cluster
>> >> worked okay for 1-2 days. No issues were noticed. On a third day my 
>> >> monitoring
>> >> system started reporting a bunch of issues from the ceph cluster as well 
>> >> as
>> >> from our virtual machines. This tend to happen between 7:20am and 7:40am 
>> >> and
>> >> lasts for about 2-3 hours before things become normal again. I've checked 
>> >> the
>> >> osd servers and there is nothing that I could find in cron or otherwise 
>> >> that
>> >> starts around 7:20am.
>> >> 
>> >> The problem is as follows: the new osd server's load goes to 400+ with 
>> >> ceph-osd
>> >> processes consuming all cpu resources. The ceph -w shows a high number of 
>> >> slow
>> >> requests which relate to osds belonging to the new osd server. The log 
>> >> files
>> >> show the following:
>> >> 
>> 

Re: [ceph-users] about slides on VAULT of 2016

[Sage Weil]

Hi,

Here are the slides:

http://www.slideshare.net/sageweil1/bluestore-a-new-faster-storage-backend-for-ceph

sage

On Thu, 28 Apr 2016, 席智勇 wrote:

> hi sage:
>        I find the slides of VAULT of 2016 on this
> page(http://events.linuxfoundation.org/events/vault/program/slides),
> it seems not the whole accoding to the schedule info, and I didn't find
> yours. Can you share your slides or any things usefull on VAULT about
> BlueStore.
> 
> 
> regards~ 
> zhiyong
> 
> ___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw crash - Infernalis

[Karol Mroz]

On Thu, Apr 28, 2016 at 05:48:33AM -0400, Brad Hubbard wrote:
[...]
> The offsets are relative to the address where the function is loaded in memory
> and I don't think searching for 0x6d, 0x2fb, 0x62a or 0x75a will give you the
> correct result if you don't know which function you are dealing with.  The
> offset is just an offset from the start of *some function* so without knowing
> which function we can't be sure what instruction we were on.  That's my
> understanding anyway.

Then I was reading the trace incorrectly. I assumed the address pointed to the
instruction in the disassembled binary :/ My mistake.

Thanks!

-- 
Regards,
Karol


signature.asc
Description: Digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RBD image mounted by command "rbd-nbd" the status is read-only.

[Mykola Golub]

On Mon, Apr 25, 2016 at 08:09:54PM +0200, Ilya Dryomov wrote:
> On Mon, Apr 25, 2016 at 7:47 PM, Stefan Lissmats  wrote:
> > Hello again!
> >
> > I understand that it's not recommended running osd and rbd-nbd on the same 
> > host and i actually moved my rbd-nbd to a completely clean host (same 
> > kernel and OS though), but with same result.
> >
> > I hope someone can resolve this and you seem to indicate it is some kind of 
> > known error but i didn't really understand the github commit that you 
> > linked.
> 
> Yes, it is a bug.  rbd-nbd code expects writes to have rval (return
> code) equal to the size of the write.  I'm pretty sure that's wrong,
> because rval for writes should be 0 or a negative error.
> 
> I think what happens is your writes complete successfully, but rbd-nbd
> then throws an -EIO to the kernel because 0 != write size.  I could be
> wrong, so let's wait for Mykola to chime in - he added that check to
> fix discards.

Sorry for delay (I missed this thread due to a wrong filter).

I don't recall details but I think I had an impression that on success
aio_write completion returned the number of bytes written. I might be
confused by this test that checks for r >= 0:

https://github.com/ceph/ceph/blob/master/src/test/librbd/test_librbd.cc#L1254

Now, looking at it again, it is certainly not true and my patch is
wrong.

I see the fix is already requested:

https://github.com/ceph/ceph/pull/8775/

Thanks.

-- 
Mykola Golub
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] hadoop on cephfs

[Oliver Dzombic]

Hi,

bad idea :-)

Its of course nice and important to drag developer towards a
new/promising technology/software.

But if the technology under the individual required specifications does
not match, you will just risk to show this developer how worst this
new/promising technology is.

So you will just reach the opposite of what you want.

So before you are doing something, usually big, like hadoop on an
unstable software, maybe you should not use it.

For the good of the developer, for your good and for the good of the
reputation of the new/promising technology/software you wish.

To force a pinguin to somehow live in the sahara, might be possible ( at
least for some time ), but usually not a good idea ;-)

-- 
Mit freundlichen Gruessen / Best regards

Oliver Dzombic
IP-Interactive

mailto:i...@ip-interactive.de

Anschrift:

IP Interactive UG ( haftungsbeschraenkt )
Zum Sonnenberg 1-3
63571 Gelnhausen

HRB 93402 beim Amtsgericht Hanau
Geschäftsführung: Oliver Dzombic

Steuer Nr.: 35 236 3622 1
UST ID: DE274086107


Am 28.04.2016 um 08:31 schrieb Bill Sharer:
> Just got into a discussion today where I may have a chance to do work
> with a db guy who wants hadoop and I want to steer him to it on cephfs. 
> While I'd really like to run gentoo with either infernalis or jewel
> (when it becomes stable in portage), odds are more likely that I will be
> required to use rhel/centos6.7 and thus stuck back at Hammer.  Any
> thoughts?
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw crash - Infernalis

[Brad Hubbard]

- Original Message -
> From: "Karol Mroz" 
> To: "Brad Hubbard" 
> Cc: "Ben Hines" , "ceph-users" 
> Sent: Thursday, 28 April, 2016 7:17:05 PM
> Subject: Re: [ceph-users] radosgw crash - Infernalis
> 
> Hi Brad,
> 
> On Wed, Apr 27, 2016 at 11:40:40PM -0400, Brad Hubbard wrote:
> [...]
> > > 0030a810 <_Z13pidfile_writePK11md_config_t@@Base>:
> > > ...
> > >   30b09d:   e8 0e 40 e4 ff  callq  14f0b0 
> > >   30b0a2:   4c 89 efmov%r13,%rdi
> > >   ---
> > > ...
> > > 
> > > So either we tripped backtrace() code from pidfile_write() _or_ we can't
> > > trust the stack. From the log snippet, it looks that we're far past the
> > > point
> > > at which we would write a pidfile to disk (ie. at process start during
> > > global_init()).
> > > Rather, we're actually handling a request and outputting some bit of
> > > debug
> > > message
> > > via MSDOp::print() and beyond...
> > 
> > It would help to know what binary this is and what OS.
> > 
> > We know the offset into the function is 0x30b0a2 but we don't know which
> > function yet AFAICT. Karol, how did you arrive at pidfile_write? Purely
> > from
> > the offset? I'm not sure that would be reliable...
> 
> Correct, from the offset. Let me clarify, I don't think pidfile_write() is
> the
> function in which we segfaulted :) Hence my suspicion of a blown stack. I

You could definitely be on the money here but IMHO it is too early to tell.

> don't
> know the specifics behind the backtrace call used to generate this stack...
> so
> maybe this is a naive question... but why do you think the offset is
> unreliable?
> Perhaps I'm not reading this trace correctly?

Well, you could have multiple functions which include an offset of 0x30b0a2.
Which function would it be in that case? The other frame shows an offset of
0xf100, can you identify that function just from the offset?

The following stack gives some good examples.

 1: /usr/bin/ceph-osd() [0xa05e32]
 2: (()+0xf100) [0x7f9ea295c100]
 3: (OSD::handle_osd_ping(MOSDPing*)+0x75a) [0x659e7a]
 4: (OSD::heartbeat_dispatch(Message*)+0x2fb) [0x65b0cb]
 5: (DispatchQueue::entry()+0x62a) [0xbc2aba]
 6: (DispatchQueue::DispatchThread::entry()+0xd) [0xae572d]
 7: (()+0x7dc5) [0x7f9ea2954dc5]
 8: (clone()+0x6d) [0x7f9ea143528d]

The offsets are relative to the address where the function is loaded in memory
and I don't think searching for 0x6d, 0x2fb, 0x62a or 0x75a will give you the
correct result if you don't know which function you are dealing with.  The
offset is just an offset from the start of *some function* so without knowing
which function we can't be sure what instruction we were on.  That's my
understanding anyway.

I agree that a stack with only two frames looks dodgy though and we may be
chasing our tails but I'm hoping we can squeeze more info out of a core or a
better stack trace with all debuginfo loaded (if the function has no name due
to lack of debuginfo and not due to stack corruption).

Cheers,
Brad

> 
> > 
> > This is a segfault so the address of the frame where we crashed should be
> > the
> > exact instruction where we crashed. I don't believe a mov from one register
> > to
> > another that does not involve a dereference ((%r13) as opposed to %r13) can
> > cause a segfault so I don't think we are on the right instruction but then,
> > as
> > you say, the stack may be corrupt.
> 
> Agreed... a mov between registers wouldn't cause a segfault.
> 
> --
> Regards,
> Karol
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] NO mon start after Jewel Upgrade using systemctl

[Karsten Heymann]

Interesting, I have

root@ceph-cap1-02:~# systemctl list-unit-files | grep ceph
ceph-create-keys@.service  static
ceph-disk@.service enabled
ceph-mds@.service  disabled
ceph-mon@.service  enabled
ceph-osd@.service  enabled
ceph-radosgw@.service  disabled
ceph.service   masked
ceph-mon.targetenabled
ceph-osd.targetenabled
ceph.targetenabled

root@ceph-cap1-02:~# apt-show-versions | grep ^ceph
ceph:amd64/jessie 10.2.0-1~bpo80+1 uptodate
ceph-base:amd64/jessie 10.2.0-1~bpo80+1 uptodate
ceph-common:amd64/jessie 10.2.0-1~bpo80+1 uptodate
ceph-deploy:all/jessie 1.5.33 uptodate
ceph-fs-common:amd64/jessie 10.2.0-1~bpo80+1 uptodate
ceph-fuse:amd64/jessie 10.2.0-1~bpo80+1 uptodate
ceph-mds:amd64/jessie 10.2.0-1~bpo80+1 uptodate
ceph-mon:amd64/jessie 10.2.0-1~bpo80+1 uptodate
ceph-osd:amd64/jessie 10.2.0-1~bpo80+1 uptodate

But I just saw you named you mon service
>  [root@cephmon03 ~]# systemctl status ceph-mon@3

I would recommend to use

# systemctl enable ceph-mon@$(hostname -s)
# systemctl start ceph-mon@$(hostname -s)

instead. As far as I know, numbers are only used for osd services,
mon- and mds-services use the short hostname to identify themselve.

Best regards
Karsten

2016-04-27 19:54 GMT+02:00 Iban Cabrillo :
> Hi Karsten,
>I have checked taht files arethe same that git ones.
>
>   -rw-r--r-- 1 root root 810 Apr 20 18:45
> /lib/systemd/system/ceph-mon@.service
>   -rw-r--r-- 1 root root 162 Apr 20 18:45
> /lib/systemd/system/ceph-mon.target
>
> [root@cephmon03 ~]# cat /lib/systemd/system/ceph-mon.target
> [Unit]
> Description=ceph target allowing to start/stop all ceph-mon@.service
> instances at once
> PartOf=ceph.target
> [Install]
> WantedBy=multi-user.target ceph.target
>
> [root@cephmon03 ~]# systemctl list-unit-files|grep ceph
> ceph-create-keys@.service  static
> ceph-disk@.service static
> ceph-mds@.service  disabled
> ceph-mon@.service  disabled
> ceph-osd@.service  disabled
> ceph-radosgw@.service  disabled
> ceph.service   masked
> ceph-mds.targetdisabled
> ceph-mon.targetenabled
> ceph-osd.targetdisabled
> ceph-radosgw.targetdisabled
> ceph.targetdisabled
>
> But still doesn't work (The upgrade was made from latest Hammer version )
> and it is running on CentOS 7. This instance is running a mon service only.
>
> [root@cephmon03 ~]# rpm -qa | grep ceph
> ceph-release-1-1.el7.noarch
> ceph-common-10.2.0-0.el7.x86_64
> ceph-mds-10.2.0-0.el7.x86_64
> libcephfs1-10.2.0-0.el7.x86_64
> python-cephfs-10.2.0-0.el7.x86_64
> ceph-selinux-10.2.0-0.el7.x86_64
> ceph-mon-10.2.0-0.el7.x86_64
> ceph-osd-10.2.0-0.el7.x86_64
> ceph-radosgw-10.2.0-0.el7.x86_64
> ceph-base-10.2.0-0.el7.x86_64
> ceph-10.2.0-0.el7.x86_64
>
> I have test it with ceph.target, with the same result.
>
> regards, I
>
>
>
>
> 2016-04-27 15:13 GMT+02:00 Karsten Heymann :
>>
>> Hi Iban,
>>
>> the current jewel packages seem to be missing some important systemd
>> files. Try to copy
>> https://github.com/ceph/ceph/blob/master/systemd/ceph-mon.target to
>> /lib/systemd/system and enable it:
>>
>> systemctl enable ceph-mon.target
>>
>> I also would disable the legacy init script with
>>
>> systemctl mask ceph.service
>>
>> There are already several open pull requests regarding this issue
>>
>> (https://github.com/ceph/ceph/pulls?utf8=%E2%9C%93=is%3Apr+is%3Aopen+systemd),
>> so I hope it will be fixed with the next point release.
>>
>> Best regards
>> Karsten
>>
>> 2016-04-27 14:18 GMT+02:00 Iban Cabrillo :
>> > Hi cephers,
>> >   I've been following the upgrade intrucctions...but..I sure I did
>> > something
>> > wrong.
>> >
>> >   I just upgrade using ceph-deploy on one monitor (after ofcourse down
>> > de
>> > mon service).
>> >   Then  the chow to var/lib/ceph and /var/log/ceph for ceph user
>> >
>> >   [root@cephmon03 ~]# systemctl start ceph.target
>> > [root@cephmon03 ~]#
>> > [root@cephmon03 ~]#
>> > [root@cephmon03 ~]# systemctl status ceph.target
>> > ● ceph.target - ceph target allowing to start/stop all ceph*@.service
>> > instances at once
>> >Loaded: loaded (/usr/lib/systemd/system/ceph.target; disabled; vendor
>> > preset: disabled)
>> >Active: active since mié 2016-04-27 13:43:24 CEST; 10min ago
>> >
>> > abr 27 13:43:24 cephmon03.ifca.es systemd[1]: Reached target ceph target
>> > allowing to start/stop all ceph*@.service instances at once.
>> > abr 27 13:43:24 cephmon03.ifca.es systemd[1]: Starting ceph target
>> > allowing
>> > to start/stop all ceph*@.service instances at once.
>> > 

Re: [ceph-users] radosgw crash - Infernalis

[Karol Mroz]

Hi Brad,

On Wed, Apr 27, 2016 at 11:40:40PM -0400, Brad Hubbard wrote:
[...]
> > 0030a810 <_Z13pidfile_writePK11md_config_t@@Base>:
> > ...
> >   30b09d:   e8 0e 40 e4 ff  callq  14f0b0 
> >   30b0a2:   4c 89 efmov%r13,%rdi
> >   ---
> > ...
> > 
> > So either we tripped backtrace() code from pidfile_write() _or_ we can't
> > trust the stack. From the log snippet, it looks that we're far past the 
> > point
> > at which we would write a pidfile to disk (ie. at process start during
> > global_init()).
> > Rather, we're actually handling a request and outputting some bit of debug
> > message
> > via MSDOp::print() and beyond...
> 
> It would help to know what binary this is and what OS.
> 
> We know the offset into the function is 0x30b0a2 but we don't know which
> function yet AFAICT. Karol, how did you arrive at pidfile_write? Purely from
> the offset? I'm not sure that would be reliable...

Correct, from the offset. Let me clarify, I don't think pidfile_write() is the
function in which we segfaulted :) Hence my suspicion of a blown stack. I don't
know the specifics behind the backtrace call used to generate this stack... so
maybe this is a naive question... but why do you think the offset is unreliable?
Perhaps I'm not reading this trace correctly?

> 
> This is a segfault so the address of the frame where we crashed should be the
> exact instruction where we crashed. I don't believe a mov from one register to
> another that does not involve a dereference ((%r13) as opposed to %r13) can
> cause a segfault so I don't think we are on the right instruction but then, as
> you say, the stack may be corrupt.

Agreed... a mov between registers wouldn't cause a segfault.

-- 
Regards,
Karol


signature.asc
Description: Digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] hadoop on cephfs

[Bill Sharer]

Just got into a discussion today where I may have a chance to do work 
with a db guy who wants hadoop and I want to steer him to it on cephfs.  
While I'd really like to run gentoo with either infernalis or jewel 
(when it becomes stable in portage), odds are more likely that I will be 
required to use rhel/centos6.7 and thus stuck back at Hammer.  Any thoughts?

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Enforce MDS map update in CephFS kernel driver

[Burkhard Linke]

Hi,

we recently stumbled over a problem with the kernel based CephFS driver 
(Ubuntu Trusty with 4.4.0-18 kernel from xenial lts backport package). 
Our MDS failed for some unknown reason, and the standby MDS became active.


After rejoining the MDS cluster, the former standby MDS stuck at the 
clientreplay state. Clients were not able to connect to it. We had to 
fail back to the original MDS to recover clients:


[Wed Apr 27 11:17:48 2016] ceph: mds0 hung
[Wed Apr 27 11:36:30 2016] ceph: mds0 came back
[Wed Apr 27 11:36:30 2016] ceph: mds0 caps went stale, renewing
[Wed Apr 27 11:36:30 2016] ceph: mds0 caps stale
[Wed Apr 27 11:36:33 2016] libceph: mds0 192.168.6.132:6809 socket 
closed (con state OPEN)

[Wed Apr 27 11:36:38 2016] libceph: mds0 192.168.6.132:6809 connection reset
[Wed Apr 27 11:36:38 2016] libceph: reset on mds0
[Wed Apr 27 11:36:38 2016] ceph: mds0 closed our session
[Wed Apr 27 11:36:38 2016] ceph: mds0 reconnect start
[Wed Apr 27 11:36:39 2016] ceph: mds0 reconnect denied
[Wed Apr 27 12:03:32 2016] libceph: mds0 192.168.6.132:6800 socket 
closed (con state OPEN)
[Wed Apr 27 12:03:33 2016] libceph: mds0 192.168.6.132:6800 socket 
closed (con state CONNECTING)
[Wed Apr 27 12:03:34 2016] libceph: mds0 192.168.6.132:6800 socket 
closed (con state CONNECTING)
[Wed Apr 27 12:03:35 2016] libceph: mds0 192.168.6.132:6800 socket 
closed (con state CONNECTING)
[Wed Apr 27 12:03:37 2016] libceph: mds0 192.168.6.132:6800 socket 
closed (con state CONNECTING)
[Wed Apr 27 12:03:41 2016] libceph: mds0 192.168.6.132:6800 socket 
closed (con state CONNECTING)

[Wed Apr 27 12:03:50 2016] ceph: mds0 reconnect start
[Wed Apr 27 12:03:50 2016] ceph: mds0 reconnect success
[Wed Apr 27 12:03:55 2016] ceph: mds0 recovery completed

(192.168.6.132 being the standby MDS)

The problem is similar to the one described in this mail thread from 
september:


http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-September/004070.html

My questions are:

- Does a recent kernel include the fix to react to MDS map changes?
- If this is the case, which is the upstream kernel release including 
the changes?
- Is it possible to manipulate the MDS map manually, e.g. by 
/sys/kernel/debug/ceph//mdsmap ?
- Does using a second MDS in active/active setup provide a way to handle 
this situation, although the configuration is not recommended (yet)?


Regards,
Burkhard
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com