Re: [ceph-users] CEPH mirror down again
Hmm. Apparently download.ceph.com = us-west.ceph.com
And there is no repomd.xml on us-east.ceph.com
This seems to happen a little too often for something that is stable and
released. Makes it seem like the old BBS days of “I want to play DOOM, so I’m
shutting the services down”
Brian Andrus
ITACS/Research Computing
Naval Postgraduate School
Monterey, California
voice: 831-656-6238
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Vy
Nguyen Tan
Sent: Friday, November 25, 2016 7:28 PM
To: Joao Eduardo Luis
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] CEPH mirror down again
Hi Matt and Joao,
Thank you for your information. I am installing Ceph with alternative mirror
(ceph-deploy install --repo-url http://hk.ceph.com/rpm-jewel/el7/ --gpg-url
http://hk.ceph.com/keys/release.asc {host}) and everything work again.
On Sat, Nov 26, 2016 at 10:12 AM, Joao Eduardo Luis
mailto:j...@suse.de>> wrote:
On 11/26/2016 03:05 AM, Vy Nguyen Tan wrote:
Hello,
I want to install CEPH on new nodes but I can't reach CEPH repo, It
seems the repo are broken. I am using CentOS 7.2 and ceph-deploy 1.5.36.
Patrick sent an email to the list informing this would happen back on Nov 18th;
quote:
Due to Dreamhost shutting down the old DreamCompute cluster in their
US-East 1 region, we are in the process of beginning the migration of
Ceph infrastructure. We will need to move
download.ceph.com,
tracker.ceph.com, and
docs.ceph.com to their US-East 2 region.
The current plan is to move the VMs on 25 NOV 2016 throughout the day.
Expect them to be down intermittently.
-Joao
P.S.: also, it's Ceph; not CEPH.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] too many PGs per OSD (326 > max 300) warning when ALL PGs are 256
David, Thanks for the info. I am getting an understanding of how this works. Now I used the ceph-deploy tool to create the rgw pools. It seems then that the tool isn’t the best at creating the pools necessary for an rgw gateway as it made all of them the default sizes for pg_num/pgp_num Perhaps, then, it is wiser to have a very low default for those so the ceph-deploy tool doesn assign a large value to something that will merely hold control or other metadata? Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 From: David Turner [mailto:david.tur...@storagecraft.com] Sent: Monday, October 10, 2016 10:33 AM To: Andrus, Brian Contractor ; ceph-users@lists.ceph.com Subject: RE: [ceph-users] too many PGs per OSD (326 > max 300) warning when ALL PGs are 256 You have 11 pools with 256 pgs, 1 pool with 128 and 1 pool with 64... that's 3,008 pgs in your entire cluster. Multiply that number by your replica size and divide by how many OSDs you have in your cluster and you'll see what your average PGs per osd is. Based on the replica size you shared, that's a total number of 6,528 copies of PGs to be divided amongst the OSDS in your cluster. Your cluster will be in warning if that number is greater than 300 per OSD, like you're seeing. When designing your cluster and how many pools, pgs, and replica size you will be setting, please consult the pgcalc tool found here http://ceph.com/pgcalc/. You cannot reduce the number of PGs in a pool, so the easiest way to resolve this issue is mostly likely going to be destroying pools and recreating them with the proper number of PGs. The PG number should be based on what percentage of the data in your cluster will be in this pool. If I'm planning to have about 1024 PGs total in my cluster and I give 256 PGs to 4 different pools, then what I'm saying is that each of those 4 pools will have the exact same amount of data as each other. On the other hand, if I believe that 1 of those pools will have 90% of the data and the other 3 pools will have very little data, then I'll probably give the larger pool 1024 PGs and the rest of them 64 PGs (or less depending on what I'm aiming for). It is beneficial to keep the pg_num and pgp_num counts to base 2 numbers. [cid:image001.jpg@01D222E8.1655BB40]<https://storagecraft.com> David Turner | Cloud Operations Engineer | StorageCraft Technology Corporation<https://storagecraft.com> 380 Data Drive Suite 300 | Draper | Utah | 84020 Office: 801.871.2760 | Mobile: 385.224.2943 If you are not the intended recipient of this message or received it erroneously, please notify the sender and delete it, together with any attachments, and be advised that any dissemination or copying of this message is prohibited. From: ceph-users [ceph-users-boun...@lists.ceph.com] on behalf of Andrus, Brian Contractor [bdand...@nps.edu] Sent: Monday, October 10, 2016 11:14 AM To: ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> Subject: [ceph-users] too many PGs per OSD (326 > max 300) warning when ALL PGs are 256 Ok, this is an odd one to me… I have several pools, ALL of them are set with pg_num and pgp_num = 256. Yet, the warning about too many PGs per OSD is showing up. Here are my pools: pool 0 'rbd' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 134 flags hashpspool stripe_width 0 pool 1 'cephfs_data' replicated size 3 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 203 flags hashpspool crash_replay_interval 45 stripe_width 0 pool 2 'cephfs_metadata' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 196 flags hashpspool stripe_width 0 pool 3 'vmimages' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 213 flags hashpspool stripe_width 0 removed_snaps [1~3] pool 25 '.rgw.root' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 6199 flags hashpspool stripe_width 0 pool 26 'default.rgw.control' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 6202 flags hashpspool stripe_width 0 pool 27 'default.rgw.data.root' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 6204 flags hashpspool stripe_width 0 pool 28 'default.rgw.gc' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 6205 flags hashpspool stripe_width 0 pool 29 'default.rgw.log' replicated size 2 min_size 1 crush_
[ceph-users] too many PGs per OSD (326 > max 300) warning when ALL PGs are 256
Ok, this is an odd one to me... I have several pools, ALL of them are set with pg_num and pgp_num = 256. Yet, the warning about too many PGs per OSD is showing up. Here are my pools: pool 0 'rbd' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 134 flags hashpspool stripe_width 0 pool 1 'cephfs_data' replicated size 3 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 203 flags hashpspool crash_replay_interval 45 stripe_width 0 pool 2 'cephfs_metadata' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 196 flags hashpspool stripe_width 0 pool 3 'vmimages' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 213 flags hashpspool stripe_width 0 removed_snaps [1~3] pool 25 '.rgw.root' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 6199 flags hashpspool stripe_width 0 pool 26 'default.rgw.control' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 6202 flags hashpspool stripe_width 0 pool 27 'default.rgw.data.root' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 6204 flags hashpspool stripe_width 0 pool 28 'default.rgw.gc' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 6205 flags hashpspool stripe_width 0 pool 29 'default.rgw.log' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 6206 flags hashpspool stripe_width 0 pool 30 'default.rgw.users.uid' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 6211 flags hashpspool stripe_width 0 pool 31 'default.rgw.meta' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 6214 flags hashpspool stripe_width 0 pool 32 'default.rgw.buckets.index' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 6216 flags hashpspool stripe_width 0 pool 33 'default.rgw.buckets.data' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 6218 flags hashpspool stripe_width 0 so why would the warning show up, and how do I get it to go away and stay away? Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] too many PGs per OSD when pg_num = 256??
Hmm. Something happened then. I only have 20 OSDs. What may cause that? Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 From: David Turner [mailto:david.tur...@storagecraft.com] Sent: Thursday, September 22, 2016 10:04 AM To: Andrus, Brian Contractor ; ceph-users@lists.ceph.com Subject: RE: too many PGs per OSD when pg_num = 256?? So you have 3,520 pgs. Assuming all of your pools are using 3 replicas, and using the 377 pgs/osd in your health_warn state, that would mean your cluster has 28 osds. When you calculate how many pgs a pool should have, you need to account for how many osds you have, how much percentage of data each pool will account for out of your entire cluster, and go from there. The ceph PG Calc tool will be an excellent resource to help you figure out how many pgs each pool should have. It takes all of those factors into account. http://ceph.com/pgcalc/ [cid:image001.jpg@01D214B9.781F53F0]<https://storagecraft.com> David Turner | Cloud Operations Engineer | StorageCraft Technology Corporation<https://storagecraft.com> 380 Data Drive Suite 300 | Draper | Utah | 84020 Office: 801.871.2760 | Mobile: 385.224.2943 If you are not the intended recipient of this message or received it erroneously, please notify the sender and delete it, together with any attachments, and be advised that any dissemination or copying of this message is prohibited. ____ From: Andrus, Brian Contractor [bdand...@nps.edu] Sent: Thursday, September 22, 2016 10:41 AM To: David Turner; ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> Subject: RE: too many PGs per OSD when pg_num = 256?? David, I have 15 pools: # ceph osd lspools|sed 's/,/\n/g' 0 rbd 1 cephfs_data 2 cephfs_metadata 3 vmimages 14 .rgw.root 15 default.rgw.control 16 default.rgw.data.root 17 default.rgw.gc 18 default.rgw.log 19 default.rgw.users.uid 20 default.rgw.users.keys 21 default.rgw.users.email 22 default.rgw.meta 23 default.rgw.buckets.index 24 default.rgw.buckets.data # ceph -s | grep -Eo '[0-9]+ pgs' 3520 pgs Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 From: David Turner [mailto:david.tur...@storagecraft.com] Sent: Thursday, September 22, 2016 8:57 AM To: Andrus, Brian Contractor mailto:bdand...@nps.edu>>; ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> Subject: RE: too many PGs per OSD when pg_num = 256?? Forgot the + for the regex. ceph -s | grep -Eo '[0-9]+ pgs' [cid:image001.jpg@01D214B9.781F53F0]<https://storagecraft.com> David Turner | Cloud Operations Engineer | StorageCraft Technology Corporation<https://storagecraft.com> 380 Data Drive Suite 300 | Draper | Utah | 84020 Office: 801.871.2760 | Mobile: 385.224.2943 If you are not the intended recipient of this message or received it erroneously, please notify the sender and delete it, together with any attachments, and be advised that any dissemination or copying of this message is prohibited. ____ From: David Turner Sent: Thursday, September 22, 2016 9:53 AM To: Andrus, Brian Contractor; ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> Subject: RE: too many PGs per OSD when pg_num = 256?? How many pools do you have? How many pgs does your total cluster have, not just your rbd pool? ceph osd lspools ceph -s | grep -Eo '[0-9] pgs' My guess is that you have other pools with pgs and the cumulative total of pgs per osd is too many. ____________ From: ceph-users [ceph-users-boun...@lists.ceph.com] on behalf of Andrus, Brian Contractor [bdand...@nps.edu] Sent: Thursday, September 22, 2016 9:33 AM To: ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> Subject: [ceph-users] too many PGs per OSD when pg_num = 256?? All, I am getting a warning: health HEALTH_WARN too many PGs per OSD (377 > max 300) pool cephfs_data has many more objects per pg than average (too few pgs?) yet, when I check the settings: # ceph osd pool get rbd pg_num pg_num: 256 # ceph osd pool get rbd pgp_num pgp_num: 256 How does something like this happen? I did create a radosgw several weeks ago and have put a single file in it for testing, but that is it. It only started giving the warning a couple days ago. Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] too many PGs per OSD when pg_num = 256??
David, I have 15 pools: # ceph osd lspools|sed 's/,/\n/g' 0 rbd 1 cephfs_data 2 cephfs_metadata 3 vmimages 14 .rgw.root 15 default.rgw.control 16 default.rgw.data.root 17 default.rgw.gc 18 default.rgw.log 19 default.rgw.users.uid 20 default.rgw.users.keys 21 default.rgw.users.email 22 default.rgw.meta 23 default.rgw.buckets.index 24 default.rgw.buckets.data # ceph -s | grep -Eo '[0-9]+ pgs' 3520 pgs Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 From: David Turner [mailto:david.tur...@storagecraft.com] Sent: Thursday, September 22, 2016 8:57 AM To: Andrus, Brian Contractor ; ceph-users@lists.ceph.com Subject: RE: too many PGs per OSD when pg_num = 256?? Forgot the + for the regex. ceph -s | grep -Eo '[0-9]+ pgs' [cid:image001.jpg@01D214B5.3D5480F0]<https://storagecraft.com> David Turner | Cloud Operations Engineer | StorageCraft Technology Corporation<https://storagecraft.com> 380 Data Drive Suite 300 | Draper | Utah | 84020 Office: 801.871.2760 | Mobile: 385.224.2943 If you are not the intended recipient of this message or received it erroneously, please notify the sender and delete it, together with any attachments, and be advised that any dissemination or copying of this message is prohibited. From: David Turner Sent: Thursday, September 22, 2016 9:53 AM To: Andrus, Brian Contractor; ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> Subject: RE: too many PGs per OSD when pg_num = 256?? How many pools do you have? How many pgs does your total cluster have, not just your rbd pool? ceph osd lspools ceph -s | grep -Eo '[0-9] pgs' My guess is that you have other pools with pgs and the cumulative total of pgs per osd is too many. From: ceph-users [ceph-users-boun...@lists.ceph.com] on behalf of Andrus, Brian Contractor [bdand...@nps.edu] Sent: Thursday, September 22, 2016 9:33 AM To: ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> Subject: [ceph-users] too many PGs per OSD when pg_num = 256?? All, I am getting a warning: health HEALTH_WARN too many PGs per OSD (377 > max 300) pool cephfs_data has many more objects per pg than average (too few pgs?) yet, when I check the settings: # ceph osd pool get rbd pg_num pg_num: 256 # ceph osd pool get rbd pgp_num pgp_num: 256 How does something like this happen? I did create a radosgw several weeks ago and have put a single file in it for testing, but that is it. It only started giving the warning a couple days ago. Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] too many PGs per OSD when pg_num = 256??
All, I am getting a warning: health HEALTH_WARN too many PGs per OSD (377 > max 300) pool cephfs_data has many more objects per pg than average (too few pgs?) yet, when I check the settings: # ceph osd pool get rbd pg_num pg_num: 256 # ceph osd pool get rbd pgp_num pgp_num: 256 How does something like this happen? I did create a radosgw several weeks ago and have put a single file in it for testing, but that is it. It only started giving the warning a couple days ago. Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] RADOSGW and LDAP
All, I have been making some progress on troubleshooting this. I am seeing that when rgw is configured for LDAP, I am getting an error in my slapd log: Sep 14 06:56:21 mgmt1 slapd[23696]: conn=1762 op=0 RESULT tag=97 err=2 text=historical protocol version requested, use LDAPv3 instead Am I correct with an interpretation that rgw does not do LDAPv3? Is there a way to enable this, or must I allow older versions in my OpenLDAP configuration? Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] LDAP and RADOSGW
All, I am working on getting RADOSGW to work with LDAP and things seem like they should be set, but I suspect that there are certain attributes that need to exist for the user to work. If I create a user using "radosgw-admin user create", I am able to use that access/secret key successfully, but when I try using an existing LDAP user, I get Access Denied. Are there specific attributes that need to exist for an LDAP user for radosgw to use the entry? Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] creating rados S3 gateway
All, I'm having trouble using ceph-deploy to create a rados gateway. I initially did it and it worked, but my default pg_num was too large so it was complaining about that. To remedy, I stopped the ceph-radosgw service and deleted the pools that were created. default.rgw.log default.rgw.gc default.rgw.data.root default.rgw.control .rgw.root .rgw.root Then, using ceph-deploy, I editing my ceph.conf to set the osd_pool_default_pg_num and osd_pool_default_pgp_num to 256 and pushed the new config out. I verified the change with ceph --show-config I ran ceph-deploy rgw create again and it still made all the pools with 1024 as pg_num and pgp_num How do I properly remove all the rados gateway artifacts and get ceph-deploy to use my setting of 256 to create all the pools for a new gateway? Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] OSDs automount all devices on a san
All,
I have found an issue with ceph OSDs that are on a SAN and Multipathed. It may
not matter that they are multipathed, but that is how our setup is where I
found the issue.
Our setup has an infiniband network which uses SRP to annunciate block devices
on a DDN.
Every LUN can be seen by every node that loads the SRP drivers. That would be
my OSSes.
I can create OSDs such that each node will have one OSD from what is available:
ceph-deploy osd create ceph-1-35a:/dev/mapper/mpathb:/dev/sda5 \
ceph-1-35b:/dev/mapper/mpathc:/dev/sda5 \
ceph-1-36a:/dev/mapper/mpathd:/dev/sda5 \
ceph-1-36b:/dev/mapper/mpathe:/dev/sda5
This creates the OSD and puts the journal as partition 5 on a local SSD on each
node.
After moment, everything is happy:
cluster b04e16d1-95d4-4f5f-8b32-318e7abbec56
health HEALTH_OK
monmap e1: 3 mons at
{gnas-1-35a=10.100.1.35:6789/0,gnas-1-35b=10.100.1.85:6789/0,gnas-1-36a=10.100.1.36:6789/0}
election epoch 4, quorum 0,1,2 gnas-1-35a,gnas-1-36a,gnas-1-35b
osdmap e19: 4 osds: 4 up, 4 in
flags sortbitwise
pgmap v39: 64 pgs, 1 pools, 0 bytes data, 0 objects
158 MB used, 171 TB / 171 TB avail
64 active+clean
Now the problem is that when the system probes the devices, ceph automatically
mounts ALL OSDs it sees:
#df
Filesystem 1K-blocks Used Available Use%
Mounted on
/dev/mapper/VG1-root 20834304131317219521132 7% /
devtmpfs13206 0 13206 0% /dev
tmpfs 132023232 0 132023232 0%
/dev/shm
tmpfs 132023232 19040 132004192 1% /run
tmpfs 132023232 0 132023232 0%
/sys/fs/cgroup
/dev/sda2 300780 126376 174404 43% /boot
/dev/sda1 307016 9680 297336 4%
/boot/efi
/dev/mapper/VG1-tmp 16766976 3305216733924 1% /tmp
/dev/mapper/VG1-var 50307072 36319649943876 1% /var
/dev/mapper/VG1-log 50307072 3712050269952 1%
/var/log
/dev/mapper/VG1-auditlog 16766976 3341216733564 1%
/var/log/audit
tmpfs26404648 026404648 0%
/run/user/0
/dev/mapper/mpathb1 46026204140 41592 46026162548 1%
/var/lib/ceph/osd/ceph-0
#partprobe /dev/mapper/mpathc
#partprobe /dev/mapper/mpathd
#partprobe /dev/mapper/mpathe
#df
Filesystem 1K-blocks Used Available Use%
Mounted on
/dev/mapper/VG1-root 20834304131317219521132 7% /
devtmpfs13206 0 13206 0% /dev
tmpfs 132023232 0 132023232 0%
/dev/shm
tmpfs 132023232 19040 132004192 1% /run
tmpfs 132023232 0 132023232 0%
/sys/fs/cgroup
/dev/sda2 300780 126376 174404 43% /boot
/dev/sda1 307016 9680 297336 4%
/boot/efi
/dev/mapper/VG1-tmp 16766976 3305216733924 1% /tmp
/dev/mapper/VG1-var 50307072 36319649943876 1% /var
/dev/mapper/VG1-log 50307072 3712050269952 1%
/var/log
/dev/mapper/VG1-auditlog 16766976 3341216733564 1%
/var/log/audit
tmpfs26404648 026404648 0%
/run/user/0
/dev/mapper/mpathb1 46026204140 41592 46026162548 1%
/var/lib/ceph/osd/ceph-0
/dev/mapper/mpathc1 46026204140 39912 46026164228 1%
/var/lib/ceph/osd/ceph-1
/dev/mapper/mpathd1 46026204140 39992 46026164148 1%
/var/lib/ceph/osd/ceph-2
/dev/mapper/mpathe1 46026204140 39964 46026164176 1%
/var/lib/ceph/osd/ceph-3
Well that causes great grief and lockups...
Is there a way within ceph to tell a particular OSS to ignore OSDs that aren't
meant for it? It's odd to me that a mere partprobe causes the OSD to mount even.
Brian Andrus
ITACS/Research Computing
Naval Postgraduate School
Monterey, California
voice: 831-656-6238
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] failing to respond to cache pressure
Yes, I use the fuse client because the kernel client isn't happy with selinux settings. I have experienced the same symptoms with both clients, however. Yes, the clients that had nothing were merely mounted and nothing, not even an 'ls' was done on the filesystem. I did do 'df' on some of the clients, but all of them ended up with the message. "let it clear" for me was to wait until I saw "HEALTH_OK" When the messages show up, I notice my io write speed line stops showing up when I do 'ceph -s'. I am assuming there is little to no writes going on and see no progress on the rsync command, so I stop it, unmount cephFS and wait. As far as layout, I do have a bit of a uniq setup in that my osds are served via SRP over infiniband from a DDN system. They are also multipathed. I currently only have 4 nodes that I map 4 OSDs to each one. The nodes are pretty beefy and I can (and have) increased the inode_max to (temporarily) alleviate the cache pressure messages. I will be rebuilding the entire filesystem tomorrow with the latest (10.2.1) at which point I will be starting the rsync job again and watching what happens. If there is anything in particular you think I should keep an eye out for, please let me know and I will collect data where I can. Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 -Original Message- From: John Spray [mailto:jsp...@redhat.com] Sent: Monday, May 16, 2016 7:36 AM To: Andrus, Brian Contractor Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] failing to respond to cache pressure On Mon, May 16, 2016 at 3:11 PM, Andrus, Brian Contractor wrote: > Both client and server are Jewel 10.2.0 So the fuse client, correct? If you are up for investigating further, with potential client bugs (or performance issues) it is often useful to compare the fuse vs. kernel clients (using the most recent kernel you can) to work out what's misbehaving. > "All kinds of issues" include that EVERY node ended up with the cache > pressure message, even if they had done no access at all. Hmm, interesting. I wonder if we do have a bug where inactive clients are being "unfairly" asked to clear some cache content but are appearing not to do so because there isn't anything much in their cache. To be clear, when you say "no access at all", you mean a client that was mounted and then just sat there (i.e. not even an ls), right? Are any of the clients holding a lot of files open? Roughly what is the workload doing? > I ended up with some 200 degraded pgs. That's extremely unlikely to be related to CephFS, other than that CephFS will be sending lots of IOs to the OSDs. You should investigate the health of your RADOS cluster (i.e. your OSDs) to work out why you're seeing degraded PGs. > Quite a few with other of the 'standard' errors of suck waiting and such. It might be useful if you just paste your ceph status so that we can see exactly which warnings you're getting. If you're getting "slow OSD request" type messages then that may also be something at the RADOS level that needs investigating. > I ended up disconnecting all mounted clients and waiting about 45 minutes for > it to clear. I couldn't effectively do any writes until I let it clear. When you say "let it clear", do you mean the cluster going completely healthy, or some particular message clearing? What happened when you tried to do writes in the interim? > I am watching my write speeds and while I can get it to peak at a couple > hundred MB/s, it is usually below 10 and often below 1. > That isn't the kind of performance I would expect from a parallel file > system, hence my questioning if it should be used in my environment. Performance is a whole other question. Knowing nothing at all about your disks, servers, network or workload, I have no clue whether you're seeing expected performance or you're seeing the outcome of a bug. John > > > Brian Andrus > ITACS/Research Computing > Naval Postgraduate School > Monterey, California > voice: 831-656-6238 > > > > > -Original Message- > From: John Spray [mailto:jsp...@redhat.com] > Sent: Monday, May 16, 2016 2:28 AM > To: Andrus, Brian Contractor > Cc: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] failing to respond to cache pressure > > On Mon, May 16, 2016 at 5:42 AM, Andrus, Brian Contractor > wrote: >> So this ‘production ready’ CephFS for jewel seems a little not quite…. >> >> >> >> Currently I have a single system mounting CephFS and merely scp-ing >> data to it. >> >> The CephFS mount has 168 TB used, 345 TB / 514 TB avail. >> >> >> >&g
Re: [ceph-users] failing to respond to cache pressure
Both client and server are Jewel 10.2.0 "All kinds of issues" include that EVERY node ended up with the cache pressure message, even if they had done no access at all. I ended up with some 200 degraded pgs. Quite a few with other of the 'standard' errors of suck waiting and such. I ended up disconnecting all mounted clients and waiting about 45 minutes for it to clear. I couldn't effectively do any writes until I let it clear. I am watching my write speeds and while I can get it to peak at a couple hundred MB/s, it is usually below 10 and often below 1. That isn't the kind of performance I would expect from a parallel file system, hence my questioning if it should be used in my environment. Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 -Original Message- From: John Spray [mailto:jsp...@redhat.com] Sent: Monday, May 16, 2016 2:28 AM To: Andrus, Brian Contractor Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] failing to respond to cache pressure On Mon, May 16, 2016 at 5:42 AM, Andrus, Brian Contractor wrote: > So this ‘production ready’ CephFS for jewel seems a little not quite…. > > > > Currently I have a single system mounting CephFS and merely scp-ing > data to it. > > The CephFS mount has 168 TB used, 345 TB / 514 TB avail. > > > > Every so often, I get a HEALTH_WARN message of mds0: Client failing to > respond to cache pressure What client, what version? > Even if I stop the scp, it will not go away until I umount/remount the > filesystem. > > > > For testing, I had the cephfs mounted on about 50 systems and when > updated started on the, I got all kinds of issues with it all. All kinds of issues...? Need more specific bug reports than that to fix things. John > I figured having updated run on a few systems would be a good ‘see > what happens’ if there is a fair amount of access to it. > > > > So, should I not be even considering using CephFS as a large storage > mount for a compute cluster? Is there a sweet spot for what CephFS > would be good for? > > > > > > Brian Andrus > > ITACS/Research Computing > > Naval Postgraduate School > > Monterey, California > > voice: 831-656-6238 > > > > > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] failing to respond to cache pressure
So this 'production ready' CephFS for jewel seems a little not quite Currently I have a single system mounting CephFS and merely scp-ing data to it. The CephFS mount has 168 TB used, 345 TB / 514 TB avail. Every so often, I get a HEALTH_WARN message of mds0: Client failing to respond to cache pressure Even if I stop the scp, it will not go away until I umount/remount the filesystem. For testing, I had the cephfs mounted on about 50 systems and when updated started on the, I got all kinds of issues with it all. I figured having updated run on a few systems would be a good 'see what happens' if there is a fair amount of access to it. So, should I not be even considering using CephFS as a large storage mount for a compute cluster? Is there a sweet spot for what CephFS would be good for? Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Mount Jewel CephFS to CentOS6
So I see that support for RHEL6 and derivatives was dropped in Jewel (http://ceph.com/releases/v10-2-0-jewel-released/) But is there backward compatibility to mount it using hammer on a node? Doesn't seem to be and that makes some sense, but how can I mount CephFS from a CentOS7-Jewel server to a CentOS6 box? Thanks in advance for any advice, Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Adding an OSD to existing Ceph using ceph-deploy
All,
I am trying to add another OSD to our cluster using ceph-deploy. This is
running Jewel.
I previously set up the other 12 OSDs on a fresh install using the command:
ceph-deploy osd create :/dev/mapper/mpath:/dev/sda
Those are all up and happy. On the systems /dev/sda is an SSD which I have
created partitions on for journals.
It seems to prepare everything fine (ceph-deploy osd prepare
ceph-1-35a:/dev/mapper/mpathn:/dev/sda8), but when it comes time to activate, I
am getting a Traceback:
[2016-05-10 11:27:58,195][ceph_deploy.osd][INFO ] Distro info: CentOS Linux
7.2.1511 Core
[2016-05-10 11:27:58,195][ceph_deploy.osd][DEBUG ] activating host ceph-1-35a
disk /dev/mapper/mpathn
[2016-05-10 11:27:58,195][ceph_deploy.osd][DEBUG ] will use init type: systemd
[2016-05-10 11:27:58,196][ceph-1-35a][INFO ] Running command: ceph-disk -v
activate --mark-init systemd --mount /dev/mapper/mpathn
[2016-05-10 11:27:58,315][ceph-1-35a][WARNING] main_activate: path =
/dev/mapper/mpathn
[2016-05-10 11:27:58,315][ceph-1-35a][WARNING] get_dm_uuid: get_dm_uuid
/dev/mapper/mpathn uuid path is /sys/dev/block/253:8/dm/uuid
[2016-05-10 11:27:58,316][ceph-1-35a][WARNING] get_dm_uuid: get_dm_uuid
/dev/mapper/mpathn uuid is mpath-360001ff09070e00c8921000c
[2016-05-10 11:27:58,316][ceph-1-35a][WARNING]
[2016-05-10 11:27:58,316][ceph-1-35a][WARNING] get_dm_uuid: get_dm_uuid
/dev/mapper/mpathn uuid path is /sys/dev/block/253:8/dm/uuid
[2016-05-10 11:27:58,316][ceph-1-35a][WARNING] get_dm_uuid: get_dm_uuid
/dev/mapper/mpathn uuid is mpath-360001ff09070e00c8921000c
[2016-05-10 11:27:58,316][ceph-1-35a][WARNING]
[2016-05-10 11:27:58,316][ceph-1-35a][WARNING] command: Running command:
/sbin/blkid -p -s TYPE -o value -- /dev/mapper/mpathn
[2016-05-10 11:27:58,316][ceph-1-35a][WARNING] Traceback (most recent call
last):
[2016-05-10 11:27:58,316][ceph-1-35a][WARNING] File "/usr/sbin/ceph-disk",
line 9, in
[2016-05-10 11:27:58,316][ceph-1-35a][WARNING]
load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
[2016-05-10 11:27:58,316][ceph-1-35a][WARNING] File
"/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4964, in run
[2016-05-10 11:27:58,316][ceph-1-35a][WARNING] main(sys.argv[1:])
[2016-05-10 11:27:58,317][ceph-1-35a][WARNING] File
"/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4915, in main
[2016-05-10 11:27:58,317][ceph-1-35a][WARNING] args.func(args)
[2016-05-10 11:27:58,317][ceph-1-35a][WARNING] File
"/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 3269, in
main_activate
[2016-05-10 11:27:58,317][ceph-1-35a][WARNING] reactivate=args.reactivate,
[2016-05-10 11:27:58,317][ceph-1-35a][WARNING] File
"/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2979, in
mount_activate
[2016-05-10 11:27:58,317][ceph-1-35a][WARNING] e,
[2016-05-10 11:27:58,317][ceph-1-35a][WARNING]
ceph_disk.main.FilesystemTypeError: Cannot discover filesystem type: device
/dev/mapper/mpathn: Line is truncated:
[2016-05-10 11:27:58,318][ceph-1-35a][ERROR ] RuntimeError: command returned
non-zero exit status: 1
[2016-05-10 11:27:58,318][ceph_deploy][ERROR ] RuntimeError: Failed to execute
command: ceph-disk -v activate --mark-init systemd --mount /dev/mapper/mpathn
This seems to be due to the command:
/sbin/blkid -p -s TYPE -o value -- /dev/mapper/mpathn
is being run instead of:
/sbin/blkid -p -s TYPE -o value -- /dev/mapper/mpathn1
Anyone have ideas on how to get these happy?
Brian Andrus
ITACS/Research Computing
Naval Postgraduate School
Monterey, California
voice: 831-656-6238
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] jewel, cephfs and selinux
All, I thought there was a way to mount CephFS using the kernel driver and be able to honor selinux labeling. Right now, if I do 'ls -lZ' on a mounted cephfs, I get question marks instead of any contexts. When I mount it, I see in dmesg: [858946.554719] SELinux: initialized (dev ceph, type ceph), not configured for labeling Is this something that is in the works and will be available to test? Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Troubleshoot blocked OSDs
Load on all nodes is 1.04 to 1.07 I am updating now to Jewel 10.2 (from 9.2) This is CephFS with SSD journals. Hopefully the update to jewel fixes lots. Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 From: Lincoln Bryant [mailto:linco...@uchicago.edu] Sent: Thursday, April 28, 2016 12:56 PM To: Andrus, Brian Contractor Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Troubleshoot blocked OSDs OK, a few more questions. What does the load look like on the OSDs with ‘iostat’ during the rsync? What version of Ceph? Are you using RBD, CephFS, something else? SSD journals or no? —Lincoln On Apr 28, 2016, at 2:53 PM, Andrus, Brian Contractor mailto:bdand...@nps.edu>> wrote: Lincoln, That was the odd thing to me. Ceph health detail listed all 4 OSDs, so I checked all the systems. I have since let it settle until it is OK again and started. Within a couple minutes, it started showing blocked requests and they are indeed on all 4 OSDs. Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 From: Lincoln Bryant [mailto:linco...@uchicago.edu] Sent: Thursday, April 28, 2016 12:31 PM To: Andrus, Brian Contractor Cc: ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> Subject: Re: [ceph-users] Troubleshoot blocked OSDs Hi Brian, The first thing you can do is “ceph health detail”, which should give you some more information about which OSD(s) have blocked requests. If it’s isolated to one OSD in particular, perhaps use iostat to check utilization and/or smartctl to check health. —Lincoln On Apr 28, 2016, at 2:26 PM, Andrus, Brian Contractor mailto:bdand...@nps.edu>> wrote: All, I have a small ceph cluster with 4 OSDs and 3 MONs on 4 systems. I was rsyncing about 50TB of files and things get very slow. To the point I stopped the rsync, but even with everything stopped, I see: health HEALTH_WARN 80 requests are blocked > 32 sec The number was as high as 218, but they seem to be draining down. I see no issues on any of the systems, CPU load is low, memory usage is low. How do I go about finding why a request is blocked for so long? These have been hitting >500 seconds for block time. Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 ___ ceph-users mailing list ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Troubleshoot blocked OSDs
Lincoln, That was the odd thing to me. Ceph health detail listed all 4 OSDs, so I checked all the systems. I have since let it settle until it is OK again and started. Within a couple minutes, it started showing blocked requests and they are indeed on all 4 OSDs. Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 From: Lincoln Bryant [mailto:linco...@uchicago.edu] Sent: Thursday, April 28, 2016 12:31 PM To: Andrus, Brian Contractor Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Troubleshoot blocked OSDs Hi Brian, The first thing you can do is “ceph health detail”, which should give you some more information about which OSD(s) have blocked requests. If it’s isolated to one OSD in particular, perhaps use iostat to check utilization and/or smartctl to check health. —Lincoln On Apr 28, 2016, at 2:26 PM, Andrus, Brian Contractor mailto:bdand...@nps.edu>> wrote: All, I have a small ceph cluster with 4 OSDs and 3 MONs on 4 systems. I was rsyncing about 50TB of files and things get very slow. To the point I stopped the rsync, but even with everything stopped, I see: health HEALTH_WARN 80 requests are blocked > 32 sec The number was as high as 218, but they seem to be draining down. I see no issues on any of the systems, CPU load is low, memory usage is low. How do I go about finding why a request is blocked for so long? These have been hitting >500 seconds for block time. Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 ___ ceph-users mailing list ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Troubleshoot blocked OSDs
All, I have a small ceph cluster with 4 OSDs and 3 MONs on 4 systems. I was rsyncing about 50TB of files and things get very slow. To the point I stopped the rsync, but even with everything stopped, I see: health HEALTH_WARN 80 requests are blocked > 32 sec The number was as high as 218, but they seem to be draining down. I see no issues on any of the systems, CPU load is low, memory usage is low. How do I go about finding why a request is blocked for so long? These have been hitting >500 seconds for block time. Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Multiple MDSes
Ah. Thanks for the info. I just need to know how to interpret the output!
Brian Andrus
ITACS/Research Computing
Naval Postgraduate School
Monterey, California
voice: 831-656-6238
-Original Message-
From: Eric Eastman [mailto:eric.east...@keepertech.com]
Sent: Friday, April 22, 2016 9:36 PM
To: Andrus, Brian Contractor
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Multiple MDSes
On Fri, Apr 22, 2016 at 9:59 PM, Andrus, Brian Contractor
wrote:
> All,
>
> Ok, I understand Jewel is considered stable for CephFS with a single
> active MDS.
>
> But, how do I add a standby MDS? What documentation I find is a bit
> confusing.
>
> I ran
>
> ceph-deploy create mds systemA
> ceph-deploy create mds systemB
>
> Then I create a ceph filesystem, but it appears systemB is the active
> and only mds:
>
> e6: 1/1/1 up {1:0=systemB=up:active}, 1 up:standby
>
> Is there something to do to get systemA up and standby?
Your output: "1 up:standby" shows that you have 1 standby MDS. On my system
with 3 MDS, running Jewel, the output is:
fsmap e12: 1/1/1 up {1:0=ede-c1-mds01=up:active}, 2 up:standby
You can prove this by shutting down your systemB and seeing that you can still
access your Ceph file system. By default, if you create multiple MDS, you get 1
active MDS and the rest are standby.
Eric
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Multiple MDSes
All,
Ok, I understand Jewel is considered stable for CephFS with a single active MDS.
But, how do I add a standby MDS? What documentation I find is a bit confusing.
I ran
ceph-deploy create mds systemA
ceph-deploy create mds systemB
Then I create a ceph filesystem, but it appears systemB is the active and only
mds:
e6: 1/1/1 up {1:0=systemB=up:active}, 1 up:standby
Is there something to do to get systemA up and standby?
Brian Andrus
ITACS/Research Computing
Naval Postgraduate School
Monterey, California
voice: 831-656-6238
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Status of CephFS
I have implemented a 171TB CephFS using Infernalis recently (it is set so I can grow that to just under 2PB). I tried using Jewel, but it had grief, so I will wait on that. I am migrating data from a lustre filesystem and so far it seems ok. I have not put it into production yet, but will be testing/playing with it one I have the 50TB of data copied over to play with. That migration is slower than I prefer, but so far so good. Brian Andrus -Original Message- From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Vincenzo Pii Sent: Wednesday, April 13, 2016 2:06 AM To: Christian Balzer Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Status of CephFS > On 13 Apr 2016, at 10:55, Christian Balzer wrote: > > On Wed, 13 Apr 2016 11:51:08 +0300 Oleksandr Natalenko wrote: > >> 13.04.2016 11:31, Vincenzo Pii wrote: >>> The setup would include five nodes, two monitors and three OSDs, so >>> data would be redundant (we would add the MDS for CephFS, of course). >> >> You need uneven number of mons. In your case I would setup mons on >> all 5 nodes, or at least on 3 of them. >> > What Oleksandr said. > And in your case MONs can be easily co-located with MDS unless they're > hopelessly underpowered. > > See the VERY recent thread "Thoughts on proposed hardware configuration" > for some more thoughts. > > As for CephFS, I think fsck is upcoming in Jewel, but don't quote me > on that, use google and the Ceph Release page. > > Christian > -- > Christian BalzerNetwork/Systems Engineer > ch...@gol.com Global OnLine Japan/Rakuten Communications > http://www.gol.com/ Hi Guys, Thanks for the tips, I checked the topic that you mentioned, but at the moment I would really need to understand the implications of using CephFS today (Infernalis) and what can go wrong. Any direct experience with CephFS? Thanks for the help! Vincenzo. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] ceph_assert_fail after upgrade from hammer to infernalis
All, I have 4 nodes each with 5 OSDs. I recently upgraded to infernalis via ceph-deploy. It went mostly ok but one of my nodes cannot mount any OSDs. When I look at the status of the service, I see: Apr 07 12:22:06 borg02 ceph-osd[3868]: 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x27a) [0x7f0086ef02aa] Apr 07 12:22:06 borg02 ceph-osd[3868]: 10: (OSDService::get_map(unsigned int)+0x3d) [0x7f008699cecd] Apr 07 12:22:06 borg02 ceph-osd[3868]: 11: (OSD::init()+0xe12) [0x7f0086951682] Apr 07 12:22:06 borg02 ceph-osd[3868]: 12: (main()+0x2998) [0x7f00868d41c8] Apr 07 12:22:06 borg02 ceph-osd[3868]: 13: (__libc_start_main()+0xf5) [0x7f0083756b15] Apr 07 12:22:06 borg02 ceph-osd[3868]: 14: (()+0x2f0959) [0x7f0086904959] Apr 07 12:22:06 borg02 ceph-osd[3868]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. Apr 07 12:22:06 borg02 systemd[1]: ceph-osd@1.service: main process exited, code=killed, status=6/ABRT Apr 07 12:22:06 borg02 systemd[1]: Unit ceph-osd@1.service entered failed state. Apr 07 12:22:06 borg02 systemd[1]: ceph-osd@1.service failed. I see a mention of a bug but it was closed with nothing added (http://tracker.ceph.com/issues/14021) Anyone have any ideas on this? I cannot seem to get these OSDs up at all. That node is also a monitor, which seems to be fine. Brian Andrus ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] About Ceph
Mohd, IIRC, disk prepare does not activate it, osd prepare does. Brian Andrus From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of zai...@nocser.net Sent: Monday, April 04, 2016 7:58 PM To: ceph-users@lists.ceph.com Subject: [ceph-users] About Ceph Hi, What is different between ceph-deploy disk prepare HOST:DISK and ceph-deploy osd prepare HOST:DISK ? Regards, Mohd Zainal Abidin Rabani Technical Support ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Jewel monitors not starting after reboot
All, So I am testing an install of jewel using ceph-deploy. I do a fresh install of CentOS7 and then install ceph and create 3 monitors. I then reboot one of them to see how things behave. It seems that the monitor daemon is not starting on boot. It is enabled but I have to manually start it before I get my full quorum back. Is this something that has been seen before? I am just starting to dig into why it is failing, but thought I would throw this out here as well. Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Using device mapper with journal on separate partition
All, I am trying to use ceph-deploy to create an OSD on a multipath device but put the journal partition on the SSD the system boots from. I have created a partition on the SSD (/dev/sda5) but ceph-deploy does not seem to like it. I am trying: ceph-deploy osd create ceph01:/dev/mapper/mpathb:/dev/sda5 It runs through and ends with: [INFO ] Running command: ceph --cluster=ceph osd stat --format=json [WARNIN] there is 1 OSD down [WARNIN] there is 1 OSD out So I try to activate it: ceph-deploy disk activate ceph-1-35a:/dev/mapper/mpathb1:/dev/sda5 But that gives me an error: [WARNIN] 2016-04-01 13:07:55.519488 7fecdd044800 -1 ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.VTxcYn: (13) Permission denied It works fine if I only specify the single multipath device. It works fine if I point to a disk for the journal, but not a partition. It seems ceph-deploy is not set up to use an existing partition on a separate disk for a journal? Is there a way to get ceph-deploy to use an existing partition for a journal? Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] LUG 2016
Does anyone know if there will be any representation of ceph at the Lustre Users' Group in Portland this year? If not, is there any event in the US that brings the ceph community together? Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Multipath devices with infernalis [solved]
All,
Ok, it was indeed me.
Firewalld does not seem happy across boots when network manager is involved
unless you use something like nm-connection-editor to put the nic in the zone
you want... grrr
Brian Andrus
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
Andrus, Brian Contractor
Sent: Thursday, February 11, 2016 2:36 PM
To: ceph-users@lists.ceph.com
Subject: [ceph-users] Multipath devices with infernalis
All,
I have a set of hardware with a few systems connected via IB along with a DDN
SFA12K.
There are 4 IB/SRP paths to each block device. Those show up as
/dev/mapper/mpath[b-d]
I am trying to do an initial install/setup of ceph on 3 nodes. Each will be a
monitor as well as host a single OSD.
I am using the ceph-deploy to do most of the heavy lifting (using CentOS
7.2.1511).
Everything is quite successful installing monitors and even the first OSD.
ceph status shows:
cluster 0d9e68e4-176d-4229-866b-d408f8055e5b
health HEALTH_OK
monmap e1: 3 mons at
{ceph-1-35a=10.100.1.35:6789/0,ceph-1-35b=10.100.1.85:6789/0,ceph-1-36a=10.100.1.36:6789/0}
election epoch 8, quorum 0,1,2 ceph-1-35a,ceph-1-36a,ceph-1-35b
osdmap e5: 1 osds: 1 up, 1 in
flags sortbitwise
pgmap v8: 64 pgs, 1 pools, 0 bytes data, 0 objects
40112 kB used, 43888 GB / 43889 GB avail
64 active+clean
But as soon as I try to add the next OSD on the next system using
ceph-deploy osd create ceph-1-35b:/dev/mapper/mpathc
things start acting up.
The last bit from the output seems ok:
[ceph-1-35b][INFO ] checking OSD status...
[ceph-1-35b][INFO ] Running command: ceph --cluster=ceph osd stat --format=json
[ceph-1-35b][WARNIN] there is 1 OSD down
[ceph-1-35b][WARNIN] there is 1 OSD out
[ceph_deploy.osd][DEBUG ] Host ceph-1-35b is now ready for osd use.
But, ceph status is now:
cluster 0d9e68e4-176d-4229-866b-d408f8055e5b
health HEALTH_OK
monmap e1: 3 mons at
{ceph-1-35a=10.100.1.35:6789/0,ceph-1-35b=10.100.1.85:6789/0,ceph-1-36a=10.100.1.36:6789/0}
election epoch 8, quorum 0,1,2 ceph-1-35a,ceph-1-36a,ceph-1-35b
osdmap e6: 2 osds: 1 up, 1 in
flags sortbitwise
pgmap v10: 64 pgs, 1 pools, 0 bytes data, 0 objects
40120 kB used, 43888 GB / 43889 GB avail
64 active+clean
And ceph osd tree:
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 42.86040 root default
-2 42.86040 host ceph-1-35a
0 42.86040 osd.0up 1.0 1.0
10 osd.1 down0 1.0
I don't understand why ceph-deploy didn't activate this one when it did for the
first. The OSD is not mounted on the other box.
I can try to activate the down OSD (ceph-deploy disk activate
ceph-1-35b:/dev/mapper/mpathc1:/dev/mapper/mpathc2)
Things look good for a bit:
cluster 0d9e68e4-176d-4229-866b-d408f8055e5b
health HEALTH_OK
monmap e1: 3 mons at
{ceph-1-35a=10.100.1.35:6789/0,ceph-1-35b=10.100.1.85:6789/0,ceph-1-36a=10.100.1.36:6789/0}
election epoch 8, quorum 0,1,2 ceph-1-35a,ceph-1-36a,ceph-1-35b
osdmap e8: 2 osds: 2 up, 2 in
flags sortbitwise
pgmap v14: 64 pgs, 1 pools, 0 bytes data, 0 objects
74804 kB used, 8 GB / 87778 GB avail
64 active+clean
But after about 1 minute, it goes down:
ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 85.72079 root default
-2 42.86040 host ceph-1-35a
0 42.86040 osd.0up 1.0 1.0
-3 42.86040 host ceph-1-35b
1 42.86040 osd.1 down 1.0 1.0
ceph status
cluster 0d9e68e4-176d-4229-866b-d408f8055e5b
health HEALTH_WARN
1/2 in osds are down
monmap e1: 3 mons at
{ceph-1-35a=10.100.1.35:6789/0,ceph-1-35b=10.100.1.85:6789/0,ceph-1-36a=10.100.1.36:6789/0}
election epoch 8, quorum 0,1,2 ceph-1-35a,ceph-1-36a,ceph-1-35b
osdmap e9: 2 osds: 1 up, 2 in
flags sortbitwise
pgmap v15: 64 pgs, 1 pools, 0 bytes data, 0 objects
74804 kB used, 8 GB / 87778 GB avail
64 active+clean
Has anyone played with getting multipath devices to work?
Of course it could be something completely different and I need to step back
and see what step is failing. Any insight into where to dig would be
appreciated.
Thanks in advance,
Brian Andrus
ITACS/Research Computing
Naval Postgraduate School
Monterey, California
voice: 831-656-6238
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Multipath devices with infernalis
All,
I have a set of hardware with a few systems connected via IB along with a DDN
SFA12K.
There are 4 IB/SRP paths to each block device. Those show up as
/dev/mapper/mpath[b-d]
I am trying to do an initial install/setup of ceph on 3 nodes. Each will be a
monitor as well as host a single OSD.
I am using the ceph-deploy to do most of the heavy lifting (using CentOS
7.2.1511).
Everything is quite successful installing monitors and even the first OSD.
ceph status shows:
cluster 0d9e68e4-176d-4229-866b-d408f8055e5b
health HEALTH_OK
monmap e1: 3 mons at
{ceph-1-35a=10.100.1.35:6789/0,ceph-1-35b=10.100.1.85:6789/0,ceph-1-36a=10.100.1.36:6789/0}
election epoch 8, quorum 0,1,2 ceph-1-35a,ceph-1-36a,ceph-1-35b
osdmap e5: 1 osds: 1 up, 1 in
flags sortbitwise
pgmap v8: 64 pgs, 1 pools, 0 bytes data, 0 objects
40112 kB used, 43888 GB / 43889 GB avail
64 active+clean
But as soon as I try to add the next OSD on the next system using
ceph-deploy osd create ceph-1-35b:/dev/mapper/mpathc
things start acting up.
The last bit from the output seems ok:
[ceph-1-35b][INFO ] checking OSD status...
[ceph-1-35b][INFO ] Running command: ceph --cluster=ceph osd stat --format=json
[ceph-1-35b][WARNIN] there is 1 OSD down
[ceph-1-35b][WARNIN] there is 1 OSD out
[ceph_deploy.osd][DEBUG ] Host ceph-1-35b is now ready for osd use.
But, ceph status is now:
cluster 0d9e68e4-176d-4229-866b-d408f8055e5b
health HEALTH_OK
monmap e1: 3 mons at
{ceph-1-35a=10.100.1.35:6789/0,ceph-1-35b=10.100.1.85:6789/0,ceph-1-36a=10.100.1.36:6789/0}
election epoch 8, quorum 0,1,2 ceph-1-35a,ceph-1-36a,ceph-1-35b
osdmap e6: 2 osds: 1 up, 1 in
flags sortbitwise
pgmap v10: 64 pgs, 1 pools, 0 bytes data, 0 objects
40120 kB used, 43888 GB / 43889 GB avail
64 active+clean
And ceph osd tree:
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 42.86040 root default
-2 42.86040 host ceph-1-35a
0 42.86040 osd.0up 1.0 1.0
10 osd.1 down0 1.0
I don't understand why ceph-deploy didn't activate this one when it did for the
first. The OSD is not mounted on the other box.
I can try to activate the down OSD (ceph-deploy disk activate
ceph-1-35b:/dev/mapper/mpathc1:/dev/mapper/mpathc2)
Things look good for a bit:
cluster 0d9e68e4-176d-4229-866b-d408f8055e5b
health HEALTH_OK
monmap e1: 3 mons at
{ceph-1-35a=10.100.1.35:6789/0,ceph-1-35b=10.100.1.85:6789/0,ceph-1-36a=10.100.1.36:6789/0}
election epoch 8, quorum 0,1,2 ceph-1-35a,ceph-1-36a,ceph-1-35b
osdmap e8: 2 osds: 2 up, 2 in
flags sortbitwise
pgmap v14: 64 pgs, 1 pools, 0 bytes data, 0 objects
74804 kB used, 8 GB / 87778 GB avail
64 active+clean
But after about 1 minute, it goes down:
ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 85.72079 root default
-2 42.86040 host ceph-1-35a
0 42.86040 osd.0up 1.0 1.0
-3 42.86040 host ceph-1-35b
1 42.86040 osd.1 down 1.0 1.0
ceph status
cluster 0d9e68e4-176d-4229-866b-d408f8055e5b
health HEALTH_WARN
1/2 in osds are down
monmap e1: 3 mons at
{ceph-1-35a=10.100.1.35:6789/0,ceph-1-35b=10.100.1.85:6789/0,ceph-1-36a=10.100.1.36:6789/0}
election epoch 8, quorum 0,1,2 ceph-1-35a,ceph-1-36a,ceph-1-35b
osdmap e9: 2 osds: 1 up, 2 in
flags sortbitwise
pgmap v15: 64 pgs, 1 pools, 0 bytes data, 0 objects
74804 kB used, 8 GB / 87778 GB avail
64 active+clean
Has anyone played with getting multipath devices to work?
Of course it could be something completely different and I need to step back
and see what step is failing. Any insight into where to dig would be
appreciated.
Thanks in advance,
Brian Andrus
ITACS/Research Computing
Naval Postgraduate School
Monterey, California
voice: 831-656-6238
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
