Re: [ceph-users] Ceph multi site outage question
Thanks - yeah jewel is old 🙂 But i meant to say nautilus and not luminous. The first option probably wont work for me. Since both sides are active and the application1 needs to write in both places as http://application1.something.com. The 2nd one in theory should work. I'm using haproxy and it does have an option to rewrite host headers. I can also replace it with nginx since i think it'll handle this kind of thing better. In such a situation, I'd set one site's radosgw to application1-master and the 2nd one to application1-slave. The reverse proxy will then rewrite application1 to application1-master or application1-slave depending on the site. Thanks From: Ed Fisher Sent: Wednesday, October 9, 2019 11:13 AM To: Melzer Pinto Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Ceph multi site outage question Boy, Jewel is pretty old. Even Luminous is getting up there. There have been a lot of multisite improvements in Mimic and now Nautilus, so you might want to consider upgrading all the way to 14.2.4. Anyway, the way we solve this is by giving each zone a different name (eg application1a/application1b), and then having a virtual IP for application1. We then move the virtual IP around whichever zone we want to have accepting traffic for that zonegroup. In our case we're advertising the virtual IP on all of the radosgw instances using bgp and then letting our routers do per-stream ECMP to load balance the traffic. Each RGW in each cluster checks the realm's period every few seconds and decides to announce/withdraw the IP based on whether that rgw's zone is the master zone for the zonegroup (plus whether the rgw instance is healthy, etc). We have both application1.example.com<http://application1.example.com> and application1a/application1b.example.com<http://application1b.example.com> as hostnames in the zonegroup config, but just application1.example.com<http://application1.example.com> for the endpoint. I'm not sure what the equivalent settings are on Jewel's multisite, if any. If you're routing radosgw traffic through a reverse proxy or load balancer you can also rewrite the host header on the fly. Hope this helps, Ed On Oct 9, 2019, at 10:02 AM, Melzer Pinto mailto:melzer.pi...@mezocliq.com>> wrote: Hello, I have a question about multi site configuration. I have 2 clusters configured in a single realm and zonegroup. One cluster is the master zone and the other the slave. Lets assume the first cluster can be reached at http://application1.something.com<http://application1.something.com/> and the 2nd one is http://application1-slave.something.com<http://application1-slave.something.com/>. My application has a number of config files that reference http://application1.something.com<http://application1.something.com/>. So if there is a site outage i'd need to change all of these files to http://application1-slave.something.com<http://application1-slave.something.com/> and restart. I was wondering if there are any alternatives where I dont have to change the config files. The best solution would be to use the same name in both clusters - http://application1.something.com<http://application1.something.com/>. But i'm not sure if that is recommended or doable even. Any suggestions? I'm using the latest version of Ceph Jewel, 10.2.11, but I am planning to upgrade to luminous soon. Thanks M ___ ceph-users mailing list ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Ceph multi site outage question
Hello, I have a question about multi site configuration. I have 2 clusters configured in a single realm and zonegroup. One cluster is the master zone and the other the slave. Lets assume the first cluster can be reached at http://application1.something.com and the 2nd one is http://application1-slave.something.com. My application has a number of config files that reference http://application1.something.com. So if there is a site outage i'd need to change all of these files to http://application1-slave.something.com and restart. I was wondering if there are any alternatives where I dont have to change the config files. The best solution would be to use the same name in both clusters - http://application1.something.com. But i'm not sure if that is recommended or doable even. Any suggestions? I'm using the latest version of Ceph Jewel, 10.2.11, but I am planning to upgrade to luminous soon. Thanks M ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] radosgw multi file upload failure
Hello, Recently we've observed on one of our ceph clusters that uploading of a large number of small files(~2000x2k) fails. The http return code shows 200 but the file upload fails. Here is an e.g of the log 2018-06-27 07:34:40.624103 7f0dc67cc700 1 == starting new request req=0x7f0dc67c68a0 = 2018-06-27 07:34:40.645039 7f0dc3fc7700 1 == starting new request req=0x7f0dc3fc18a0 = 2018-06-27 07:34:40.682108 7f0dc3fc7700 0 WARNING: couldn't find acl header for object, generating default 2018-06-27 07:34:40.962674 7f0dcbfd7700 0 ERROR: client_io->complete_request() returned -5 2018-06-27 07:34:40.962689 7f0dcbfd7700 1 == req done req=0x7f0dcbfd18a0 op status=0 http_status=200 == 2018-06-27 07:34:40.962738 7f0dcbfd7700 1 civetweb: 0x7f0df4004160: 10.x.x.x. - - [27/Jun/2018:07:34:34 +] "POST - HTTP/1.1" 200 0 - aws-sdk-java/1.6.4 Linux/3.17.6-200.fc20.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.73-b02 I tried tuning the performance using the below parameters but the number of file upload failures still exists so I'm suspecting this is not a concurrency issue. rgw num rados handles = 8 rgw thread pool size = 512 rgw frontends = civetweb port=7480 num_threads=512 I also tried increasing the logging level for rgw and civetweb to 20/5 but I dont see anything that can point to the issue. 2018-06-28 18:00:24.575460 7f3d7dfc3700 20 get_obj_state: s->obj_tag was set empty 2018-06-28 18:00:24.575491 7f3d7dfc3700 20 get_obj_state: rctx=0x7f3d7dfbcff0 obj=files:_multipart_-.error.2~Rh1AqHvzgCPc0NGWMl-FHE0Y-HvCcmk.1 state=0x7f3e04024d88 s->prefetch_data=0 2018-06-28 18:00:24.575496 7f3d7dfc3700 20 prepare_atomic_modification: state is not atomic. state=0x7f3e04024d88 2018-06-28 18:00:24.57 7f3d7dfc3700 20 reading from default.rgw.data.root:.bucket.meta.files:-.6432.11 2018-06-28 18:00:24.575567 7f3d7dfc3700 20 get_system_obj_state: rctx=0x7f3d7dfbb5d0 obj=default.rgw.data.root:.bucket.meta.files:-.6432.11 state=0x7f3e04001228 s->prefetch_data 2018-06-28 18:00:24.575577 7f3d7dfc3700 10 cache get: name=default.rgw.data.root+.bucket.meta.files:-.6432.11 : hit (requested=22, cached=23) 2018-06-28 18:00:24.575586 7f3d7dfc3700 20 get_system_obj_state: s->obj_tag was set empty 2018-06-28 18:00:24.575592 7f3d7dfc3700 10 cache get: name=default.rgw.data.root+.bucket.meta.files:-.6432.11 : hit (requested=17, cached=23) 2018-06-28 18:00:24.575614 7f3d7dfc3700 20 bucket index object: .dir.-.6432.11 2018-06-28 18:00:24.606933 7f3d67796700 2 req 9567:5.505460:s3:POST -.error:init_multipart:completing 2018-06-28 18:00:24.607025 7f3d67796700 0 ERROR: client_io->complete_request() returned -5 2018-06-28 18:00:24.607036 7f3d67796700 2 req 9567:5.505572:s3:POST -.error:init_multipart:op status=0 2018-06-28 18:00:24.607040 7f3d67796700 2 req 9567:5.505578:s3:POST -.error:init_multipart:http status=200 2018-06-28 18:00:24.607046 7f3d67796700 1 == req done req=0x7f3d677908a0 op status=0 http_status=200 == The cluster is a 12 node Ceph jewel(10.2.10-1~bpo80+1) one. Operating system is Debian 8.9 ceph.conf [global] fsid = 314d4121-46b1-4433-9bae-fdd2803fc24b mon_initial_members = ceph-1,ceph-2,ceph-3 mon_host = 10.x.x.x, 10.x.x.x, 10.x.x.x auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx public network = 10.x.x.x osd_journal_size = 10240 osd_mount_options_xfs = rw,noexec,noatime,nodiratime,inode64 osd_pool_default_size = 3 osd_pool_default_min_size = 2 osd_pool_default_pg_num = 900 osd_pool_default_pgp_num = 900 log to syslog = true err to syslog = true clog to syslog = true rgw dns name = xxx.com rgw num rados handles = 8 rgw thread pool size = 512 rgw frontends = civetweb port=7480 num_threads=512 debug rgw = 20/5 debug civetweb = 20/5 [mon] mon cluster log to syslog = true Any idea what the issue could be here? Thanks Mel ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph drives not detected
Hi Federico, Yep I understand that. This is for legacy reasons. We already have 3 older clusters running with a similar setup with minor differences (hardware, etc.) and this one is being setup to test something [:(] Thanks From: Federico Lucifredi Sent: Friday, April 7, 2017 5:05:36 PM To: Melzer Pinto Cc: ceph-us...@ceph.com Subject: Re: [ceph-users] Ceph drives not detected Hi Melzer, Somewhat pointing out the obvious, but just in case: Ceph is in rapid development, and Giant is way behind where the state of the art is. If this is your first Ceph experience, it is definitely recommended you look at Jewel or even Kraken -- In Linux terms, it is almost as if you were running a 2.4 kernel ;-) Good luck --F _ -- "'Problem' is a bleak word for challenge" - Richard Fish (Federico L. Lucifredi) - federico at redhat.com - GnuPG 0x4A73884C On Fri, Apr 7, 2017 at 11:19 AM, Melzer Pinto wrote: > Hello, > > I am setting up a 9 node ceph cluster. For legacy reasons I'm using Ceph > giant (0.87) on Fedora 21. Each OSD node has 4x4TB SATA drives with journals > on a separate SSD. The server is an HP XL190 Gen 9 with latest firmware. > > The issue I'm seeing is that only 2 drives get detected and mounted on > almost all the nodes. During the initial creation all drives were created > and mounted but now only 2 show up. > > Usually a partprobe forces the drives to come online but in this case it > doesnt. > > On a reboot a different set of OSDs will get detected. For e.g. of osds 0 > and 1 are up, on a reboot osds 0 and 3 will be detected. On another reboot > osds 1 and 2 may come up. > > > $ ceph osd tree > > # idweight type name up/down reweight > -1 131 root default > -2 14.56 host xxx-a5-34 > 0 3.64osd.0 up 1 > 1 3.64osd.1 down0 > 2 3.64osd.2 up 1 > 3 3.64osd.3 down0 > -3 14.56 host xxx-a5-36 > 4 3.64osd.4 down0 > 5 3.64osd.5 down0 > 6 3.64osd.6 up 1 > 7 3.64osd.7 up 1 > -4 14.56 host xxx-a5-37 > 8 3.64osd.8 down0 > 9 3.64osd.9 up 1 > 10 3.64osd.10 up 1 > 11 3.64osd.11 down0 > -5 14.56 host xxx-b5-34 > 12 3.64osd.12 up 1 > 13 3.64osd.13 down0 > 14 3.64osd.14 up 1 > 15 3.64osd.15 down0 > -6 14.56 host xxx-b5-36 > 16 3.64osd.16 up 1 > 17 3.64osd.17 up 1 > 18 3.64osd.18 down0 > 19 3.64osd.19 down0 > -7 14.56 host xxx-b5-37 > 20 3.64osd.20 up 1 > 21 3.64osd.21 up 1 > 22 3.64osd.22 down0 > 23 3.64osd.23 down0 > -8 14.56 host xxx-c5-34 > 24 3.64osd.24 up 1 > 25 3.64osd.25 up 1 > 26 3.64osd.26 up 1 > 27 3.64osd.27 up 1 > -9 14.56 host xxx-c5-36 > 28 3.64osd.28 down0 > 29 3.64osd.29 up 1 > 30 3.64osd.30 down0 > 31 3.64osd.31 up 1 > -10 14.56 host xxx-c5-37 > 32 3.64osd.32 up 1 > 33 3.64osd.33 up 1 > 34 3.64osd.34 up 1 > 35 3.64osd.35 up 1 > > Anyone seen this problem before and know what the issue could be? > > Thanks > > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Ceph drives not detected
Hello, I am setting up a 9 node ceph cluster. For legacy reasons I'm using Ceph giant (0.87) on Fedora 21. Each OSD node has 4x4TB SATA drives with journals on a separate SSD. The server is an HP XL190 Gen 9 with latest firmware. The issue I'm seeing is that only 2 drives get detected and mounted on almost all the nodes. During the initial creation all drives were created and mounted but now only 2 show up. Usually a partprobe forces the drives to come online but in this case it doesnt. On a reboot a different set of OSDs will get detected. For e.g. of osds 0 and 1 are up, on a reboot osds 0 and 3 will be detected. On another reboot osds 1 and 2 may come up. $ ceph osd tree # idweight type name up/down reweight -1 131 root default -2 14.56 host xxx-a5-34 0 3.64osd.0 up 1 1 3.64osd.1 down0 2 3.64osd.2 up 1 3 3.64osd.3 down0 -3 14.56 host xxx-a5-36 4 3.64osd.4 down0 5 3.64osd.5 down0 6 3.64osd.6 up 1 7 3.64osd.7 up 1 -4 14.56 host xxx-a5-37 8 3.64osd.8 down0 9 3.64osd.9 up 1 10 3.64osd.10 up 1 11 3.64osd.11 down0 -5 14.56 host xxx-b5-34 12 3.64osd.12 up 1 13 3.64osd.13 down0 14 3.64osd.14 up 1 15 3.64osd.15 down0 -6 14.56 host xxx-b5-36 16 3.64osd.16 up 1 17 3.64osd.17 up 1 18 3.64osd.18 down0 19 3.64osd.19 down0 -7 14.56 host xxx-b5-37 20 3.64osd.20 up 1 21 3.64osd.21 up 1 22 3.64osd.22 down0 23 3.64osd.23 down0 -8 14.56 host xxx-c5-34 24 3.64osd.24 up 1 25 3.64osd.25 up 1 26 3.64osd.26 up 1 27 3.64osd.27 up 1 -9 14.56 host xxx-c5-36 28 3.64osd.28 down0 29 3.64osd.29 up 1 30 3.64osd.30 down0 31 3.64osd.31 up 1 -10 14.56 host xxx-c5-37 32 3.64osd.32 up 1 33 3.64osd.33 up 1 34 3.64osd.34 up 1 35 3.64osd.35 up 1 Anyone seen this problem before and know what the issue could be? Thanks ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
