So I did the exact same thing using Kraken and the same set of VMs, no
issue. What is the magic to make it work in Luminous? Anyone lucky enough
to have this RGW ElasticSearch working using Luminous?
On Mon, Jan 8, 2018 at 10:26 AM, Youzhong Yang <youzh...@gmail.com> wrote:
> Hi Yehuda,
>
> Thanks for replying.
>
> >radosgw failed to connect to your ceph cluster. Does the rados command
> >with the same connection params work?
>
> I am not quite sure what to do by running rados command to test.
>
> So I tried again, could you please take a look and check what could have
> gone wrong?
>
> Here are what I did:
>
> **** On ceph admin node, I removed installation on ceph-rgw1 and
> ceph-rgw2, reinstalled rgw on ceph-rgw1, stoped rgw service, removed all
> rgw pools. Elasticsearch is running on ceph-rgw2 node on port 9200.
>
> *ceph-deploy purge ceph-rgw1*
> *ceph-deploy purge ceph-rgw2*
> *ceph-deploy purgedata ceph-rgw2*
> *ceph-deploy purgedata ceph-rgw1*
> *ceph-deploy install --release luminous ceph-rgw1*
> *ceph-deploy admin ceph-rgw1*
> *ceph-deploy rgw create ceph-rgw1*
> *ssh ceph-rgw1 sudo systemctl stop ceph-rado...@rgw.ceph-rgw1*
> *rados rmpool default.rgw.log default.rgw.log
> --yes-i-really-really-mean-it*
> *rados rmpool default.rgw.meta default.rgw.meta
> --yes-i-really-really-mean-it*
> *rados rmpool default.rgw.control default.rgw.control
> --yes-i-really-really-mean-it*
> *rados rmpool .rgw.root .rgw.root --yes-i-really-really-mean-it*
>
> **** On ceph-rgw1 node:
>
> *export RGWHOST="ceph-rgw1"*
> *export ELASTICHOST="ceph-rgw2"*
> *export REALM="demo"*
> *export ZONEGRP="zone1"*
> *export ZONE1="zone1-a"*
> *export ZONE2="zone1-b"*
> *export SYNC_AKEY="$( cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 20 |
> head -n 1 )"*
> *export SYNC_SKEY="$( cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 40 |
> head -n 1 )"*
>
>
> *radosgw-admin realm create --rgw-realm=${REALM} --default*
> *radosgw-admin zonegroup create --rgw-realm=${REALM}
> --rgw-zonegroup=${ZONEGRP} --endpoints=http://${RGWHOST}:8000 --master
> --default*
> *radosgw-admin zone create --rgw-realm=${REALM} --rgw-zonegroup=${ZONEGRP}
> --rgw-zone=${ZONE1} --endpoints=http://${RGWHOST}:8000
> --access-key=${SYNC_AKEY} --secret=${SYNC_SKEY} --master --default*
> *radosgw-admin user create --uid=sync --display-name="zone sync"
> --access-key=${SYNC_AKEY} --secret=${SYNC_SKEY} --system*
> *radosgw-admin period update --commit*
>
> *sudo systemctl start ceph-radosgw@rgw.${RGWHOST}*
>
> *radosgw-admin zone create --rgw-realm=${REALM} --rgw-zonegroup=${ZONEGRP}
> --rgw-zone=${ZONE2} --access-key=${SYNC_AKEY} --secret=${SYNC_SKEY}
> --endpoints=http://${RGWHOST}:8002*
> *radosgw-admin zone modify --rgw-realm=${REALM} --rgw-zonegroup=${ZONEGRP}
> --rgw-zone=${ZONE2} --tier-type=elasticsearch
> --tier-config=endpoint=http://${ELASTICHOST}:9200,num_replicas=1,num_shards=10*
> *radosgw-admin period update --commit*
>
> *sudo systemctl restart ceph-radosgw@rgw.${RGWHOST}*
>
> *sudo radosgw --keyring /etc/ceph/ceph.client.admin.keyring -f
> --rgw-zone=${ZONE2} --rgw-frontends="civetweb port=8002"*
> *2018-01-08 00:21:54.389432 7f0fe9cd2e80 -1 Couldn't init storage provider
> (RADOS)*
>
> **** As you can see, starting rgw on port 8002 failed, but rgw on port
> 8000 was started successfully.
> **** Here are some more info which may be useful for diagnosis:
>
> $ cat /etc/ceph/ceph.conf
> [global]
> fsid = 3e5a32d4-e45e-48dd-a3c5-f6f28fef8edf
> mon_initial_members = ceph-mon1, ceph-osd1, ceph-osd2, ceph-osd3
> mon_host = 172.30.212.226,172.30.212.227,172.30.212.228,172.30.212.250
> auth_cluster_required = cephx
> auth_service_required = cephx
> auth_client_required = cephx
> osd_pool_default_size = 2
> osd_pool_default_min_size = 2
> osd_pool_default_pg_num = 100
> osd_pool_default_pgp_num = 100
> bluestore_compression_algorithm = zlib
> bluestore_compression_mode = force
> rgw_max_put_size = 21474836480
> [osd]
> osd_max_object_size = 1073741824
> [mon]
> mon_allow_pool_delete = true
> [client.rgw.ceph-rgw1]
> host = ceph-rgw1
> rgw frontends = civetweb port=8000
>
> $ wget -O - -q http://ceph-rgw2:9200/
> {
> "name" : "Hippolyta",
> "cluster_name" : "elasticsearch",
> "version" : {
> "number" : "2.3.1",
> "build_hash" : "bd980929010aef404e7cb0843e61d0665269fc39",
> "build_timestamp" : "2016-04-04T12:25:05Z",
> "build_snapshot" : false,
> "lucene_version" : "5.5.0"
> },
> "tagline" : "You Know, for Search"
> }
>
> $ ceph df
> GLOBAL:
> SIZE AVAIL RAW USED %RAW USED
> 719G 705G 14473M 1.96
> POOLS:
> NAME ID USED %USED MAX AVAIL OBJECTS
> .rgw.root 17 6035 0 333G 19
> zone1-a.rgw.control 18 0 0 333G 8
> zone1-a.rgw.meta 19 350 0 333G 2
> zone1-a.rgw.log 20 50 0 333G 176
> zone1-b.rgw.control 21 0 0 333G 8
> zone1-b.rgw.meta 22 0 0 333G 0
>
> $ rados df
> POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND
> DEGRADED RD_OPS RD WR_OPS WR
> .rgw.root 6035 19 0 38 0 0
> 0 817 553k 55 37888
> zone1-a.rgw.control 0 8 0 16 0 0
> 0 0 0 0 0
> zone1-a.rgw.log 50 176 0 352 0 0
> 0 3703 3522k 2451 1024
> zone1-a.rgw.meta 350 2 0 4 0 0
> 0 9 7168 3 2048
> zone1-b.rgw.control 0 8 0 16 0 0
> 0 0 0 0 0
> zone1-b.rgw.meta 0 0 0 0 0 0
> 0 0 0 0 0
>
> total_objects 213
> total_used 14473M
> total_avail 705G
> total_space 719G
>
> $ radosgw-admin zonegroup get
> {
> "id": "80a70f06-3b8f-458a-b2fc-6ea648871891",
> "name": "zone1",
> "api_name": "zone1",
> "is_master": "true",
> "endpoints": [
> "http://ceph-rgw1:8000";
> ],
> "hostnames": [],
> "hostnames_s3website": [],
> "master_zone": "0f265e95-d12b-43af-a60c-3685ee31d267",
> "zones": [
> {
> "id": "0f265e95-d12b-43af-a60c-3685ee31d267",
> "name": "zone1-a",
> "endpoints": [
> "http://ceph-rgw1:8000";
> ],
> "log_meta": "false",
> "log_data": "true",
> "bucket_index_max_shards": 0,
> "read_only": "false",
> "tier_type": "",
> "sync_from_all": "true",
> "sync_from": []
> },
> {
> "id": "37c249e7-5159-4838-b287-a3641df6f1fb",
> "name": "zone1-b",
> "endpoints": [
> "http://ceph-rgw1:8002";
> ],
> "log_meta": "false",
> "log_data": "true",
> "bucket_index_max_shards": 0,
> "read_only": "false",
> "tier_type": "elasticsearch",
> "sync_from_all": "true",
> "sync_from": []
> }
> ],
> "placement_targets": [
> {
> "name": "default-placement",
> "tags": []
> }
> ],
> "default_placement": "default-placement",
> "realm_id": "b9e5c955-a8d9-4a2a-bd0a-9b786269e909"
> }
>
> $ radosgw-admin zone get --rgw-zone=zone1-a
> {
> "id": "0f265e95-d12b-43af-a60c-3685ee31d267",
> "name": "zone1-a",
> "domain_root": "zone1-a.rgw.meta:root",
> "control_pool": "zone1-a.rgw.control",
> "gc_pool": "zone1-a.rgw.log:gc",
> "lc_pool": "zone1-a.rgw.log:lc",
> "log_pool": "zone1-a.rgw.log",
> "intent_log_pool": "zone1-a.rgw.log:intent",
> "usage_log_pool": "zone1-a.rgw.log:usage",
> "reshard_pool": "zone1-a.rgw.log:reshard",
> "user_keys_pool": "zone1-a.rgw.meta:users.keys",
> "user_email_pool": "zone1-a.rgw.meta:users.email",
> "user_swift_pool": "zone1-a.rgw.meta:users.swift",
> "user_uid_pool": "zone1-a.rgw.meta:users.uid",
> "system_key": {
> "access_key": "BoBwc6kUhr5L5GXZSKu0",
> "secret_key": "TPOQcoZf9ZDtb8pRB7UsbhkxPdLRGE0ruSBSBEuJ"
> },
> "placement_pools": [
> {
> "key": "default-placement",
> "val": {
> "index_pool": "zone1-a.rgw.buckets.index",
> "data_pool": "zone1-a.rgw.buckets.data",
> "data_extra_pool": "zone1-a.rgw.buckets.non-ec",
> "index_type": 0,
> "compression": ""
> }
> }
> ],
> "metadata_heap": "",
> "tier_config": [],
> "realm_id": ""
> }
>
> $ radosgw-admin zone get --rgw-zone=zone1-b
> {
> "id": "37c249e7-5159-4838-b287-a3641df6f1fb",
> "name": "zone1-b",
> "domain_root": "zone1-b.rgw.meta:root",
> "control_pool": "zone1-b.rgw.control",
> "gc_pool": "zone1-b.rgw.log:gc",
> "lc_pool": "zone1-b.rgw.log:lc",
> "log_pool": "zone1-b.rgw.log",
> "intent_log_pool": "zone1-b.rgw.log:intent",
> "usage_log_pool": "zone1-b.rgw.log:usage",
> "reshard_pool": "zone1-b.rgw.log:reshard",
> "user_keys_pool": "zone1-b.rgw.meta:users.keys",
> "user_email_pool": "zone1-b.rgw.meta:users.email",
> "user_swift_pool": "zone1-b.rgw.meta:users.swift",
> "user_uid_pool": "zone1-b.rgw.meta:users.uid",
> "system_key": {
> "access_key": "BoBwc6kUhr5L5GXZSKu0",
> "secret_key": "TPOQcoZf9ZDtb8pRB7UsbhkxPdLRGE0ruSBSBEuJ"
> },
> "placement_pools": [
> {
> "key": "default-placement",
> "val": {
> "index_pool": "zone1-b.rgw.buckets.index",
> "data_pool": "zone1-b.rgw.buckets.data",
> "data_extra_pool": "zone1-b.rgw.buckets.non-ec",
> "index_type": 0,
> "compression": ""
> }
> }
> ],
> "metadata_heap": "",
> "tier_config": [
> {
> "key": "endpoint",
> "val": "http://ceph-rgw2:9200";
> },
> {
> "key": "num_replicas",
> "val": "1"
> },
> {
> "key": "num_shards",
> "val": "10"
> }
> ],
> "realm_id": "b9e5c955-a8d9-4a2a-bd0a-9b786269e909"
> }
>
> # radosgw --keyring /etc/ceph/ceph.client.admin.keyring -f
> --rgw-zone=zone1-b --rgw-frontends="civetweb port=8002" --debug-rgw=255
> 2018-01-08 09:54:53.118526 7f681a211e80 -1 Couldn't init storage provider
> (RADOS)
>
> # tail -12 /var/log/ceph/ceph-client.admin.log
> 2018-01-08 09:54:52.914513 7f681a211e80 20 zone zone1-b
> 2018-01-08 09:54:52.919424 7f681a211e80 20 add_watcher() i=0
> 2018-01-08 09:54:52.924267 7f681a211e80 20 add_watcher() i=1
> 2018-01-08 09:54:52.928440 7f681a211e80 20 add_watcher() i=2
> 2018-01-08 09:54:52.933062 7f681a211e80 20 add_watcher() i=3
> 2018-01-08 09:54:52.937396 7f681a211e80 20 add_watcher() i=4
> 2018-01-08 09:54:52.942317 7f681a211e80 20 add_watcher() i=5
> 2018-01-08 09:54:52.946186 7f681a211e80 20 add_watcher() i=6
> 2018-01-08 09:54:52.950883 7f681a211e80 20 add_watcher() i=7
> 2018-01-08 09:54:52.950904 7f681a211e80 2 all 8 watchers are set,
> enabling cache
> 2018-01-08 09:54:52.950930 7f681a211e80 20 generating connection object
> for zone zone1-a id 0f265e95-d12b-43af-a60c-3685ee31d267
> 2018-01-08 09:54:53.118526 7f681a211e80 -1 Couldn't init storage provider
> (RADOS)
>
> Thanks very much,
>
> --Youzhong
>
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
