Hi Mohammed,
Thanks for the hint, I think I remember seeing this when Jewel came out but I
assumed it must be a mistake, or a mere recommendation but not a mandatory
requirement because I always upgraded the OSDs last ones.
Today I upgraded my OSD nodes in the test environment to Jewel and regained
write access to the buckets.
In production we have multiple RGW nodes behind load balancers, so we can
upgrade them one at a time.
If we have to upgrade first all OSD nodes (which takes much longer considering
they are many more) while the old Hammer RGW cannot talk to a Jewel cluster,
then it means one cannot perform a live upgrade of Ceph, which I think breaks
the promise of a large, distributed, always on storage system...
Now I'll have to test what happens with the cinder volumes attached to a Hammer
cluster that's being upgraded to Jewel, and if upgrading the Ceph packages on
the compute nodes to Jewel will require a restart of the VMs or reboot of the
servers.
Thank you again for your help,
George
> On Jan 25, 2017, at 19:10, Mohammed Naser <mna...@vexxhost.com> wrote:
>
> George,
>
> I believe the supported upgrade model is monitors, OSDs, metadata servers and
> object gateways finally.
>
> I would suggest trying to support path, if you’re still having issues *with*
> the correct upgrade sequence, I would look further into it
>
> Thanks
> Mohammed
>
>> On Jan 25, 2017, at 6:24 PM, George Mihaiescu <lmihaie...@gmail.com> wrote:
>>
>>
>> Hi,
>>
>> I need your help with upgrading our cluster from Hammer (last version) to
>> Jewel 10.2.5 without loosing write access to Radosgw.
>>
>> We have a fairly large cluster (4.3 PB raw) mostly used to store large S3
>> objects, and we currently have more than 500 TB of data in the
>> ".rgw.buckets" pool, so I'm very cautious about upgrading it to Jewel.
>> The plan is to upgrade Ceph-mon and Radosgw to 10.2.5, while keeping the OSD
>> nodes on Hammer, then slowly update them as well.
>>
>>
>> I am currently testing the upgrade procedure in a lab environment, but once
>> I update ceph-mon and radosgw to Jewel, I cannot upload files into new or
>> existing buckets anymore, but I can still create new buckets.
>>
>>
>> I read [1], [2], [3] and [4] and even ran the script in [4] as it can be
>> seen below, but still cannot upload new objects.
>>
>> I was hoping that if I wait long enough to update from Hammer to Jewel, most
>> of the big issues will be solved by point releases, but it seems that I'm
>> doing something wrong, probably because of lack of up to date documentation.
>>
>>
>>
>> After the update to Jewel, this is how things look in my test environment.
>>
>> root@ceph-mon1:~# radosgw zonegroup get
>>
>> root@ceph-mon1:~# radosgw-admin period get
>> period init failed: (2) No such file or directory
>> 2017-01-25 10:13:06.941018 7f98f0d13900 0 RGWPeriod::init failed to init
>> realm id : (2) No such file or directory
>>
>> root@ceph-mon1:~# radosgw-admin zonegroup get
>> failed to init zonegroup: (2) No such file or directory
>>
>> root@ceph-mon1:~# ceph --version
>> ceph version 10.2.5 (c461ee19ecbc0c5c330aca20f7392c9a00730367)
>>
>> root@ceph-mon1:~# radosgw-admin realm list
>> {
>> "default_info": "",
>> "realms": []
>> }
>>
>> root@ceph-mon1:~# radosgw-admin period list
>> {
>> "periods": []
>> }
>>
>> root@ceph-mon1:~# radosgw-admin period get
>> period init failed: (2) No such file or directory
>> 2017-01-25 12:26:07.217986 7f97ca82e900 0 RGWPeriod::init failed to init
>> realm id : (2) No such file or directory
>>
>> root@ceph-mon1:~# radosgw-admin zonegroup get --rgw-zonegroup=default
>> {
>> "id": "default",
>> "name": "default",
>> "api_name": "",
>> "is_master": "true",
>> "endpoints": [],
>> "hostnames": [],
>> "hostnames_s3website": [],
>> "master_zone": "default",
>> "zones": [
>> {
>> "id": "default",
>> "name": "default",
>> "endpoints": [],
>> "log_meta": "false",
>> "log_data": "false",
>> "bucket_index_max_shards": 0,
>> "read_only": "false"
>> }
>> ],
>> "placement_targets": [
>> {
>> "name": "default-placement",
>> "tags": []
>> }
>> ],
>> "default_placement": "default-placement",
>> "realm_id": ""
>> }
>>
>> root@ceph-mon1:~# radosgw-admin zone get --zone-id=default
>> {
>> "id": "default",
>> "name": "default",
>> "domain_root": ".rgw",
>> "control_pool": ".rgw.control",
>> "gc_pool": ".rgw.gc",
>> "log_pool": ".log",
>> "intent_log_pool": ".intent-log",
>> "usage_log_pool": ".usage",
>> "user_keys_pool": ".users",
>> "user_email_pool": ".users.email",
>> "user_swift_pool": ".users.swift",
>> "user_uid_pool": ".users.uid",
>> "system_key": {
>> "access_key": "",
>> "secret_key": ""
>> },
>> "placement_pools": [
>> {
>> "key": "default-placement",
>> "val": {
>> "index_pool": ".rgw.buckets.index",
>> "data_pool": ".rgw.buckets",
>> "data_extra_pool": ".rgw.buckets.extra",
>> "index_type": 0
>> }
>> }
>> ],
>> "metadata_heap": ".rgw.meta",
>> "realm_id": ""
>> }
>>
>> root@ceph-mon1:~# rados df
>> pool name KB objects clones degraded
>> unfound rd rd KB wr wr KB
>> .log 0 127 0 0
>> 0 41402 41275 41402 0
>> .rgw 4 14 0 0
>> 0 147 117 35 14
>> .rgw.buckets 11635 4 0 0
>> 0 4 4969 38 11637
>> .rgw.buckets.index 0 56 0 0
>> 0 1871 1815 119 0
>> .rgw.control 0 8 0 0
>> 0 0 0 0 0
>> .rgw.gc 0 32 0 0
>> 0 5214 5182 3519 0
>> .rgw.meta 2 8 0 0
>> 0 0 0 20 8
>> .rgw.root 2 4 0 0
>> 0 72 48 12 8
>> .usage 0 2 0 0
>> 0 87 87 174 0
>> .users.uid 1 4 0 0
>> 0 104 96 44 2
>> rbd 0 0 0 0
>> 0 0 0 0 0
>> total used 432024 259
>> total avail 84622260
>> total space 85054284
>>
>> root@ceph-mon1:~# ceph -s
>> cluster XXX
>> health HEALTH_OK
>> monmap e1: 3 mons at
>> {ceph-mon1=172.25.12.15:6789/0,ceph-mon2=172.25.12.16:6789/0,ceph-mon3=172.25.12.17:6789/0}
>> election epoch 68, quorum 0,1,2 ceph-mon1,ceph-mon2,ceph-mon3
>> osdmap e212: 9 osds: 9 up, 9 in
>> pgmap v8162: 1344 pgs, 11 pools, 11640 kB data, 259 objects
>> 421 MB used, 82638 MB / 83060 MB avail
>> 1344 active+clean
>>
>> root@ceph-mon1:~# /etc/init.d/radosgw stop
>>
>> root@ceph-mon1:~# cat fix_rgw.sh
>> #!/bin/sh
>>
>> set -x
>>
>> RADOSGW_ADMIN=radosgw-admin
>>
>> echo "Exercise initialization code"
>> $RADOSGW_ADMIN user info --uid=foo # exercise init code (???)
>>
>> echo "Get default zonegroup"
>> $RADOSGW_ADMIN zonegroup get --rgw-zonegroup=default | sed 's/"id":.*/"id":
>> "default",/g' | sed 's/"master_zone.*/"master_zone": "default",/g' >
>> default-zg.json
>>
>> echo "Get default zone"
>> $RADOSGW_ADMIN zone get --zone-id=default > default-zone.json
>>
>> echo "Creating realm"
>> $RADOSGW_ADMIN realm create --rgw-realm=myrealm
>>
>> echo "Creating default zonegroup"
>> $RADOSGW_ADMIN zonegroup set --rgw-zonegroup=default < default-zg.json
>>
>> echo "Creating default zone"
>> $RADOSGW_ADMIN zone set --rgw-zone=default < default-zone.json
>>
>> echo "Setting default zonegroup to 'default'"
>> $RADOSGW_ADMIN zonegroup default --rgw-zonegroup=default
>>
>> echo "Setting default zone to 'default'"
>> $RADOSGW_ADMIN zone default --rgw-zone=default
>>
>> root@ceph-mon1:~# chmod +x fix_rgw.sh
>>
>> root@ceph-mon1:~# ./fix_rgw.sh
>> + RADOSGW_ADMIN=radosgw-admin
>> + echo Exercise initialization code
>> Exercise initialization code
>> + radosgw-admin user info --uid=foo
>> could not fetch user info: no user info saved
>> + echo Get default zonegroup
>> Get default zonegroup
>> + sed s/"master_zone.*/"master_zone": "default",/g
>> + sed s/"id":.*/"id": "default",/g
>> + radosgw-admin zonegroup get --rgw-zonegroup=default
>> + echo Get default zone
>> Get default zone
>> + radosgw-admin zone get --zone-id=default
>> + echo Creating realm
>> Creating realm
>> + radosgw-admin realm create --rgw-realm=myrealm
>> {
>> "id": "7c95f6f4-d437-45c0-bc4d-12f17f7ca4ca",
>> "name": "myrealm",
>> "current_period": "a79d06d5-4836-4f2b-ab3d-961d62a00815",
>> "epoch": 1
>> }
>>
>> + echo Creating default zonegroup
>> Creating default zonegroup
>> + radosgw-admin zonegroup set --rgw-zonegroup=default
>> {
>> "id": "default",
>> "name": "default",
>> "api_name": "",
>> "is_master": "true",
>> "endpoints": [],
>> "hostnames": [],
>> "hostnames_s3website": [],
>> "master_zone": "default",
>> "zones": [
>> {
>> "id": "default",
>> "name": "default",
>> "endpoints": [],
>> "log_meta": "false",
>> "log_data": "false",
>> "bucket_index_max_shards": 0,
>> "read_only": "false"
>> }
>> ],
>> "placement_targets": [
>> {
>> "name": "default-placement",
>> "tags": []
>> }
>> ],
>> "default_placement": "default-placement",
>> "realm_id": "7c95f6f4-d437-45c0-bc4d-12f17f7ca4ca"
>> }
>> + echo Creating default zone
>> Creating default zone
>> + radosgw-admin zone set --rgw-zone=default
>> zone id default{
>> "id": "default",
>> "name": "default",
>> "domain_root": ".rgw",
>> "control_pool": ".rgw.control",
>> "gc_pool": ".rgw.gc",
>> "log_pool": ".log",
>> "intent_log_pool": ".intent-log",
>> "usage_log_pool": ".usage",
>> "user_keys_pool": ".users",
>> "user_email_pool": ".users.email",
>> "user_swift_pool": ".users.swift",
>> "user_uid_pool": ".users.uid",
>> "system_key": {
>> "access_key": "",
>> "secret_key": ""
>> },
>> "placement_pools": [
>> {
>> "key": "default-placement",
>> "val": {
>> "index_pool": ".rgw.buckets.index",
>> "data_pool": ".rgw.buckets",
>> "data_extra_pool": ".rgw.buckets.extra",
>> "index_type": 0
>> }
>> }
>> ],
>> "metadata_heap": ".rgw.meta",
>> "realm_id": "7c95f6f4-d437-45c0-bc4d-12f17f7ca4ca"
>> }
>> + echo Setting default zonegroup to 'default'
>> Setting default zonegroup to 'default'
>> + radosgw-admin zonegroup default --rgw-zonegroup=default
>> + echo Setting default zone to 'default'
>> Setting default zone to 'default'
>> + radosgw-admin zone default --rgw-zone=default
>> root@ceph-mon1:~# radosgw-admin zonegroup get
>> {
>> "id": "default",
>> "name": "default",
>> "api_name": "",
>> "is_master": "true",
>> "endpoints": [],
>> "hostnames": [],
>> "hostnames_s3website": [],
>> "master_zone": "default",
>> "zones": [
>> {
>> "id": "default",
>> "name": "default",
>> "endpoints": [],
>> "log_meta": "false",
>> "log_data": "false",
>> "bucket_index_max_shards": 0,
>> "read_only": "false"
>> }
>> ],
>> "placement_targets": [
>> {
>> "name": "default-placement",
>> "tags": []
>> }
>> ],
>> "default_placement": "default-placement",
>> "realm_id": "7c95f6f4-d437-45c0-bc4d-12f17f7ca4ca"
>> }
>>
>>
>>
>> After running the script:
>>
>> root@ceph-mon1:/var/log/ceph# radosgw-admin zonegroup get
>> {
>> "id": "default",
>> "name": "default",
>> "api_name": "",
>> "is_master": "true",
>> "endpoints": [],
>> "hostnames": [],
>> "hostnames_s3website": [],
>> "master_zone": "default",
>> "zones": [
>> {
>> "id": "default",
>> "name": "default",
>> "endpoints": [],
>> "log_meta": "false",
>> "log_data": "false",
>> "bucket_index_max_shards": 0,
>> "read_only": "false"
>> }
>> ],
>> "placement_targets": [
>> {
>> "name": "default-placement",
>> "tags": []
>> }
>> ],
>> "default_placement": "default-placement",
>> "realm_id": "7c95f6f4-d437-45c0-bc4d-12f17f7ca4ca"
>> }
>>
>> root@ceph-mon1:~# radosgw-admin zone get
>> {
>> "id": "default",
>> "name": "default",
>> "domain_root": ".rgw",
>> "control_pool": ".rgw.control",
>> "gc_pool": ".rgw.gc",
>> "log_pool": ".log",
>> "intent_log_pool": ".intent-log",
>> "usage_log_pool": ".usage",
>> "user_keys_pool": ".users",
>> "user_email_pool": ".users.email",
>> "user_swift_pool": ".users.swift",
>> "user_uid_pool": ".users.uid",
>> "system_key": {
>> "access_key": "",
>> "secret_key": ""
>> },
>> "placement_pools": [
>> {
>> "key": "default-placement",
>> "val": {
>> "index_pool": ".rgw.buckets.index",
>> "data_pool": ".rgw.buckets",
>> "data_extra_pool": ".rgw.buckets.extra",
>> "index_type": 0
>> }
>> }
>> ],
>> "metadata_heap": ".rgw.meta",
>> "realm_id": "7c95f6f4-d437-45c0-bc4d-12f17f7ca4ca"
>> }
>>
>> root@ceph-mon1:~# vi /etc/ceph/ceph.conf -> enabled debug for rgw
>>
>> root@ceph-mon1:~# /etc/init.d/radosgw start
>> Starting client.radosgw.ceph-mon1...
>>
>>
>> The log file with debugging enabled for the failed upload request is
>> uploaded at http://pastebin.com/1eLfrazn
>>
>> I would appreciate any help with this as I spent a lot of time trying
>> different things without any progress so far.
>>
>> Thank you,
>> George
>>
>> [1]
>> http://lists.opennebula.org/pipermail/ceph-users-ceph.com/2016-July/011797.html
>> [2] http://robbat2.livejournal.com/242849.html
>> [3] http://www.spinics.net/lists/ceph-users/msg28100.html
>> [4] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-July/011157.html
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
