Re: [ceph-users] Federated gateways

[ghislain.chevalier]

Hi,

We did a PoC at Orange and encountered some difficulties in configurating 
federation.

Can you check that placements  targets are identical on each zone?

brgds

De : ceph-users [mailto:ceph-users-boun...@lists.ceph.com] De la part de 
wd_hw...@wistron.com
Envoyé : vendredi 6 novembre 2015 03:01
À : cle...@centraldesktop.com
Cc : ceph-users@lists.ceph.com
Objet : Re: [ceph-users] Federated gateways

Hi Craig,
I am testing the federated gateway of 1 region with 2 zones. And I found only 
metadata is replicated, the data is NOT.
According to your check list, I am sure all thinks are checked. Could you 
review my configuration scripts? The configuration files are similar to 
http://docs.ceph.com/docs/master/radosgw/federated-config/.

1. For the master zone with 5 nodes of the region
(1) create keyring
  sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring
  sudo chmod +r /etc/ceph/ceph.client.radosgw.keyring
  sudo ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n 
client.radosgw.us-east-1 --gen-key
  sudo ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n 
client.radosgw.us-west-1 --gen-key
  sudo ceph-authtool -n client.radosgw.us-east-1 --cap osd 'allow rwx' --cap 
mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring
  sudo ceph-authtool -n client.radosgw.us-west-1 --cap osd 'allow rwx' --cap 
mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring
  sudo ceph -k /etc/ceph/ceph.client.admin.keyring auth add 
client.radosgw.us-east-1 -i /etc/ceph/ceph.client.radosgw.keyring
  sudo ceph -k /etc/ceph/ceph.client.admin.keyring auth add 
client.radosgw.us-west-1 -i /etc/ceph/ceph.client.radosgw.keyring

(2) modify Ceph cluster configurations and synchronize it
  ceph-deploy --overwrite-conf config push node1 node2 node3 node4 node5

(3) configure Apache
(4) configure region
  sudo apt-get install -y radosgw radosgw-agent python-pip
  sudo radosgw-admin region set --infile /home/ceph/us.json --name 
client.radosgw.us-east-1
  sudo radosgw-admin region set --infile /home/ceph/us.json --name 
client.radosgw.us-west-1
  sudo rados -p .us.rgw.root rm region_info.default
  sudo radosgw-admin region default --rgw-region=us --name 
client.radosgw.us-east-1
  sudo radosgw-admin region default --rgw-region=us --name 
client.radosgw.us-west-1
  sudo radosgw-admin regionmap update --name client.radosgw.us-east-1
  sudo radosgw-admin regionmap update --name client.radosgw.us-west-1

(5) create master zone
  sudo radosgw-admin zone set --rgw-zone=us-east --infile 
/home/ceph/us-east.json --name client.radosgw.us-east-1
  sudo radosgw-admin zone set --rgw-zone=us-east --infile 
/home/ceph/us-east.json --name client.radosgw.us-west-1
  sudo radosgw-admin zone set --rgw-zone=us-west --infile 
/home/ceph/us-west.json --name client.radosgw.us-east-1
  sudo radosgw-admin zone set --rgw-zone=us-west --infile 
/home/ceph/us-west.json --name client.radosgw.us-west-1
  sudo rados -p .rgw.root rm zone_info.default
  sudo radosgw-admin regionmap update --name client.radosgw.us-east-1
  sudo radosgw-admin regionmap update --name client.radosgw.us-west-1

(6) create master zone's users
  sudo radosgw-admin user create --uid="us-east" --display-name="Region-us 
Zone-East" --name client.radosgw.us-east-1 --system --access_key=us_access_key 
--secret=us_secret_key
  sudo radosgw-admin user create --uid="us-west" --display-name="Region-us 
Zone-West" --name client.radosgw.us-west-1 --system --access_key=us_access_key 
--secret=us_secret_key

(7) restart Ceph & apache2 & radosgw services



2. For the secondary zone with 5 nodes of the region
(1) copy the keyring file 'ceph.client.radosgw.keyring' from master zone and 
import the keyring
  sudo ceph -k /etc/ceph/ceph.client.admin.keyring auth add 
client.radosgw.us-east-1 -i /etc/ceph/ceph.client.radosgw.keyring
  sudo ceph -k /etc/ceph/ceph.client.admin.keyring auth add 
client.radosgw.us-west-1 -i /etc/ceph/ceph.client.radosgw.keyring

(2) modify Ceph cluster configurations and synchronize it
ceph-deploy --overwrite-conf config push node1 node2 node3 node4 node5

(3) configure Apache
(4) copy infile '/home/ceph/us.json' for the master zone and create 'us' region
  sudo apt-get install -y radosgw radosgw-agent python-pip
  sudo radosgw-admin region set --infile /home/ceph/us.json --name 
client.radosgw.us-east-1
  sudo radosgw-admin region set --infile /home/ceph/us.json --name 
client.radosgw.us-west-1
  sudo radosgw-admin region default --rgw-region=us --name 
client.radosgw.us-east-1
  sudo radosgw-admin region default --rgw-region=us --name 
client.radosgw.us-west-1
  sudo radosgw-admin regionmap update --name client.radosgw.us-east-1
  sudo radosgw-admin regionmap update --name client.radosgw.us-west-1

(5) create secondary one
  sudo radosgw-admin zone set --rgw-zone=us-east --infile 
/home/ceph/us-east.json --name client.radosgw

[ceph-users] Federated gateways sync error - Too many open files

[WD_Hwang]

Hi,
 I am testing federated gateways asynchronization with one region and two 
zones. After several files synchronized completed, I found some error messages 
was appeared in the log file.
 It told me that 'Too many open files' when connected to secondary zone. I have 
modified the parameters of system files:
(1) at /etc/sysctl.conf add: 
fs.file-max=131072

and then 'sudo sysctl -p'

(2) at /usr/include/linux/limits.h change:
NR_OPEN = 65536

(3) at /etc/security/limits.conf add:
*softnofile  65535
*hardnofile  65535

 But these seem not work. Any help would be much appreciated.
-
Wed, 11 Nov 2015 09:37:26 GMT
/admin/replica_log
2015-11-11 17:37:30,220 14491 [boto][DEBUG ] Signature:
AWS west_access_key:SnkgYfieKNQzQwK9Iwy0B9/YxKs=
2015-11-11 17:37:30,220 14491 [boto][DEBUG ] encountered error exception, 
reconnecting
2015-11-11 17:37:30,220 14491 [boto][DEBUG ] establishing HTTP connection: 
kwargs={'port': 80, 'timeout': 70}
2015-11-11 17:37:35,331 14491 [radosgw_agent.sync][WARNING] error preparing for 
sync, will retry. Traceback:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/radosgw_agent/sync.py", line 30, in 
prepare_sync
  File "/usr/lib/python2.7/dist-packages/radosgw_agent/sync.py", line 213, in 
prepare
  File "/usr/lib/python2.7/dist-packages/radosgw_agent/sync.py", line 179, in 
get_worker_bound
  File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 521, in 
get_worker_bound
  File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 207, in 
request
NetworkError: unable to connect to node1-west.ceph.com [Errno 24] Too many open 
files
2015-11-11 17:37:45,340 14491 [boto][DEBUG ] path=/admin/replica_log?bounds
2015-11-11 17:37:45,340 14491 [boto][DEBUG ] auth_path=/admin/replica_log?bounds
2015-11-11 17:37:45,341 14491 [boto][DEBUG ] StringToSign:
GET
-

Best Regards,
wdhwang

---
This email contains confidential or legally privileged information and is for 
the sole use of its intended recipient. 
Any unauthorized review, use, copying or distribution of this email or the 
content of this email is strictly prohibited.
If you are not the intended recipient, you may reply to the sender and should 
delete this e-mail immediately.
---
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Federated gateways

[WD_Hwang]

Hi, Craig:
  I used 10 VMs for federated gateway testing. There are 5 nodes for us-east, 
and the others are for us-west. The two zones are independent.
  Before the configurations of the region and zone, I have the two zones with 
the same 'client.radosgw.[zone]' setting of ceph.conf.
-
[client.radosgw.us-east-1]
rgw region = us
rgw region root pool = .us.rgw.root
rgw zone = us-east
rgw zone root pool = .us-east.rgw.root
host = node1-east
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw socket path = /tmp/radosgw.us-east.sock
log file = /var/log/ceph/radosgw.us-east.log
rgw dns name = node1-east.ceph.com

[client.radosgw.us-west-1]
rgw region = us
rgw region root pool = .us.rgw.root
rgw zone = us-west
rgw zone root pool = .us-west.rgw.root
host = node1-west
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw socket path = /tmp/radosgw.us-west.sock
log file = /var/log/ceph/radosgw.us-west.log
rgw dns name = node1-west.ceph.com
-

  The needed pools were created by manual. And I also created a normal user in 
the two zones with same access key and secret key. After that, I used 's3cmd' 
to create a bucket as 'BUCKETa' and put 'ceph.conf' in the bucket to test the 
synchronization. There was some error message in the logging file of 
'radosgw-agent'. I found the file of 'ceph.conf' was sent to the secondary zone 
and DELETED later by unknown reason.
-
application/json; charset=UTF-8
Fri, 06 Nov 2015 09:18:01 GMT
x-amz-copy-source:BUCKETa/ceph.conf
/BUCKETa/ceph.conf
2015-11-06 17:18:01,174 4558 [boto][DEBUG ] Signature:
AWS us_access_key:p6+AscqnndOpcWfJMBO7ADDpNek=
2015-11-06 17:18:01,175 4558 [boto][DEBUG ] url = 
'http://node1-west.ceph.com/BUCKETa/ceph.conf'
params={'rgwx-op-id': 'admin1:4463:2', 'rgwx-source-zone': u'us-east', 
'rgwx-client-id': 'radosgw-agent'}
headers={'Content-Length': '0', 'User-Agent': 'Boto/2.20.1 Python/2.7.6 
Linux/3.13.0-66-generic', 'x-amz-copy-source': 'BUCKETa/ceph.conf', 'Date': 
'Fri, 06 Nov 2015 09
:18:01 GMT', 'Content-Type': 'application/json; charset=UTF-8', 
'Authorization': 'AWS us_access_key:p6+AscqnndOpcWfJMBO7ADDpNek='}
data=None
2015-11-06 17:18:01,175 4558 [boto][DEBUG ] Method: PUT
2015-11-06 17:18:01,175 4558 [boto][DEBUG ] Path: 
/BUCKETa/ceph.conf?rgwx-op-id=admin1%3A4463%3A2&rgwx-source-zone=us-east&rgwx-client-id=radosgw-agent
2015-11-06 17:18:01,175 4558 [boto][DEBUG ] Data:
2015-11-06 17:18:01,176 4558 [boto][DEBUG ] Headers: {'Content-Type': 
'application/json; charset=UTF-8', 'x-amz-copy-source': 'BUCKETa/ceph.conf'}
2015-11-06 17:18:01,176 4558 [boto][DEBUG ] Host: node1-west.ceph.com
2015-11-06 17:18:01,176 4558 [boto][DEBUG ] Port: 80
2015-11-06 17:18:01,176 4558 [boto][DEBUG ] Params: {'rgwx-op-id': 
'admin1%3A4463%3A2', 'rgwx-source-zone': 'us-east', 'rgwx-client-id': 
'radosgw-agent'}
2015-11-06 17:18:01,177 4558 [boto][DEBUG ] Token: None
2015-11-06 17:18:01,177 4558 [boto][DEBUG ] StringToSign:
PUT

application/json; charset=UTF-8
Fri, 06 Nov 2015 09:18:01 GMT
x-amz-copy-source:BUCKETa/ceph.conf
/BUCKETa/ceph.conf
2015-11-06 17:18:01,177 4558 [boto][DEBUG ] Signature:
AWS us_access_key:p6+AscqnndOpcWfJMBO7ADDpNek=
2015-11-06 17:18:01,203 4558 [radosgw_agent.worker][DEBUG ] object 
"BUCKETa/ceph.conf" not found on master, deleting from secondary
2015-11-06 17:18:01,203 4558 [boto][DEBUG ] path=/BUCKETa/
2015-11-06 17:18:01,203 4558 [boto][DEBUG ] auth_path=/BUCKETa/
2015-11-06 17:18:01,203 4558 [boto][DEBUG ] path=/BUCKETa/?max-keys=0
2015-11-06 17:18:01,204 4558 [boto][DEBUG ] auth_path=/BUCKETa/?max-keys=0
2015-11-06 17:18:01,204 4558 [boto][DEBUG ] Method: GET
2015-11-06 17:18:01,204 4558 [boto][DEBUG ] Path: /BUCKETa/?max-keys=0
2015-11-06 17:18:01,204 4558 [boto][DEBUG ] Data:
2015-11-06 17:18:01,204 4558 [boto][DEBUG ] Headers: {}
2015-11-06 17:18:01,205 4558 [boto][DEBUG ] Host: node1-west.ceph.com
2015-11-06 17:18:01,205 4558 [boto][DEBUG ] Port: 80
2015-11-06 17:18:01,205 4558 [boto][DEBUG ] Params: {}
2015-11-06 17:18:01,206 4558 [boto][DEBUG ] establishing HTTP connection: 
kwargs={'port': 80, 'timeout': 70}
2015-11-06 17:18:01,206 4558 [boto][DEBUG ] Token: None
2015-11-06 17:18:01,206 4558 [boto][DEBUG ] StringToSign:
-

Any help would be much appreciated.

Best Regards,
Wdhwang


-Original Messa

Re: [ceph-users] Federated gateways

[WD_Hwang]

Hi Craig,
I am testing the federated gateway of 1 region with 2 zones. And I found only 
metadata is replicated, the data is NOT.
According to your check list, I am sure all thinks are checked. Could you 
review my configuration scripts? The configuration files are similar to 
http://docs.ceph.com/docs/master/radosgw/federated-config/.

1. For the master zone with 5 nodes of the region
(1) create keyring
  sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring
  sudo chmod +r /etc/ceph/ceph.client.radosgw.keyring
  sudo ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n 
client.radosgw.us-east-1 --gen-key
  sudo ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n 
client.radosgw.us-west-1 --gen-key
  sudo ceph-authtool -n client.radosgw.us-east-1 --cap osd 'allow rwx' --cap 
mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring
  sudo ceph-authtool -n client.radosgw.us-west-1 --cap osd 'allow rwx' --cap 
mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring
  sudo ceph -k /etc/ceph/ceph.client.admin.keyring auth add 
client.radosgw.us-east-1 -i /etc/ceph/ceph.client.radosgw.keyring
  sudo ceph -k /etc/ceph/ceph.client.admin.keyring auth add 
client.radosgw.us-west-1 -i /etc/ceph/ceph.client.radosgw.keyring

(2) modify Ceph cluster configurations and synchronize it
  ceph-deploy --overwrite-conf config push node1 node2 node3 node4 node5

(3) configure Apache
(4) configure region
  sudo apt-get install -y radosgw radosgw-agent python-pip
  sudo radosgw-admin region set --infile /home/ceph/us.json --name 
client.radosgw.us-east-1
  sudo radosgw-admin region set --infile /home/ceph/us.json --name 
client.radosgw.us-west-1
  sudo rados -p .us.rgw.root rm region_info.default
  sudo radosgw-admin region default --rgw-region=us --name 
client.radosgw.us-east-1
  sudo radosgw-admin region default --rgw-region=us --name 
client.radosgw.us-west-1
  sudo radosgw-admin regionmap update --name client.radosgw.us-east-1
  sudo radosgw-admin regionmap update --name client.radosgw.us-west-1

(5) create master zone
  sudo radosgw-admin zone set --rgw-zone=us-east --infile 
/home/ceph/us-east.json --name client.radosgw.us-east-1
  sudo radosgw-admin zone set --rgw-zone=us-east --infile 
/home/ceph/us-east.json --name client.radosgw.us-west-1
  sudo radosgw-admin zone set --rgw-zone=us-west --infile 
/home/ceph/us-west.json --name client.radosgw.us-east-1
  sudo radosgw-admin zone set --rgw-zone=us-west --infile 
/home/ceph/us-west.json --name client.radosgw.us-west-1
  sudo rados -p .rgw.root rm zone_info.default
  sudo radosgw-admin regionmap update --name client.radosgw.us-east-1
  sudo radosgw-admin regionmap update --name client.radosgw.us-west-1

(6) create master zone's users
  sudo radosgw-admin user create --uid="us-east" --display-name="Region-us 
Zone-East" --name client.radosgw.us-east-1 --system --access_key=us_access_key 
--secret=us_secret_key
  sudo radosgw-admin user create --uid="us-west" --display-name="Region-us 
Zone-West" --name client.radosgw.us-west-1 --system --access_key=us_access_key 
--secret=us_secret_key

(7) restart Ceph & apache2 & radosgw services



2. For the secondary zone with 5 nodes of the region
(1) copy the keyring file 'ceph.client.radosgw.keyring' from master zone and 
import the keyring
  sudo ceph -k /etc/ceph/ceph.client.admin.keyring auth add 
client.radosgw.us-east-1 -i /etc/ceph/ceph.client.radosgw.keyring
  sudo ceph -k /etc/ceph/ceph.client.admin.keyring auth add 
client.radosgw.us-west-1 -i /etc/ceph/ceph.client.radosgw.keyring

(2) modify Ceph cluster configurations and synchronize it
ceph-deploy --overwrite-conf config push node1 node2 node3 node4 node5

(3) configure Apache
(4) copy infile '/home/ceph/us.json' for the master zone and create 'us' region
  sudo apt-get install -y radosgw radosgw-agent python-pip
  sudo radosgw-admin region set --infile /home/ceph/us.json --name 
client.radosgw.us-east-1
  sudo radosgw-admin region set --infile /home/ceph/us.json --name 
client.radosgw.us-west-1
  sudo radosgw-admin region default --rgw-region=us --name 
client.radosgw.us-east-1
  sudo radosgw-admin region default --rgw-region=us --name 
client.radosgw.us-west-1
  sudo radosgw-admin regionmap update --name client.radosgw.us-east-1
  sudo radosgw-admin regionmap update --name client.radosgw.us-west-1

(5) create secondary one
  sudo radosgw-admin zone set --rgw-zone=us-east --infile 
/home/ceph/us-east.json --name client.radosgw.us-east-1
  sudo radosgw-admin zone set --rgw-zone=us-east --infile 
/home/ceph/us-east.json --name client.radosgw.us-west-1
  sudo radosgw-admin zone set --rgw-zone=us-west --infile 
/home/ceph/us-west.json --name client.radosgw.us-east-1
  sudo radosgw-admin zone set --rgw-zone=us-west --infile 
/home/ceph/us-west.json --name client.radosgw.us-west-1
  sudo rados -p .rgw.root rm zone_info.default
  sudo radosgw-admin regionmap update --name client.radosgw.us-east-1
  sudo radosgw-admin regionmap update --name client.radosgw.

Re: [ceph-users] Federated gateways

[Craig Lewis]

I have identical regionmaps in both clusters.

I only created the zone's pools in that cluster.  I didn't delete the
default .rgw.* pools, so those exist in both zones.

Both users need to be system on both ends, and have identical access and
secrets.  If they're not, this is likely your problem.


On Fri, Nov 14, 2014 at 11:38 AM, Aaron Bassett 
wrote:

> Well I upgraded both clusters to giant this morning just to see if that
> would help, and it didn’t. I have a couple questions though. I have the
> same regionmap on both clusters, with both zones in it, but then i only
> have the buckets and zone info for one zone in each cluster, is this right?
> Or do I need all the buckets and zones in both clusters? Reading the docs
> it doesn’t seem like i do because I’m expecting data to sync from one zone
> in one cluster to the other zone on the other cluster, but I don’t know
> what to think anymore.
>
> Also do both users need to be system users on both ends?
>
> Aaron
>
>
>
> On Nov 12, 2014, at 4:00 PM, Craig Lewis 
> wrote:
>
> http://tracker.ceph.com/issues/9206
>
> My post to the ML: http://www.spinics.net/lists/ceph-users/msg12665.html
>
>
> IIRC, the system uses didn't see the other user's bucket in a bucket
> listing, but they could read and write the objects fine.
>
>
>
> On Wed, Nov 12, 2014 at 11:16 AM, Aaron Bassett 
> wrote:
>
>> In playing around with this a bit more, I noticed that the two users on
>> the secondary node cant see each others buckets. Is this a problem?
>>
>
> IIRC, the system user couldn't see each other's buckets, but they could
> read and write the objects.
>
>> On Nov 11, 2014, at 6:56 PM, Craig Lewis 
>> wrote:
>>
>> I see you're running 0.80.5.  Are you using Apache 2.4?  There is a known
>>> issue with Apache 2.4 on the primary and replication.  It's fixed, just
>>> waiting for the next firefly release.  Although, that causes 40x errors
>>> with Apache 2.4, not 500 errors.
>>>
>>> It is apache 2.4, but I’m actually running 0.80.7 so I probably have
>>> that bug fix?
>>>
>>>
>> No, the unreleased 0.80.8 has the fix.
>>
>>
>>
>>>
>>> Have you verified that both system users can read and write to both
>>> clusters?  (Just make sure you clean up the writes to the slave cluster).
>>>
>>> Yes I can write everywhere and radosgw-agent isn’t getting any 403s like
>>> it was earlier when I had mismatched keys. The .us-nh.rgw.buckets.index
>>> pool is syncing properly, as are the users. It seems like really the only
>>> thing that isn’t syncing is the .zone.rgw.buckets pool.
>>>
>>
>> That's pretty much the same behavior I was seeing with Apache 2.4.
>>
>> Try downgrading the primary cluster to Apache 2.2.  In my testing, the
>> secondary cluster could run 2.2 or 2.4.
>>
>> Do you have a link to that bug#? I want to see if it gives me any clues.
>>
>> Aaron
>>
>>
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Federated gateways

[Aaron Bassett]

Well I upgraded both clusters to giant this morning just to see if that would 
help, and it didn’t. I have a couple questions though. I have the same 
regionmap on both clusters, with both zones in it, but then i only have the 
buckets and zone info for one zone in each cluster, is this right? Or do I need 
all the buckets and zones in both clusters? Reading the docs it doesn’t seem 
like i do because I’m expecting data to sync from one zone in one cluster to 
the other zone on the other cluster, but I don’t know what to think anymore. 

Also do both users need to be system users on both ends? 

Aaron


> On Nov 12, 2014, at 4:00 PM, Craig Lewis  wrote:
> 
> http://tracker.ceph.com/issues/9206 
> 
> My post to the ML: http://www.spinics.net/lists/ceph-users/msg12665.html 
> 
> 
> 
> IIRC, the system uses didn't see the other user's bucket in a bucket listing, 
> but they could read and write the objects fine.
> 
> 
> 
> On Wed, Nov 12, 2014 at 11:16 AM, Aaron Bassett  > wrote:
> In playing around with this a bit more, I noticed that the two users on the 
> secondary node cant see each others buckets. Is this a problem?
> 
> IIRC, the system user couldn't see each other's buckets, but they could read 
> and write the objects. 
>> On Nov 11, 2014, at 6:56 PM, Craig Lewis > > wrote:
>> 
>>> I see you're running 0.80.5.  Are you using Apache 2.4?  There is a known 
>>> issue with Apache 2.4 on the primary and replication.  It's fixed, just 
>>> waiting for the next firefly release.  Although, that causes 40x errors 
>>> with Apache 2.4, not 500 errors.
>> It is apache 2.4, but I’m actually running 0.80.7 so I probably have that 
>> bug fix?
>> 
>> 
>> No, the unreleased 0.80.8 has the fix.
>>  
>>  
>>> 
>>> Have you verified that both system users can read and write to both 
>>> clusters?  (Just make sure you clean up the writes to the slave cluster).
>> Yes I can write everywhere and radosgw-agent isn’t getting any 403s like it 
>> was earlier when I had mismatched keys. The .us-nh.rgw.buckets.index pool is 
>> syncing properly, as are the users. It seems like really the only thing that 
>> isn’t syncing is the .zone.rgw.buckets pool.
>> 
>> That's pretty much the same behavior I was seeing with Apache 2.4.
>> 
>> Try downgrading the primary cluster to Apache 2.2.  In my testing, the 
>> secondary cluster could run 2.2 or 2.4.
> Do you have a link to that bug#? I want to see if it gives me any clues. 
> 
> Aaron 
> 
> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Federated gateways

[Craig Lewis]

http://tracker.ceph.com/issues/9206

My post to the ML: http://www.spinics.net/lists/ceph-users/msg12665.html


IIRC, the system uses didn't see the other user's bucket in a bucket
listing, but they could read and write the objects fine.



On Wed, Nov 12, 2014 at 11:16 AM, Aaron Bassett 
wrote:

> In playing around with this a bit more, I noticed that the two users on
> the secondary node cant see each others buckets. Is this a problem?
>

IIRC, the system user couldn't see each other's buckets, but they could
read and write the objects.

> On Nov 11, 2014, at 6:56 PM, Craig Lewis 
> wrote:
>
> I see you're running 0.80.5.  Are you using Apache 2.4?  There is a known
>> issue with Apache 2.4 on the primary and replication.  It's fixed, just
>> waiting for the next firefly release.  Although, that causes 40x errors
>> with Apache 2.4, not 500 errors.
>>
>> It is apache 2.4, but I’m actually running 0.80.7 so I probably have that
>> bug fix?
>>
>>
> No, the unreleased 0.80.8 has the fix.
>
>
>
>>
>> Have you verified that both system users can read and write to both
>> clusters?  (Just make sure you clean up the writes to the slave cluster).
>>
>> Yes I can write everywhere and radosgw-agent isn’t getting any 403s like
>> it was earlier when I had mismatched keys. The .us-nh.rgw.buckets.index
>> pool is syncing properly, as are the users. It seems like really the only
>> thing that isn’t syncing is the .zone.rgw.buckets pool.
>>
>
> That's pretty much the same behavior I was seeing with Apache 2.4.
>
> Try downgrading the primary cluster to Apache 2.2.  In my testing, the
> secondary cluster could run 2.2 or 2.4.
>
> Do you have a link to that bug#? I want to see if it gives me any clues.
>
> Aaron
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Federated gateways

[Aaron Bassett]

In playing around with this a bit more, I noticed that the two users on the 
secondary node cant see each others buckets. Is this a problem?
> On Nov 11, 2014, at 6:56 PM, Craig Lewis  wrote:
> 
>> I see you're running 0.80.5.  Are you using Apache 2.4?  There is a known 
>> issue with Apache 2.4 on the primary and replication.  It's fixed, just 
>> waiting for the next firefly release.  Although, that causes 40x errors with 
>> Apache 2.4, not 500 errors.
> It is apache 2.4, but I’m actually running 0.80.7 so I probably have that bug 
> fix?
> 
> 
> No, the unreleased 0.80.8 has the fix.
>  
>  
>> 
>> Have you verified that both system users can read and write to both 
>> clusters?  (Just make sure you clean up the writes to the slave cluster).
> Yes I can write everywhere and radosgw-agent isn’t getting any 403s like it 
> was earlier when I had mismatched keys. The .us-nh.rgw.buckets.index pool is 
> syncing properly, as are the users. It seems like really the only thing that 
> isn’t syncing is the .zone.rgw.buckets pool.
> 
> That's pretty much the same behavior I was seeing with Apache 2.4.
> 
> Try downgrading the primary cluster to Apache 2.2.  In my testing, the 
> secondary cluster could run 2.2 or 2.4.
Do you have a link to that bug#? I want to see if it gives me any clues. 

Aaron 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Federated gateways

[Craig Lewis]

>
> I see you're running 0.80.5.  Are you using Apache 2.4?  There is a known
> issue with Apache 2.4 on the primary and replication.  It's fixed, just
> waiting for the next firefly release.  Although, that causes 40x errors
> with Apache 2.4, not 500 errors.
>
> It is apache 2.4, but I’m actually running 0.80.7 so I probably have that
> bug fix?
>
>
No, the unreleased 0.80.8 has the fix.



>
> Have you verified that both system users can read and write to both
> clusters?  (Just make sure you clean up the writes to the slave cluster).
>
> Yes I can write everywhere and radosgw-agent isn’t getting any 403s like
> it was earlier when I had mismatched keys. The .us-nh.rgw.buckets.index
> pool is syncing properly, as are the users. It seems like really the only
> thing that isn’t syncing is the .zone.rgw.buckets pool.
>

That's pretty much the same behavior I was seeing with Apache 2.4.

Try downgrading the primary cluster to Apache 2.2.  In my testing, the
secondary cluster could run 2.2 or 2.4.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Federated gateways

[Aaron Bassett]

> On Nov 11, 2014, at 4:21 PM, Craig Lewis  wrote:
> 
> Is that radosgw log from the primary or the secondary zone?  Nothing in that 
> log jumps out at me.
This is the log from the secondary zone. That HTTP 500 response code coming 
back is the only problem I can find. There are a bunch of 404s from other 
requests to logs and stuff, but I assume those are normal because there’s no 
activity going on. I guess it’s just that cryptic  WARNING: set_req_state_err 
err_no=5 resorting to 500 line that’s the problem. I think I need to get a 
stack trace from that somehow. 

> I see you're running 0.80.5.  Are you using Apache 2.4?  There is a known 
> issue with Apache 2.4 on the primary and replication.  It's fixed, just 
> waiting for the next firefly release.  Although, that causes 40x errors with 
> Apache 2.4, not 500 errors.
It is apache 2.4, but I’m actually running 0.80.7 so I probably have that bug 
fix?

> 
> Have you verified that both system users can read and write to both clusters? 
>  (Just make sure you clean up the writes to the slave cluster).
Yes I can write everywhere and radosgw-agent isn’t getting any 403s like it was 
earlier when I had mismatched keys. The .us-nh.rgw.buckets.index pool is 
syncing properly, as are the users. It seems like really the only thing that 
isn’t syncing is the .zone.rgw.buckets pool.

Thanks, Aaron 
> 
> 
> 
> 
> On Tue, Nov 11, 2014 at 6:51 AM, Aaron Bassett  > wrote:
> Ok I believe I’ve made some progress here. I have everything syncing *except* 
> data. The data is getting 500s when it tries to sync to the backup zone. I 
> have a log from the radosgw with debug cranked up to 20:
> 
> 2014-11-11 14:37:06.688331 7f54447f0700  1 == starting new request 
> req=0x7f546800f3b0 =
> 2014-11-11 14:37:06.688978 7f54447f0700  0 WARNING: couldn't find acl header 
> for bucket, generating default
> 2014-11-11 14:37:06.689358 7f54447f0700  1 -- 172.16.10.103:0/1007381 
>  --> 172.16.10.103:6934/14875 
>  -- osd_op(client.5673295.0:1783 
> statelog.obj_opstate.97 [call statelog.add] 193.1cf20a5a ondisk+write e47531) 
> v4 -- ?+0 0x7f534800d770 con 0x7f53f00053f0
> 2014-11-11 14:37:06.689396 7f54447f0700 20 -- 172.16.10.103:0/1007381 
>  submit_message osd_op(client.5673295.0:1783 
> statelog.obj_opstate.97 [call statelog.add] 193.1cf20a5a ondisk+write e47531) 
> v4 remote, 172.16.10.103:6934/14875 , have 
> pipe.
> 2014-11-11 14:37:06.689481 7f51ff1f1700 10 -- 172.16.10.103:0/1007381 
>  >> 172.16.10.103:6934/14875 
>  pipe(0x7f53f0005160 sd=61 :33168 s=2 
> pgs=2524 cs=1 l=1 c=0x7f53f00053f0).writer: state = open policy.server=0
> 2014-11-11 14:37:06.689592 7f51ff1f1700 20 -- 172.16.10.103:0/1007381 
>  >> 172.16.10.103:6934/14875 
>  pipe(0x7f53f0005160 sd=61 :33168 s=2 
> pgs=2524 cs=1 l=1 c=0x7f53f00053f0).writer encoding 48 features 
> 17592186044415 0x7f534800d770 osd_op(client.5673295.0:1783 
> statelog.obj_opstate.97 [call statelog.add] 193.1cf20a5a ondisk+write e47531) 
> v4
> 2014-11-11 14:37:06.689756 7f51ff1f1700 20 -- 172.16.10.103:0/1007381 
>  >> 172.16.10.103:6934/14875 
>  pipe(0x7f53f0005160 sd=61 :33168 s=2 
> pgs=2524 cs=1 l=1 c=0x7f53f00053f0).writer signed seq # 48): sig = 
> 206599450695048354
> 2014-11-11 14:37:06.689804 7f51ff1f1700 20 -- 172.16.10.103:0/1007381 
>  >> 172.16.10.103:6934/14875 
>  pipe(0x7f53f0005160 sd=61 :33168 s=2 
> pgs=2524 cs=1 l=1 c=0x7f53f00053f0).writer sending 48 0x7f534800d770
> 2014-11-11 14:37:06.689884 7f51ff1f1700 10 -- 172.16.10.103:0/1007381 
>  >> 172.16.10.103:6934/14875 
>  pipe(0x7f53f0005160 sd=61 :33168 s=2 
> pgs=2524 cs=1 l=1 c=0x7f53f00053f0).writer: state = open policy.server=0
> 2014-11-11 14:37:06.689915 7f51ff1f1700 20 -- 172.16.10.103:0/1007381 
>  >> 172.16.10.103:6934/14875 
>  pipe(0x7f53f0005160 sd=61 :33168 s=2 
> pgs=2524 cs=1 l=1 c=0x7f53f00053f0).writer sleeping
> 2014-11-11 14:37:06.694968 7f51ff0f0700 20 -- 172.16.10.103:0/1007381 
>  >> 172.16.10.103:6934/14875 
>  pipe(0x7f53f0005160 sd=61 :33168 s=2 
> pgs=2524 cs=1 l=1 c=0x7f53f00053f0).reader got ACK
> 2014-11-11 14:37:06.695053 7f51ff0f0700 15 -- 172.16.10.103:0/1007381 
>  >> 172.16.10.103:6934/14875 
>  pipe(0x7f53f0005160 sd=61 :33168 s=2 
> pgs=2524 cs=1 l=1 c=0x7f53f00053f0).reader got ack seq 48
> 2014-11-11 14:37:06.695067 7f51ff0f0700 20 -- 172.16.10

Re: [ceph-users] Federated gateways

[Craig Lewis]

Is that radosgw log from the primary or the secondary zone?  Nothing in
that log jumps out at me.

I see you're running 0.80.5.  Are you using Apache 2.4?  There is a known
issue with Apache 2.4 on the primary and replication.  It's fixed, just
waiting for the next firefly release.  Although, that causes 40x errors
with Apache 2.4, not 500 errors.

Have you verified that both system users can read and write to both
clusters?  (Just make sure you clean up the writes to the slave cluster).




On Tue, Nov 11, 2014 at 6:51 AM, Aaron Bassett 
wrote:

> Ok I believe I’ve made some progress here. I have everything syncing
> *except* data. The data is getting 500s when it tries to sync to the backup
> zone. I have a log from the radosgw with debug cranked up to 20:
>
> 2014-11-11 14:37:06.688331 7f54447f0700  1 == starting new request
> req=0x7f546800f3b0 =
> 2014-11-11 14:37:06.688978 7f54447f0700  0 WARNING: couldn't find acl
> header for bucket, generating default
> 2014-11-11 14:37:06.689358 7f54447f0700  1 -- 172.16.10.103:0/1007381 -->
> 172.16.10.103:6934/14875 -- osd_op(client.5673295.0:1783
> statelog.obj_opstate.97 [call statelog.add] 193.1cf20a5a ondisk+write
> e47531) v4 -- ?+0 0x7f534800d770 con 0x7f53f00053f0
> 2014-11-11 14:37:06.689396 7f54447f0700 20 -- 172.16.10.103:0/1007381
> submit_message osd_op(client.5673295.0:1783 statelog.obj_opstate.97 [call
> statelog.add] 193.1cf20a5a ondisk+write e47531) v4 remote,
> 172.16.10.103:6934/14875, have pipe.
> 2014-11-11 14:37:06.689481 7f51ff1f1700 10 -- 172.16.10.103:0/1007381 >>
> 172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524
> cs=1 l=1 c=0x7f53f00053f0).writer: state = open policy.server=0
> 2014-11-11 14:37:06.689592 7f51ff1f1700 20 -- 172.16.10.103:0/1007381 >>
> 172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524
> cs=1 l=1 c=0x7f53f00053f0).writer encoding 48 features 17592186044415
> 0x7f534800d770 osd_op(client.5673295.0:1783 statelog.obj_opstate.97 [call
> statelog.add] 193.1cf20a5a ondisk+write e47531) v4
> 2014-11-11 14:37:06.689756 7f51ff1f1700 20 -- 172.16.10.103:0/1007381 >>
> 172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524
> cs=1 l=1 c=0x7f53f00053f0).writer signed seq # 48): sig = 206599450695048354
> 2014-11-11 14:37:06.689804 7f51ff1f1700 20 -- 172.16.10.103:0/1007381 >>
> 172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524
> cs=1 l=1 c=0x7f53f00053f0).writer sending 48 0x7f534800d770
> 2014-11-11 14:37:06.689884 7f51ff1f1700 10 -- 172.16.10.103:0/1007381 >>
> 172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524
> cs=1 l=1 c=0x7f53f00053f0).writer: state = open policy.server=0
> 2014-11-11 14:37:06.689915 7f51ff1f1700 20 -- 172.16.10.103:0/1007381 >>
> 172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524
> cs=1 l=1 c=0x7f53f00053f0).writer sleeping
> 2014-11-11 14:37:06.694968 7f51ff0f0700 20 -- 172.16.10.103:0/1007381 >>
> 172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524
> cs=1 l=1 c=0x7f53f00053f0).reader got ACK
> 2014-11-11 14:37:06.695053 7f51ff0f0700 15 -- 172.16.10.103:0/1007381 >>
> 172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524
> cs=1 l=1 c=0x7f53f00053f0).reader got ack seq 48
> 2014-11-11 14:37:06.695067 7f51ff0f0700 20 -- 172.16.10.103:0/1007381 >>
> 172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524
> cs=1 l=1 c=0x7f53f00053f0).reader reading tag...
> 2014-11-11 14:37:06.695079 7f51ff0f0700 20 -- 172.16.10.103:0/1007381 >>
> 172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524
> cs=1 l=1 c=0x7f53f00053f0).reader got MSG
> 2014-11-11 14:37:06.695093 7f51ff0f0700 20 -- 172.16.10.103:0/1007381 >>
> 172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524
> cs=1 l=1 c=0x7f53f00053f0).reader got envelope type=43 src osd.25 front=190
> data=0 off 0
> 2014-11-11 14:37:06.695108 7f51ff0f0700 10 -- 172.16.10.103:0/1007381 >>
> 172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524
> cs=1 l=1 c=0x7f53f00053f0).reader wants 190 from dispatch throttler
> 0/104857600
> 2014-11-11 14:37:06.695135 7f51ff0f0700 20 -- 172.16.10.103:0/1007381 >>
> 172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524
> cs=1 l=1 c=0x7f53f00053f0).reader got front 190
> 2014-11-11 14:37:06.695150 7f51ff0f0700 10 -- 172.16.10.103:0/1007381 >>
> 172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524
> cs=1 l=1 c=0x7f53f00053f0).aborted = 0
> 2014-11-11 14:37:06.695158 7f51ff0f0700 20 -- 172.16.10.103:0/1007381 >>
> 172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524
> cs=1 l=1 c=0x7f53f00053f0).reader got 190 + 0 + 0 byte message
> 2014-11-11 14:37:06.695284 7f51ff0f0700 10 -- 172.16.10.103:0/1007381 >>
> 172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524
> cs=1 l=1 c=0x7f53f00053f0).reader got message 48 0x7f51b4001950
> osd_op_reply(1783 statelog.ob

Re: [ceph-users] Federated gateways

[Aaron Bassett]

Ok I believe I’ve made some progress here. I have everything syncing *except* 
data. The data is getting 500s when it tries to sync to the backup zone. I have 
a log from the radosgw with debug cranked up to 20:

2014-11-11 14:37:06.688331 7f54447f0700  1 == starting new request 
req=0x7f546800f3b0 =
2014-11-11 14:37:06.688978 7f54447f0700  0 WARNING: couldn't find acl header 
for bucket, generating default
2014-11-11 14:37:06.689358 7f54447f0700  1 -- 172.16.10.103:0/1007381 --> 
172.16.10.103:6934/14875 -- osd_op(client.5673295.0:1783 
statelog.obj_opstate.97 [call statelog.add] 193.1cf20a5a ondisk+write e47531) 
v4 -- ?+0 0x7f534800d770 con 0x7f53f00053f0
2014-11-11 14:37:06.689396 7f54447f0700 20 -- 172.16.10.103:0/1007381 
submit_message osd_op(client.5673295.0:1783 statelog.obj_opstate.97 [call 
statelog.add] 193.1cf20a5a ondisk+write e47531) v4 remote, 
172.16.10.103:6934/14875, have pipe.
2014-11-11 14:37:06.689481 7f51ff1f1700 10 -- 172.16.10.103:0/1007381 >> 
172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524 cs=1 l=1 
c=0x7f53f00053f0).writer: state = open policy.server=0
2014-11-11 14:37:06.689592 7f51ff1f1700 20 -- 172.16.10.103:0/1007381 >> 
172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524 cs=1 l=1 
c=0x7f53f00053f0).writer encoding 48 features 17592186044415 0x7f534800d770 
osd_op(client.5673295.0:1783 statelog.obj_opstate.97 [call statelog.add] 
193.1cf20a5a ondisk+write e47531) v4
2014-11-11 14:37:06.689756 7f51ff1f1700 20 -- 172.16.10.103:0/1007381 >> 
172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524 cs=1 l=1 
c=0x7f53f00053f0).writer signed seq # 48): sig = 206599450695048354
2014-11-11 14:37:06.689804 7f51ff1f1700 20 -- 172.16.10.103:0/1007381 >> 
172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524 cs=1 l=1 
c=0x7f53f00053f0).writer sending 48 0x7f534800d770
2014-11-11 14:37:06.689884 7f51ff1f1700 10 -- 172.16.10.103:0/1007381 >> 
172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524 cs=1 l=1 
c=0x7f53f00053f0).writer: state = open policy.server=0
2014-11-11 14:37:06.689915 7f51ff1f1700 20 -- 172.16.10.103:0/1007381 >> 
172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524 cs=1 l=1 
c=0x7f53f00053f0).writer sleeping
2014-11-11 14:37:06.694968 7f51ff0f0700 20 -- 172.16.10.103:0/1007381 >> 
172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524 cs=1 l=1 
c=0x7f53f00053f0).reader got ACK
2014-11-11 14:37:06.695053 7f51ff0f0700 15 -- 172.16.10.103:0/1007381 >> 
172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524 cs=1 l=1 
c=0x7f53f00053f0).reader got ack seq 48
2014-11-11 14:37:06.695067 7f51ff0f0700 20 -- 172.16.10.103:0/1007381 >> 
172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524 cs=1 l=1 
c=0x7f53f00053f0).reader reading tag...
2014-11-11 14:37:06.695079 7f51ff0f0700 20 -- 172.16.10.103:0/1007381 >> 
172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524 cs=1 l=1 
c=0x7f53f00053f0).reader got MSG
2014-11-11 14:37:06.695093 7f51ff0f0700 20 -- 172.16.10.103:0/1007381 >> 
172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524 cs=1 l=1 
c=0x7f53f00053f0).reader got envelope type=43 src osd.25 front=190 data=0 off 0
2014-11-11 14:37:06.695108 7f51ff0f0700 10 -- 172.16.10.103:0/1007381 >> 
172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524 cs=1 l=1 
c=0x7f53f00053f0).reader wants 190 from dispatch throttler 0/104857600
2014-11-11 14:37:06.695135 7f51ff0f0700 20 -- 172.16.10.103:0/1007381 >> 
172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524 cs=1 l=1 
c=0x7f53f00053f0).reader got front 190
2014-11-11 14:37:06.695150 7f51ff0f0700 10 -- 172.16.10.103:0/1007381 >> 
172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524 cs=1 l=1 
c=0x7f53f00053f0).aborted = 0
2014-11-11 14:37:06.695158 7f51ff0f0700 20 -- 172.16.10.103:0/1007381 >> 
172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524 cs=1 l=1 
c=0x7f53f00053f0).reader got 190 + 0 + 0 byte message
2014-11-11 14:37:06.695284 7f51ff0f0700 10 -- 172.16.10.103:0/1007381 >> 
172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524 cs=1 l=1 
c=0x7f53f00053f0).reader got message 48 0x7f51b4001950 osd_op_reply(1783 
statelog.obj_opstate.97 [call] v47531'13 uv13 ondisk = 0) v6
2014-11-11 14:37:06.695313 7f51ff0f0700 20 -- 172.16.10.103:0/1007381 queue 
0x7f51b4001950 prio 127
2014-11-11 14:37:06.695374 7f51ff0f0700 20 -- 172.16.10.103:0/1007381 >> 
172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524 cs=1 l=1 
c=0x7f53f00053f0).reader reading tag...
2014-11-11 14:37:06.695384 7f51ff1f1700 10 -- 172.16.10.103:0/1007381 >> 
172.16.10.103:6934/14875 pipe(0x7f53f0005160 sd=61 :33168 s=2 pgs=2524 cs=1 l=1 
c=0x7f53f00053f0).writer: state = open policy.server=0
2014-11-11 14:37:06.695426 7f51ff1f1700 10 -- 172.16.10.103:0/1007381 >> 
172.16.10.103:6

Re: [ceph-users] Federated gateways

[Aaron Bassett]

Ah so I need both users in both clusters? I think I missed that bit, let me see 
if that does the trick.

Aaron 
> On Nov 5, 2014, at 2:59 PM, Craig Lewis  wrote:
> 
> One region two zones is the standard setup, so that should be fine.
> 
> Is metadata (users and buckets) being replicated, but not data (objects)? 
> 
> 
> Let's go through a quick checklist:
> Verify that you enabled log_meta and log_data in the region.json for the 
> master zone
> Verify that RadosGW is using your region map with radosgw-admin regionmap get 
> --name client.radosgw. 
> Verifu 
> Verify that RadosGW is using your zone map with radosgw-admin zone get --name 
> client.radosgw. 
> Verify that all the pools in your zone exist (RadosGW only auto-creates the 
> basic ones).
> Verify that your system users exist in both zones with the same access and 
> secret.
> Hopefully that gives you an idea what's not working correctly.  
> 
> If it doesn't, crank up the logging on the radosgw daemon on both sides, and 
> check the logs.  Add debug rgw = 20 to both ceph.conf (in the 
> client.radosgw. section), and restart.  Hopefully those logs will tell 
> you what's wrong.
> 
> 
> On Wed, Nov 5, 2014 at 11:39 AM, Aaron Bassett  > wrote:
> Hello everyone, 
> I am attempted to setup a two cluster situation for object storage disaster 
> recovery. I have two physically separate sites so using 1 big cluster isn’t 
> an option. I’m attempting to follow the guide at: 
> http://ceph.com/docs/v0.80.5/radosgw/federated-config/ 
>  . After a couple 
> days of flailing, I’ve settled on using 1 region with two zones, where each 
> cluster is a zone. I’m now attempting to set up an agent as per the 
> “Multi-Site Data Replication section. The agent kicks off ok and starts 
> making all sorts of connections, but no objects were being copied to the 
> non-master zone. I re-ran the agent with the -v flag and saw a lot of:
> 
> DEBUG:urllib3.connectionpool:"GET 
> /admin/opstate?client-id=radosgw-agent&object=test%2F_shadow_.JjVixjWmebQTrRed36FL6D0vy2gDVZ__39&op-id=phx-r1-head1%3A2451615%3A1
>  HTTP/1.1" 200 None   
>   
> DEBUG:radosgw_agent.worker:op state is [] 
>  
> DEBUG:radosgw_agent.worker:error geting op state: list index out of range 
>  
> 
> So it appears something is still wrong with my agent though I have no idea 
> what. I can’t seem to find any errors in any other logs. Does anyone have any 
> insight here? 
> 
> I’m also wondering if what I’m attempting with two cluster in the same region 
> as separate zones makes sense?
> 
> Thanks, Aaron 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
> 
> 
> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Federated gateways

[Craig Lewis]

One region two zones is the standard setup, so that should be fine.

Is metadata (users and buckets) being replicated, but not data (objects)?


Let's go through a quick checklist:

   - Verify that you enabled log_meta and log_data in the region.json for
   the master zone
   - Verify that RadosGW is using your region map with radosgw-admin
   regionmap get --name client.radosgw.
   - Verifu
   - Verify that RadosGW is using your zone map with radosgw-admin zone get
   --name client.radosgw.
   - Verify that all the pools in your zone exist (RadosGW only
   auto-creates the basic ones).
   - Verify that your system users exist in both zones with the same access
   and secret.

Hopefully that gives you an idea what's not working correctly.

If it doesn't, crank up the logging on the radosgw daemon on both sides,
and check the logs.  Add debug rgw = 20 to both ceph.conf (in the
client.radosgw. section), and restart.  Hopefully those logs will
tell you what's wrong.


On Wed, Nov 5, 2014 at 11:39 AM, Aaron Bassett 
wrote:

> Hello everyone,
> I am attempted to setup a two cluster situation for object storage
> disaster recovery. I have two physically separate sites so using 1 big
> cluster isn’t an option. I’m attempting to follow the guide at:
> http://ceph.com/docs/v0.80.5/radosgw/federated-config/ . After a couple
> days of flailing, I’ve settled on using 1 region with two zones, where each
> cluster is a zone. I’m now attempting to set up an agent as per the
> “Multi-Site Data Replication section. The agent kicks off ok and starts
> making all sorts of connections, but no objects were being copied to the
> non-master zone. I re-ran the agent with the -v flag and saw a lot of:
>
> DEBUG:urllib3.connectionpool:"GET
> /admin/opstate?client-id=radosgw-agent&object=test%2F_shadow_.JjVixjWmebQTrRed36FL6D0vy2gDVZ__39&op-id=phx-r1-head1%3A2451615%3A1
> HTTP/1.1" 200 None
>
> DEBUG:radosgw_agent.worker:op state is []
>
> DEBUG:radosgw_agent.worker:error geting op state: list index out of range
>
>
> So it appears something is still wrong with my agent though I have no idea
> what. I can’t seem to find any errors in any other logs. Does anyone have
> any insight here?
>
> I’m also wondering if what I’m attempting with two cluster in the same
> region as separate zones makes sense?
>
> Thanks, Aaron
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Federated gateways

[Aaron Bassett]

Hello everyone, 
I am attempted to setup a two cluster situation for object storage disaster 
recovery. I have two physically separate sites so using 1 big cluster isn’t an 
option. I’m attempting to follow the guide at: 
http://ceph.com/docs/v0.80.5/radosgw/federated-config/ 
 . After a couple days 
of flailing, I’ve settled on using 1 region with two zones, where each cluster 
is a zone. I’m now attempting to set up an agent as per the “Multi-Site Data 
Replication section. The agent kicks off ok and starts making all sorts of 
connections, but no objects were being copied to the non-master zone. I re-ran 
the agent with the -v flag and saw a lot of:

DEBUG:urllib3.connectionpool:"GET 
/admin/opstate?client-id=radosgw-agent&object=test%2F_shadow_.JjVixjWmebQTrRed36FL6D0vy2gDVZ__39&op-id=phx-r1-head1%3A2451615%3A1
 HTTP/1.1" 200 None 

DEBUG:radosgw_agent.worker:op state is []   
   
DEBUG:radosgw_agent.worker:error geting op state: list index out of range   
   

So it appears something is still wrong with my agent though I have no idea 
what. I can’t seem to find any errors in any other logs. Does anyone have any 
insight here? 

I’m also wondering if what I’m attempting with two cluster in the same region 
as separate zones makes sense?

Thanks, Aaron ___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Federated gateways (our planning use case)

[David Barker]

I've had some luck putting a load balancer infront of multiple zones to get
around the multiple URL issue. You can get the LB to send POST/DELETE et al
to the primary zone, but GET requests can be distributed to multiple zones.
The only issue is the replication delay; your data may not be available on
the secondary for reading yet...

I'm pretty sure you can most LB's to check multiple zones for the object
existing when doing a GET, and redirect to the primary if replication
hasn't caught up; for what I was looking at, I didn't need this!

Dave

On Tue, Oct 7, 2014 at 1:33 AM, Craig Lewis 
wrote:

> This sounds doable, with a few caveats.
>
>
> Currently, replication is only one direction.  You can only write to the
> primary zone, and you can read from the primary or secondary zones.  A
> cluster can have many zones on it.
>
> I'm thinking your setup would be a star topology.  Each telescope will be
> a primary zone, and replicate to a secondary zone in the main storage
> cluster.  The main cluster will have one read-only secondary zone for each
> telescope.  If you have other needs to write data to the main cluster, you
> can create another zone that only exists on the main cluster (possibly
> replicated to one of the telescopes with a good network connection).
>
> Each zone has it's own URL (primary and secondary), so you'd have a bit of
> management problem remembering to use the correct URL. The URLs can be
> whatever.  Convention follow's Amazon's naming scheme, but you'd probably
> want to create your own scheme, something like
> http://telescope_name-site.inasan.ru/ and
> http://telescope_name-campus.inasan.ru/
>
>
> You might have some problems with the replication if your VPN connections
> aren't stable.  The replication agent isn't very tolerant of cluster
> problem, so I suspect (but haven't tested) that long VPN outages will need
> a replication agent restart.  For sites that don't have permanent
> connections, just make the replication agent startup and shutdown part of
> the connection startup and shutdown process.  Replication state is
> available via a REST api, so it can be monitored.
>
>
> I have tested large backlogs in replication.  When I initially imported my
> data, I deliberately imported faster than I had bandwidth to replicate.  At
> one point, my secondary cluster was ~10 million objects, and ~10TB behind
> the primary cluster.  It eventually caught up, but the process doesn't
> handle stops and restarts well.  Restarting the replication while it was
> dealing with the backlog will start from the beginning of the backlog.
> This can be a problem if your backlog is so large that it won't finish in a
> day, because log rotation will restart the replication agent.  If that's
> something you think might be a problem, I have some strategies to deal with
> it, but they're manual and hacky.
>
>
> Does that sound feasible?
>
>
> On Mon, Oct 6, 2014 at 5:42 AM, Pavel V. Kaygorodov 
> wrote:
>
>> Hi!
>>
>> Our institute now planning to deploy a set of robotic telescopes across a
>> country.
>> Most of the telescopes will have low bandwidth and high latency, or even
>> not permanent internet connectivity.
>> I think, we can set up synchronization of observational data with ceph,
>> using federated gateways:
>>
>> 1. The main big storage ceph cluster will be set up in our institute main
>> building
>> 2. The small ceph clusters will be set up near each telescope, to store
>> only the data from local telescope
>> 3. VPN tunnels will be set up from each telescope site to our institute
>> 4. Federated gateways mechanism will do all the magic to synchronize data
>>
>> Is this a realistic plan?
>> What problems we can meet with this setup?
>>
>> Thanks in advance,
>>   Pavel.
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Federated gateways (our planning use case)

[Craig Lewis]

This sounds doable, with a few caveats.


Currently, replication is only one direction.  You can only write to the
primary zone, and you can read from the primary or secondary zones.  A
cluster can have many zones on it.

I'm thinking your setup would be a star topology.  Each telescope will be a
primary zone, and replicate to a secondary zone in the main storage
cluster.  The main cluster will have one read-only secondary zone for each
telescope.  If you have other needs to write data to the main cluster, you
can create another zone that only exists on the main cluster (possibly
replicated to one of the telescopes with a good network connection).

Each zone has it's own URL (primary and secondary), so you'd have a bit of
management problem remembering to use the correct URL. The URLs can be
whatever.  Convention follow's Amazon's naming scheme, but you'd probably
want to create your own scheme, something like
http://telescope_name-site.inasan.ru/ and
http://telescope_name-campus.inasan.ru/


You might have some problems with the replication if your VPN connections
aren't stable.  The replication agent isn't very tolerant of cluster
problem, so I suspect (but haven't tested) that long VPN outages will need
a replication agent restart.  For sites that don't have permanent
connections, just make the replication agent startup and shutdown part of
the connection startup and shutdown process.  Replication state is
available via a REST api, so it can be monitored.


I have tested large backlogs in replication.  When I initially imported my
data, I deliberately imported faster than I had bandwidth to replicate.  At
one point, my secondary cluster was ~10 million objects, and ~10TB behind
the primary cluster.  It eventually caught up, but the process doesn't
handle stops and restarts well.  Restarting the replication while it was
dealing with the backlog will start from the beginning of the backlog.
This can be a problem if your backlog is so large that it won't finish in a
day, because log rotation will restart the replication agent.  If that's
something you think might be a problem, I have some strategies to deal with
it, but they're manual and hacky.


Does that sound feasible?


On Mon, Oct 6, 2014 at 5:42 AM, Pavel V. Kaygorodov  wrote:

> Hi!
>
> Our institute now planning to deploy a set of robotic telescopes across a
> country.
> Most of the telescopes will have low bandwidth and high latency, or even
> not permanent internet connectivity.
> I think, we can set up synchronization of observational data with ceph,
> using federated gateways:
>
> 1. The main big storage ceph cluster will be set up in our institute main
> building
> 2. The small ceph clusters will be set up near each telescope, to store
> only the data from local telescope
> 3. VPN tunnels will be set up from each telescope site to our institute
> 4. Federated gateways mechanism will do all the magic to synchronize data
>
> Is this a realistic plan?
> What problems we can meet with this setup?
>
> Thanks in advance,
>   Pavel.
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Federated gateways (our planning use case)

[Pavel V. Kaygorodov]

Hi!

Our institute now planning to deploy a set of robotic telescopes across a 
country.
Most of the telescopes will have low bandwidth and high latency, or even not 
permanent internet connectivity.
I think, we can set up synchronization of observational data with ceph, using 
federated gateways:

1. The main big storage ceph cluster will be set up in our institute main 
building
2. The small ceph clusters will be set up near each telescope, to store only 
the data from local telescope
3. VPN tunnels will be set up from each telescope site to our institute
4. Federated gateways mechanism will do all the magic to synchronize data

Is this a realistic plan?
What problems we can meet with this setup?

Thanks in advance,
  Pavel.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Federated gateways

[Brian Andrus]

Those backslashes as output by radosgw-admin are escape characters
preceding the forward slash. They should be removed when you are connecting
with most clients. AFAIK, s3cmd would work fine with your original key, had
you stripped out the escape chars. You could also just regenerate or
specify a key without them.

Brian Andrus
Storage Consultant, Inktank


On Tue, Apr 15, 2014 at 9:45 AM, Craig Lewis wrote:

>  Also good to know that s3cmd does not handle those escapes correctly.
> Thanks!
>
>*Craig Lewis*
>  Senior Systems Engineer
> Office +1.714.602.1309
> Email cle...@centraldesktop.com
>
>  *Central Desktop. Work together in ways you never thought possible.*
>  Connect with us   Website   |  
> Twitter |
> Facebook   |  
> LinkedIn |
> Blog 
>   On 4/15/14 01:47 , Peter wrote:
>
> fixed! thank you for the reply. It was the backslashes in the secret that
> was the issue. I generated a new gateway user with:
>
> radosgw-admin user create --uid=test2 --display-name=test2
> --access-key={key} --secret={secret_without_slashes} --name
> client.radosgw.gateway
>
>
> and that worked.
>
> On 04/14/2014 09:57 PM, Craig Lewis wrote:
>
>
>   2014-04-14 12:39:20.556085 7f133f7ee700 10 auth_hdr:
> GET
>
>
>
> x-amz-date:Mon, 14 Apr 2014 11:39:01 +
> /
> 2014-04-14 12:39:20.556125 7f133f7ee700 15 *calculated
> digest=TQ5LP8ZeufSqKLumak6Aez4o+Pg=*
> 2014-04-14 12:39:20.556127 7f133f7ee700 15 *
> auth_sign=hx94rY3BJn7HQKA6ERaksNMQPRs=*
> 2014-04-14 12:39:20.556127 7f133f7ee700 15 compare=20
> 2014-04-14 12:39:20.556130 7f133f7ee700 10 *failed to authorize request*
> 2014-04-14 12:39:20.556167 7f133f7ee700  2 req 2:0.009095:s3:GET
> /:list_buckets:http status=403
> 2014-04-14 12:39:20.556396 7f133f7ee700  1 == req done req=0x8ca280
> http_status=403 ==
>
>
>  Did you create all of the rados pools that are mentioned in ceph.conf
> and the region and zone maps?
>
>
> The hash s3cmd computed is different that the one RGW computed.
>
> Can you verify that the access and secret keys in .s3cfg match the output
> of:
> radosgw-admin user info --uid=test1 --name client.radosgw.gateway
>
> Does the secret have a backslash (\) character in it?  The docs warn that
> not everything handles it well.  I regenerated my keys, rather than testing
> if s3cmd worked correctly.
>
>
>
>
>
> ___
> ceph-users mailing 
> listceph-us...@lists.ceph.comhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
> --
> Systems and Storage Engineer, Digital Repository of Ireland (DRI)
> High Performance & Research Computing, IS Services
> Lloyd Building, Trinity College Dublin, Dublin 2, 
> Ireland.http://www.tchpc.tcd.ie/ | ptier...@tchpc.tcd.ie
> Tel: +353-1-896-4466
>
>
>
> ___
> ceph-users mailing 
> listceph-us...@lists.ceph.comhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Federated gateways

[Craig Lewis]

Also good to know that s3cmd does not handle those escapes correctly.  
Thanks!


*Craig Lewis*
Senior Systems Engineer
Office +1.714.602.1309
Email cle...@centraldesktop.com 

*Central Desktop. Work together in ways you never thought possible.*
Connect with us Website   | Twitter 
  | Facebook 
  | LinkedIn 
  | Blog 



On 4/15/14 01:47 , Peter wrote:
fixed! thank you for the reply. It was the backslashes in the secret 
that was the issue. I generated a new gateway user with:


radosgw-admin user create --uid=test2 --display-name=test2 
--access-key={key} --secret={secret_without_slashes} --name 
client.radosgw.gateway


and that worked.

On 04/14/2014 09:57 PM, Craig Lewis wrote:



2014-04-14 12:39:20.556085 7f133f7ee700 10 auth_hdr:
GET



x-amz-date:Mon, 14 Apr 2014 11:39:01 +
/
2014-04-14 12:39:20.556125 7f133f7ee700 15 *calculated 
digest=TQ5LP8ZeufSqKLumak6Aez4o+Pg=*
2014-04-14 12:39:20.556127 7f133f7ee700 15 
*auth_sign=hx94rY3BJn7HQKA6ERaksNMQPRs=*

2014-04-14 12:39:20.556127 7f133f7ee700 15 compare=20
2014-04-14 12:39:20.556130 7f133f7ee700 10 *failed to authorize 
request*
2014-04-14 12:39:20.556167 7f133f7ee700  2 req 2:0.009095:s3:GET 
/:list_buckets:http status=403
2014-04-14 12:39:20.556396 7f133f7ee700  1 == req done 
req=0x8ca280 http_status=403 ==


Did you create all of the rados pools that are mentioned in ceph.conf 
and the region and zone maps?



The hash s3cmd computed is different that the one RGW computed.

Can you verify that the access and secret keys in .s3cfg match the 
output of:

radosgw-admin user info --uid=test1 --name client.radosgw.gateway

Does the secret have a backslash (\) character in it?  The docs warn 
that not everything handles it well.  I regenerated my keys, rather 
than testing if s3cmd worked correctly.






___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



--
Systems and Storage Engineer, Digital Repository of Ireland (DRI)
High Performance & Research Computing, IS Services
Lloyd Building, Trinity College Dublin, Dublin 2, Ireland.
http://www.tchpc.tcd.ie/  |ptier...@tchpc.tcd.ie
Tel: +353-1-896-4466


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Federated gateways

[Peter]

fixed! thank you for the reply. It was the backslashes in the secret 
that was the issue. I generated a new gateway user with:


radosgw-admin user create --uid=test2 --display-name=test2 
--access-key={key} --secret={secret_without_slashes} --name 
client.radosgw.gateway


and that worked.

On 04/14/2014 09:57 PM, Craig Lewis wrote:



2014-04-14 12:39:20.556085 7f133f7ee700 10 auth_hdr:
GET



x-amz-date:Mon, 14 Apr 2014 11:39:01 +
/
2014-04-14 12:39:20.556125 7f133f7ee700 15 *calculated 
digest=TQ5LP8ZeufSqKLumak6Aez4o+Pg=*
2014-04-14 12:39:20.556127 7f133f7ee700 15 
*auth_sign=hx94rY3BJn7HQKA6ERaksNMQPRs=*

2014-04-14 12:39:20.556127 7f133f7ee700 15 compare=20
2014-04-14 12:39:20.556130 7f133f7ee700 10 *failed to authorize request*
2014-04-14 12:39:20.556167 7f133f7ee700  2 req 2:0.009095:s3:GET 
/:list_buckets:http status=403
2014-04-14 12:39:20.556396 7f133f7ee700  1 == req done 
req=0x8ca280 http_status=403 ==


Did you create all of the rados pools that are mentioned in ceph.conf 
and the region and zone maps?



The hash s3cmd computed is different that the one RGW computed.

Can you verify that the access and secret keys in .s3cfg match the 
output of:

radosgw-admin user info --uid=test1 --name client.radosgw.gateway

Does the secret have a backslash (\) character in it?  The docs warn 
that not everything handles it well.  I regenerated my keys, rather 
than testing if s3cmd worked correctly.






___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



--
Systems and Storage Engineer, Digital Repository of Ireland (DRI)
High Performance & Research Computing, IS Services
Lloyd Building, Trinity College Dublin, Dublin 2, Ireland.
http://www.tchpc.tcd.ie/ | ptier...@tchpc.tcd.ie
Tel: +353-1-896-4466

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Federated gateways

[Craig Lewis]

2014-04-14 12:39:20.556085 7f133f7ee700 10 auth_hdr:
GET



x-amz-date:Mon, 14 Apr 2014 11:39:01 +
/
2014-04-14 12:39:20.556125 7f133f7ee700 15 *calculated 
digest=TQ5LP8ZeufSqKLumak6Aez4o+Pg=*
2014-04-14 12:39:20.556127 7f133f7ee700 15 
*auth_sign=hx94rY3BJn7HQKA6ERaksNMQPRs=*

2014-04-14 12:39:20.556127 7f133f7ee700 15 compare=20
2014-04-14 12:39:20.556130 7f133f7ee700 10 *failed to authorize request*
2014-04-14 12:39:20.556167 7f133f7ee700  2 req 2:0.009095:s3:GET 
/:list_buckets:http status=403
2014-04-14 12:39:20.556396 7f133f7ee700  1 == req done 
req=0x8ca280 http_status=403 ==


Did you create all of the rados pools that are mentioned in ceph.conf 
and the region and zone maps?



The hash s3cmd computed is different that the one RGW computed.

Can you verify that the access and secret keys in .s3cfg match the 
output of:

radosgw-admin user info --uid=test1 --name client.radosgw.gateway

Does the secret have a backslash (\) character in it?  The docs warn 
that not everything handles it well.  I regenerated my keys, rather than 
testing if s3cmd worked correctly.




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Federated gateways

[Peter]

Here is log output for request to gateway:


2014-04-14 12:39:20.547012 7f1377aa97c0 20 enqueued request req=0x8ca280
2014-04-14 12:39:20.547036 7f1377aa97c0 20 RGWWQ:
2014-04-14 12:39:20.547038 7f1377aa97c0 20 req: 0x8ca280
2014-04-14 12:39:20.547044 7f1377aa97c0 10 allocated request req=0x8a6d30
2014-04-14 12:39:20.547062 7f133f7ee700 20 dequeued request req=0x8ca280
2014-04-14 12:39:20.547066 7f133f7ee700 20 RGWWQ: empty
2014-04-14 12:39:20.547072 7f133f7ee700  1 == starting new request 
req=0x8ca280 =
2014-04-14 12:39:20.547171 7f133f7ee700  2 req 2:0.99::GET 
/::initializing
2014-04-14 12:39:20.547178 7f133f7ee700 10 host=cephgw.example.com 
rgw_dns_name=cephgw

2014-04-14 12:39:20.547190 7f133f7ee700 10 meta>> HTTP_X_AMZ_DATE
2014-04-14 12:39:20.547197 7f133f7ee700 10 x>> x-amz-date:Mon, 14 Apr 
2014 11:39:01 +
2014-04-14 12:39:20.547227 7f133f7ee700 10 s->object= 
s->bucket=

2014-04-14 12:39:20.547234 7f133f7ee700 20 FCGI_ROLE=RESPONDER
2014-04-14 12:39:20.547235 7f133f7ee700 20 SCRIPT_URL=/
2014-04-14 12:39:20.547235 7f133f7ee700 20 
SCRIPT_URI=http://cephgw.example.com
2014-04-14 12:39:20.547236 7f133f7ee700 20 HTTP_AUTHORIZATION=AWS 
718UJ2F9TFIEC5FT8XYU:hx94rY3BJn7HQKA6ERaksNMQPRs=

2014-04-14 12:39:20.547237 7f133f7ee700 20 HTTP_HOST=cephgw.example.com
2014-04-14 12:39:20.547237 7f133f7ee700 20 HTTP_ACCEPT_ENCODING=identity
2014-04-14 12:39:20.547238 7f133f7ee700 20 CONTENT_LENGTH=0
2014-04-14 12:39:20.547239 7f133f7ee700 20 HTTP_X_AMZ_DATE=Mon, 14 Apr 
2014 11:39:01 +
2014-04-14 12:39:20.547239 7f133f7ee700 20 HTTP_VIA=1. 
proxy.example.com.:3128 (squid/2.6.STABLE21)
2014-04-14 12:39:20.547240 7f133f7ee700 20 
HTTP_X_FORWARDED_FOR=10.10.10.10
2014-04-14 12:39:20.547241 7f133f7ee700 20 
HTTP_CACHE_CONTROL=max-age=259200

2014-04-14 12:39:20.547241 7f133f7ee700 20 HTTP_CONNECTION=keep-alive
2014-04-14 12:39:20.547242 7f133f7ee700 20 
PATH=/usr/local/bin:/usr/bin:/bin
2014-04-14 12:39:20.547243 7f133f7ee700 20 
SERVER_SIGNATURE=Apache/2.2.22 (Ubuntu) Server at 
cephgw.example.com Port 80


2014-04-14 12:39:20.547255 7f133f7ee700 20 
SERVER_SOFTWARE=Apache/2.2.22 (Ubuntu)

2014-04-14 12:39:20.547256 7f133f7ee700 20 SERVER_NAME=cephgw.example.com
2014-04-14 12:39:20.547257 7f133f7ee700 20 SERVER_ADDR=10.10.10.10
2014-04-14 12:39:20.547258 7f133f7ee700 20 SERVER_PORT=80
2014-04-14 12:39:20.547258 7f133f7ee700 20 REMOTE_ADDR=10.10.10.11
2014-04-14 12:39:20.547259 7f133f7ee700 20 DOCUMENT_ROOT=/var/www
2014-04-14 12:39:20.547260 7f133f7ee700 20 
SERVER_ADMIN=t...@cephgw.example.com
2014-04-14 12:39:20.547260 7f133f7ee700 20 
SCRIPT_FILENAME=/var/www/s3gw.fcgi

2014-04-14 12:39:20.547261 7f133f7ee700 20 REMOTE_PORT=51981
2014-04-14 12:39:20.547262 7f133f7ee700 20 GATEWAY_INTERFACE=CGI/1.1
2014-04-14 12:39:20.547262 7f133f7ee700 20 SERVER_PROTOCOL=HTTP/1.0
2014-04-14 12:39:20.547263 7f133f7ee700 20 REQUEST_METHOD=GET
2014-04-14 12:39:20.547264 7f133f7ee700 20 QUERY_STRING=page=¶ms=
2014-04-14 12:39:20.547264 7f133f7ee700 20 REQUEST_URI=/
2014-04-14 12:39:20.547265 7f133f7ee700 20 SCRIPT_NAME=/
2014-04-14 12:39:20.547266 7f133f7ee700  2 req 2:0.000195:s3:GET 
/::getting op
2014-04-14 12:39:20.547287 7f133f7ee700  2 req 2:0.000216:s3:GET 
/:list_buckets:authorizing
2014-04-14 12:39:20.547326 7f133f7ee700 20 get_obj_state: 
rctx=0x7f135c01d1a0 obj=.us-master.users:718UJ2F9TFIEC5FT8XYU 
state=0x7f135c01d268 s->prefetch_data=0
2014-04-14 12:39:20.547336 7f133f7ee700 10 moving 
.us-master.users+718UJ2F9TFIEC5FT8XYU to cache LRU end
2014-04-14 12:39:20.547339 7f133f7ee700 10 cache get: 
name=.us-master.users+718UJ2F9TFIEC5FT8XYU : type miss (requested=6, 
cached=3)
2014-04-14 12:39:20.554368 7f133f7ee700 10 cache put: 
name=.us-master.users+718UJ2F9TFIEC5FT8XYU
2014-04-14 12:39:20.554372 7f133f7ee700 10 moving 
.us-master.users+718UJ2F9TFIEC5FT8XYU to cache LRU end
2014-04-14 12:39:20.554381 7f133f7ee700 20 get_obj_state: s->obj_tag 
was set empty
2014-04-14 12:39:20.554390 7f133f7ee700 10 moving 
.us-master.users+718UJ2F9TFIEC5FT8XYU to cache LRU end
2014-04-14 12:39:20.554391 7f133f7ee700 10 cache get: 
name=.us-master.users+718UJ2F9TFIEC5FT8XYU : hit
2014-04-14 12:39:20.554445 7f133f7ee700 20 get_obj_state: 
rctx=0x7f135c01d580 obj=.us-master.users.uid:test state=0x7f135c01da58 
s->prefetch_data=0
2014-04-14 12:39:20.554451 7f133f7ee700 10 moving 
.us-master.users.uid+test to cache LRU end
2014-04-14 12:39:20.554452 7f133f7ee700 10 cache get: 
name=.us-master.users.uid+test : type miss (requested=6, cached=3)
2014-04-14 12:39:20.555975 7f133f7ee700 10 cache put: 
name=.us-master.users.uid+test
2014-04-14 12:39:20.555977 7f133f7ee700 10 moving 
.us-master.users.uid+test to cache LRU end
2014-04-14 12:39:20.555981 7f133f7ee700 20 get_obj_state: s->obj_tag 
was set empty
2014-04-14 12:39:20.555984 7f133f7ee700 10 moving 
.us-master.users.uid+test to cache LRU end
2014-04-14 12:39:20.555986 7f133f7ee700 10 cache get: 
name=.us-master.users.uid+test : hit

2014-04-14 12:39:20.556082 7f133f7

Re: [ceph-users] Federated gateways

[Peter Tiernan]

i have the following in ceph.conf:

[client.radosgw.gateway]
  host = cephgw
  keyring = /etc/ceph/keyring.radosgw.gateway
  rgw print continue = false
  rgw region = us
  rgw region root pool = .us.rgw.root
  rgw zone = us-master
  rgw zone root pool = .us-master.rgw.root
  rgw dns name = cephgw
  rgw socket path = /tmp/radosgw-us-master.sock
  log file = /var/log/ceph/radosgw-us-master.log
  rgw data = /var/lib/ceph/radosgw/ceph-radosgw.us-master-1

i run the following to create a gateway user on gateway node:

 radosgw-admin user create --uid=test1 --display-name=test --name 
client.radosgw.gateway


Then i setup s3cmd with the access and secret key, and still the same, 
'Access Denied'


On 04/12/2014 09:35 PM, Craig Lewis wrote:


On 4/11/14 02:36 , Peter wrote:

Hello,

I am testing out federated gateways. I have created one gateway with 
one region and one zone. The gateway appears to work. I am trying to 
test it with s3cmd before I continue with more regions and zones.


I create a test gateway user:

radosgw-admin user create --uid=test --display-name=test

but s3cmd keeps getting 'ERROR: S3 error: 403 (AccessDenied):'

i have tried specifying zone and region when creating test gateway user:

radosgw-admin user create --uid=test1 --rgw-zone=us-master 
--rgw-region=us --display-name=test


but i still get S3 403 error.

What am i doing wrong?

thanks

Try your radosgw-admin user create command again with 
--name=$radosgw_name .


$radosgw_name is whatever name you gave the radosgw agent in 
ceph.conf.  Mine is client.radosgw.ceph2, on the host named ceph2.





--
Systems and Storage Engineer, Digital Repository of Ireland (DRI)
High Performance & Research Computing, IS Services
Lloyd Building, Trinity College Dublin, Dublin 2, Ireland.
http://www.tchpc.tcd.ie/  |ptier...@tchpc.tcd.ie
Tel: +353-1-896-4466



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] federated gateways can sync files with English filename but can't sync files with Chinese filename

[wsnote]

Hi, everyone!
After serevel day's attempt, the files can sync between two zones.
But now, I find another quesion.Just files with English filename were synced, 
but files with Chinese filename were not synced.
Does anyone come with the same question?
Thanks!



 master zone radosgw's log following:
2014-04-14 15:37:00.775529 7fd54dde2700 10 moving 
.fj-xm.domain.rgw+.bucket.meta.test:fj-xm.4302.1 to cache LRU end
2014-04-14 15:37:00.775557 7fd54dde2700 10 cache get: 
name=.fj-xm.domain.rgw+.bucket.meta.test:fj-xm.4302.1 : type miss (requested=1
7, cached=22)
2014-04-14 15:37:00.775574 7fd54dde2700 20 get_obj_state: rctx=0x7fd580001b50 
obj=.fj-xm.domain.rgw:.bucket.meta.test:fj-xm.4302.1 s
tate=0x7fd58000d1d8 s->prefetch_data=0
2014-04-14 15:37:00.775580 7fd54dde2700 20 get_obj_state: rctx=0x7fd580001b50 
obj=.fj-xm.domain.rgw:.bucket.meta.test:fj-xm.4302.1 s
tate=0x7fd58000d1d8 s->prefetch_data=0
2014-04-14 15:37:00.775581 7fd54dde2700 20 state for 
obj=.fj-xm.domain.rgw:.bucket.meta.test:fj-xm.4302.1 is not atomic, not appendi
ng atomic test
2014-04-14 15:37:00.775612 7fd54dde2700 20 rados->read obj-ofs=0 read_ofs=0 
read_len=524288
2014-04-14 15:37:00.776528 7fd54dde2700 20 rados->read r=0 bl.length=153
2014-04-14 15:37:00.776581 7fd54dde2700 10 cache put: 
name=.fj-xm.domain.rgw+.bucket.meta.test:fj-xm.4302.1
2014-04-14 15:37:00.776595 7fd54dde2700 10 moving 
.fj-xm.domain.rgw+.bucket.meta.test:fj-xm.4302.1 to cache LRU end
2014-04-14 15:37:00.777375 7fd54dde2700 15 Read 
AccessControlPolicyhttp://s3.amazonaws.com/doc/2006-03-0
1/">testnametest 
namehttp://www.w
3.org/2001/XMLSchema-instance" 
xsi:type="Group">http://acs.amazonaws.com/groups/global/AllUsersREAD
http://www.w3.org/2001/XMLSchema-instance"; 
xsi:type="CanonicalUser">testname
test 
nameFULL_CONTROL
2014-04-14 15:37:00.777417 7fd54dde2700 20 get_obj_state: rctx=0x7fd580001b50 
obj=test:mytest/win7鏃楄埌婵€娲婚槄璇?txt state=0x7fd
58000ffb8 s->prefetch_data=1
2014-04-14 15:37:00.785713 7fd54dde2700 15 Read 
AccessControlPolicyhttp://s3.amazonaws.com/doc/2006-03-0
1/">testnametest 
namehttp://www.w
3.org/2001/XMLSchema-instance" 
xsi:type="Group">http://acs.amazonaws.com/groups/global/AllUsersREAD
http://www.w3.org/2001/XMLSchema-instance"; 
xsi:type="CanonicalUser">testname
test 
nameFULL_CONTROL
2014-04-14 15:37:00.785923 7fd54dde2700 10 read_permissions on 
test(@{i=.fj-xm.rgw.buckets.index}.fj-xm.rgw.buckets[fj-xm.4302.1]):m
ytest/win7鏃楄埌婵€娲婚槄璇?txt only_bucket=0 ret=-2
2014-04-14 15:37:00.785993 7fd54dde2700  2 req 2:0.063973:s3:HEAD 
/test/mytest%2Fwin7%E6%97%97%E8%88%B0%E6%BF%80%E6%B4%BB%E9%98%85%E
8%AF%BB.txt:get_obj:http status=404
2014-04-14 15:37:00.786126 7fd54dde2700  1
 == req done req=0x2527870 http_status=404 ==


 slave zone radosgw's log following:
Mon Apr 14 07:37:53 2014
x-amz-copy-source:test/mytest/win7%E6%97%97%E8%88%B0%E6%BF%80%E6%B4%BB%E9%98%85%E8%AF%BB.txt
/test/mytest%2Fwin7%FFE6%FF97%FF97%FFE8%FF88%FFB0%FFE6%FFBF%FF80%FFE6%FFB4%FFBB%
FFE9%FF98%FF85%FFE8%FFAF%FFBB.txt
2014-04-14 15:37:53.713088 7f460a3e9700 15 generated auth header: AWS 
UWN9EF6XJ8E9KUQX4XAE:jRhG5XJ6va5nFFmXZK4inGLTO4A=
2014-04-14 15:37:53.722011 7f460a3e9700 20 sending request to 
http://s3.ceph69.com:80/test/mytest%2Fwin7%FFE6%FF97%FF97%
FFE8%FF88%FFB0%FFE6%FFBF%FF80%FFE6%FFB4%FFBB%FFE9%FF98%FF85%FFE8%FFAF%FF
BB.txt?rgwx-uid=fj-xm&rgwx-region=fj&rgwx-prepend-metadata=fj
2014-04-14 15:37:53.863186 7f460a3e9700 10 receive_http_header
2014-04-14 15:37:53.863215 7f460a3e9700 10 received header:HTTP/1.1 400 Bad 
Request
2014-04-14 15:37:53.863219 7f460a3e9700 10 receive_http_header
2014-04-14 15:37:53.863220 7f460a3e9700 10 received header:Date: Mon, 14 Apr 
2014 07:37:51 GMT
2014-04-14 15:37:53.863240 7f460a3e9700 10 receive_http_header
2014-04-14 15:37:53.863241 7f460a3e9700 10 received header:Server: 
Apache/2.2.22 (Fedora)
2014-04-14 15:37:53.863244 7f460a3e9700 10 receive_http_header
2014-04-14 15:37:53.863244 7f460a3e9700 10 received header:Accept-Ranges: bytes
2014-04-14 15:37:53.872057 7f460a3e9700 10 receive_http_header
2014-04-14 15:37:53.872272 7f460a3e9700 10 received header:Content-Length: 83
2014-04-14 15:37:53.872285 7f460a3e9700 10 receive_http_header
2014-04-14 15:37:53.872286 7f460a3e9700 10 received header:Connection: close
2014-04-14 15:37:53.872290 7f460a3e9700 10 receive_http_header
2014-04-14 15:37:53.872291 7f460a3e9700 10 received header:Content-Type: 
application/xml
2014-04-14 15:37:53.872293 7f460a3e9700 10 receive_http_header
2014-04-14 15:37:53.872294 7f460a3e9700 10 received header:
2014-04-14 15:37:53.875450 7f460a3e9700  2 req 463:0.273254:s3:PUT 
/test/mytest/win7%25E6%2597%2597%25E8%2588%25B0%25E6%25BF%2580%25
E6%25B4%25BB%25E9%2598%2585%25E8%25AF%25BB.txt:copy_obj:http status=400
201

[ceph-users] Federated gateways

[Peter]

Hello,

I am testing out federated gateways. I have created one gateway with one 
region and one zone. The gateway appears to work. I am trying to test it 
with s3cmd before I continue with more regions and zones.


I create a test gateway user:

radosgw-admin user create --uid=test --display-name=test

but s3cmd keeps getting 'ERROR: S3 error: 403 (AccessDenied):'

i have tried specifying zone and region when creating test gateway user:

radosgw-admin user create --uid=test1 --rgw-zone=us-master 
--rgw-region=us --display-name=test


but i still get S3 403 error.

What am i doing wrong?

thanks


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com