Increasing the logging further, and I notice the following: 2014-03-13 00:27:28.617100 7f6036ffd700 20 rgw_create_bucket returned ret=-1 bucket=test(@.rgw.buckets[us-west-1.15849318.1])
But hope that .rgw.buckets doesn't have to exist... and that @.rgw.buckets is perhaps telling of something? I did notice that .us-west-1.rgw.buckets and .us-west-1.rgw.buckets.index weren't created. I created those, restarted radosgw, and still 403 errors. On Wed, Mar 12, 2014 at 8:00 PM, Greg Poirier <greg.poir...@opower.com>wrote: > And the debug log because that last log was obviously not helpful... > > 2014-03-12 23:57:49.497780 7ff97e7dd700 1 ====== starting new request > req=0x23bc650 ===== > 2014-03-12 23:57:49.498198 7ff97e7dd700 2 req 1:0.000419::PUT > /test::initializing > 2014-03-12 23:57:49.498233 7ff97e7dd700 10 > host=s3.amazonaws.comrgw_dns_name=us-west-1.domain > 2014-03-12 23:57:49.498366 7ff97e7dd700 10 s->object=<NULL> s->bucket=test > 2014-03-12 23:57:49.498437 7ff97e7dd700 2 req 1:0.000659:s3:PUT > /test::getting op > 2014-03-12 23:57:49.498448 7ff97e7dd700 2 req 1:0.000670:s3:PUT > /test:create_bucket:authorizing > 2014-03-12 23:57:49.498508 7ff97e7dd700 10 cache get: > name=.us-west-1.users+BLAHBLAHBLAH : miss > 2014-03-12 23:57:49.500852 7ff97e7dd700 10 cache put: > name=.us-west-1.users+BLAHBLAHBLAH > 2014-03-12 23:57:49.500865 7ff97e7dd700 10 adding > .us-west-1.users+BLAHBLAHBLAH to cache LRU end > 2014-03-12 23:57:49.500886 7ff97e7dd700 10 moving > .us-west-1.users+BLAHBLAHBLAH to cache LRU end > 2014-03-12 23:57:49.500889 7ff97e7dd700 10 cache get: > name=.us-west-1.users+BLAHBLAHBLAH : type miss (requested=1, cached=6) > 2014-03-12 23:57:49.500907 7ff97e7dd700 10 moving > .us-west-1.users+BLAHBLAHBLAH to cache LRU end > 2014-03-12 23:57:49.500910 7ff97e7dd700 10 cache get: > name=.us-west-1.users+BLAHBLAHBLAH : hit > 2014-03-12 23:57:49.502663 7ff97e7dd700 10 cache put: > name=.us-west-1.users+BLAHBLAHBLAH > 2014-03-12 23:57:49.502667 7ff97e7dd700 10 moving > .us-west-1.users+BLAHBLAHBLAH to cache LRU end > 2014-03-12 23:57:49.502700 7ff97e7dd700 10 cache get: > name=.us-west-1.users.uid+test : miss > 2014-03-12 23:57:49.505128 7ff97e7dd700 10 cache put: > name=.us-west-1.users.uid+test > 2014-03-12 23:57:49.505138 7ff97e7dd700 10 adding > .us-west-1.users.uid+test to cache LRU end > 2014-03-12 23:57:49.505157 7ff97e7dd700 10 moving > .us-west-1.users.uid+test to cache LRU end > 2014-03-12 23:57:49.505160 7ff97e7dd700 10 cache get: > name=.us-west-1.users.uid+test : type miss (requested=1, cached=6) > 2014-03-12 23:57:49.505176 7ff97e7dd700 10 moving > .us-west-1.users.uid+test to cache LRU end > 2014-03-12 23:57:49.505178 7ff97e7dd700 10 cache get: > name=.us-west-1.users.uid+test : hit > 2014-03-12 23:57:49.507401 7ff97e7dd700 10 cache put: > name=.us-west-1.users.uid+test > 2014-03-12 23:57:49.507406 7ff97e7dd700 10 moving > .us-west-1.users.uid+test to cache LRU end > 2014-03-12 23:57:49.507521 7ff97e7dd700 10 get_canon_resource(): dest=/test > 2014-03-12 23:57:49.507529 7ff97e7dd700 10 auth_hdr: > PUT > > binary/octet-stream > Wed, 12 Mar 2014 23:57:51 GMT > /test > 2014-03-12 23:57:49.507674 7ff97e7dd700 2 req 1:0.009895:s3:PUT > /test:create_bucket:reading permissions > 2014-03-12 23:57:49.507682 7ff97e7dd700 2 req 1:0.009904:s3:PUT > /test:create_bucket:verifying op mask > 2014-03-12 23:57:49.507695 7ff97e7dd700 2 req 1:0.009917:s3:PUT > /test:create_bucket:verifying op permissions > 2014-03-12 23:57:49.509604 7ff97e7dd700 2 req 1:0.011826:s3:PUT > /test:create_bucket:verifying op params > 2014-03-12 23:57:49.509615 7ff97e7dd700 2 req 1:0.011836:s3:PUT > /test:create_bucket:executing > 2014-03-12 23:57:49.509694 7ff97e7dd700 10 cache get: > name=.us-west-1.domain.rgw+test : miss > 2014-03-12 23:57:49.512229 7ff97e7dd700 10 cache put: > name=.us-west-1.domain.rgw+test > 2014-03-12 23:57:49.512259 7ff97e7dd700 10 adding > .us-west-1.domain.rgw+test to cache LRU end > 2014-03-12 23:57:49.512333 7ff97e7dd700 10 cache get: > name=.us-west-1.domain.rgw+.pools.avail : miss > 2014-03-12 23:57:49.518216 7ff97e7dd700 10 cache put: > name=.us-west-1.domain.rgw+.pools.avail > 2014-03-12 23:57:49.518228 7ff97e7dd700 10 adding > .us-west-1.domain.rgw+.pools.avail to cache LRU end > 2014-03-12 23:57:49.518248 7ff97e7dd700 10 moving > .us-west-1.domain.rgw+.pools.avail to cache LRU end > 2014-03-12 23:57:49.518251 7ff97e7dd700 10 cache get: > name=.us-west-1.domain.rgw+.pools.avail : type miss (requested=1, cached=6) > 2014-03-12 23:57:49.518270 7ff97e7dd700 10 moving > .us-west-1.domain.rgw+.pools.avail to cache LRU end > 2014-03-12 23:57:49.518272 7ff97e7dd700 10 cache get: > name=.us-west-1.domain.rgw+.pools.avail : hit > 2014-03-12 23:57:49.520295 7ff97e7dd700 10 cache put: > name=.us-west-1.domain.rgw+.pools.avail > 2014-03-12 23:57:49.520348 7ff97e7dd700 10 moving > .us-west-1.domain.rgw+.pools.avail to cache LRU end > 2014-03-12 23:57:49.522672 7ff97e7dd700 2 req 1:0.024893:s3:PUT > /test:create_bucket:http status=403 > 2014-03-12 23:57:49.523204 7ff97e7dd700 1 ====== req done req=0x23bc650 > http_status=403 ====== > > > On Wed, Mar 12, 2014 at 7:36 PM, Greg Poirier <greg.poir...@opower.com>wrote: > >> The saga continues... >> >> So, after fiddling with haproxy a bit, I managed to make sure that my >> requests were hitting the RADOS Gateway. >> >> NOW, I get a 403 from my ruby script: >> >> 2014-03-12 23:34:08.289670 7fda9bfbf700 1 ====== starting new request >> req=0x215a780 ===== >> 2014-03-12 23:34:08.305105 7fda9bfbf700 1 ====== req done req=0x215a780 >> http_status=403 ====== >> >> The aws-s3 gem forces the Host header to be set to s3.amazonaws.com -- >> and I am wondering if this could potentially cause a problem. Is that the >> case? Or is radosgw able to just know where it is supposed to look for >> objects based on its configuration? I assume the latter, otherwise we would >> have multi-tenant radosgw instead of needing to have an instance per zone. >> >> So does it ignore the Host header? >> >> Is this 403 related to my problems with the user error I found earlier >> where I'm unable to view the user with the radosgw-admin tool? >> >> >> >> On Wed, Mar 12, 2014 at 1:54 PM, Greg Poirier <greg.poir...@opower.com>wrote: >> >>> Also... what are linger_ops? >>> >>> ceph --admin-daemon /var/run/ceph/ceph-client.radosgw.<hostname>.asok >>> objecter_requests >>> { "ops": [], >>> "linger_ops": [ >>> { "linger_id": 1, >>> "pg": "7.4322fa9f", >>> "osd": 25, >>> "object_id": "notify.0", >>> "object_locator": "@7", >>> "snapid": "head", >>> "registering": "head", >>> "registered": "head"}, >>> { "linger_id": 2, >>> "pg": "7.16dafda0", >>> "osd": 132, >>> "object_id": "notify.1", >>> "object_locator": "@7", >>> "snapid": "head", >>> "registering": "head", >>> "registered": "head"}, >>> { "linger_id": 3, >>> "pg": "7.88aa5c95", >>> "osd": 32, >>> "object_id": "notify.2", >>> "object_locator": "@7", >>> "snapid": "head", >>> "registering": "head", >>> "registered": "head"}, >>> { "linger_id": 4, >>> "pg": "7.f8c99aee", >>> "osd": 62, >>> "object_id": "notify.3", >>> "object_locator": "@7", >>> "snapid": "head", >>> "registering": "head", >>> "registered": "head"}, >>> { "linger_id": 5, >>> "pg": "7.a204812d", >>> "osd": 129, >>> "object_id": "notify.4", >>> "object_locator": "@7", >>> "snapid": "head", >>> "registering": "head", >>> "registered": "head"}, >>> { "linger_id": 6, >>> "pg": "7.31099063", >>> "osd": 28, >>> "object_id": "notify.5", >>> "object_locator": "@7", >>> "snapid": "head", >>> "registering": "head", >>> "registered": "head"}, >>> { "linger_id": 7, >>> "pg": "7.97c520d4", >>> "osd": 135, >>> "object_id": "notify.6", >>> "object_locator": "@7", >>> "snapid": "head", >>> "registering": "head", >>> "registered": "head"}, >>> { "linger_id": 8, >>> "pg": "7.84ada7c9", >>> "osd": 94, >>> "object_id": "notify.7", >>> "object_locator": "@7", >>> "snapid": "head", >>> "registering": "head", >>> "registered": "head"}], >>> "pool_ops": [], >>> "pool_stat_ops": [], >>> "statfs_ops": [], >>> "command_ops": []} >>> >>> >>> On Wed, Mar 12, 2014 at 10:45 AM, Greg Poirier >>> <greg.poir...@opower.com>wrote: >>> >>>> Rados GW and Ceph versions installed: >>>> Version: 0.67.7-1precise >>>> >>>> I create a user: >>>> radosgw-admin --name client.radosgw.<hostname> user create --uid test >>>> --display-name "Test User" >>>> >>>> It outputs some JSON that looks convincing: >>>> { "user_id": "test", >>>> "display_name": "test user", >>>> "email": "", >>>> "suspended": 0, >>>> "max_buckets": 1000, >>>> "auid": 0, >>>> "subusers": [], >>>> "keys": [ >>>> { "user": "test", >>>> "access_key": "<snip>", >>>> "secret_key": "<snip>"}, >>>> { "user": "test", >>>> "access_key": "<snip>", >>>> "secret_key": "<snip>"}], >>>> "swift_keys": [], >>>> "caps": [], >>>> "op_mask": "read, write, delete", >>>> "default_placement": "", >>>> "placement_tags": []} >>>> >>>> There are two keys because I have tried this twice. >>>> >>>> I can see it in metadata list: >>>> radosgw-admin --name client.radosgw.<hostname> metadata list user >>>> [ >>>> "test", >>>> "us-east-2", >>>> "us-west-1"] >>>> >>>> I then try to get user info: >>>> >>>> radosgw-admin --name client.radosgw.<hostname> user info test >>>> could not fetch user info: no user info saved >>>> >>>> I try to create a bucket with the user using Ruby's aws/s3 API: >>>> >>>> equire 'aws/s3' >>>> >>>> AWS::S3::Base.establish_connection!( >>>> access_key_id: '<snip>', >>>> secret_access_key: '<snip>', >>>> use_ssl: true, >>>> server: '<snip>', >>>> persistent: true >>>> ) >>>> >>>> AWS::S3::Bucket.create('test') >>>> >>>> file = 'sloth.txt' >>>> >>>> AWS::S3::S3Object.store(file, open(file), 'test') >>>> >>>> bucket = AWS::S3::Bucket.find('test') >>>> >>>> puts bucket >>>> >>>> bucket.each do |object| >>>> puts >>>> "#{object.key}\t#{object.about['content-length']}\t#{object.about['last-modified']}" >>>> end >>>> >>>> And I get the following: >>>> #<AWS::S3::Bucket:0x007f9852854e58> >>>> /Users/greg.poirier/.rvm/gems/ruby-1.9.3-p429/gems/aws-s3-0.6.3/lib/aws/s3/base.rb:235:in >>>> `method_missing': undefined local variable or method `name' for >>>> #<AWS::S3::Bucket:0x007f9852854e58> (NameError) >>>> from >>>> /Users/greg.poirier/.rvm/gems/ruby-1.9.3-p429/gems/aws-s3-0.6.3/lib/aws/s3/bucket.rb:313:in >>>> `reload!' >>>> from >>>> /Users/greg.poirier/.rvm/gems/ruby-1.9.3-p429/gems/aws-s3-0.6.3/lib/aws/s3/bucket.rb:242:in >>>> `objects' >>>> from >>>> /Users/greg.poirier/.rvm/gems/ruby-1.9.3-p429/gems/aws-s3-0.6.3/lib/aws/s3/bucket.rb:253:in >>>> `each' >>>> from test.rb:21:in `<main>' >>>> >>>> The bucket fails to be created: >>>> >>>> radosgw-admin --name client.radosgw.<hostname> bucket list >>>> [] >>>> >>>> And also this: >>>> >>>> radosgw-admin --name client.radosgw.<hostname> metadata list bucket >>>> []2014-03-12 17:42:42.221112 7f426b779780 -1 failed to list objects >>>> pool_iterate returned r=-2 >>>> >>>> >>>> So clearly there is something going on here. My questions: >>>> >>>> Is this failure to create a bucket related to do the "no user info >>>> saved" error? >>>> >>>> What would cause the "no user info saved" error? >>>> >>>> What may be causing the bucket to not be created? >>>> >>>> >>>> >>> >> >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com