An outbound handoff of partition error enotconn
Hi Riak-users, I have found some errors related to handoff of partition in /etc/riak/log/errors. Details are as below: 2015-07-30 16:04:33.643 [error] <0.12872.15>@riak_core_handoff_sender:start_fold:262 ownership_transfer transfer of riak_kv_vnode from 'riak@10.21.136.76' 45671926166590716193865151022383844364247891968 to 'riak@10.21.136.93' 45671926166590716193865151022383844364247891968 failed because of enotconn 2015-07-30 16:04:33.643 [error] <0.197.0>@riak_core_handoff_manager:handle_info:289 An outbound handoff of partition riak_kv_vnode 45671926166590716193865151022383844364247891968 was terminated for reason: {shutdown,{error,enotconn}} I have searched it with google and found related articles. However, there's no solution. http://lists.basho.com/pipermail/riak-users_lists.basho.com/2014-October/016052.html -- Amao Wang Best & Regards ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Fwd: riak-users post from wang.chang...@gmail.com requires approval
-- Forwarded message -- From: changmao wang Date: Tue, Aug 4, 2015 at 6:37 PM Subject: Re: riak-users post from wang.chang...@gmail.com requires approval To: Matthew Brender Hi riak-users, Thanks for your reply. However, I have some problems with my riak cluster. It's pending on leaving status for 7 days after I removed 4 new added nodes. root@cluster-s3-hd1:~# riak-admin member-status = Membership == Status RingPendingNode --- leaving10.9% 10.9%'riak@10.21.136.91' leaving 9.4% 10.9%'riak@10.21.136.92' leaving 7.8% 10.9%'riak@10.21.136.93' leaving 7.8% 10.9%'riak@10.21.136.94' valid 10.9% 10.9%'riak@10.21.136.66' valid 10.9% 10.9%'riak@10.21.136.71' valid 14.1% 10.9%'riak@10.21.136.76' valid 17.2% 12.5%'riak@10.21.136.81' valid 10.9% 10.9%'riak@10.21.136.86' below is error log from riak@10.21.136.81 2015-08-04 01:04:03.883 [error] <0.27021.262>@riak_core_handoff_sender:start_fold:262 ownership_transfer transfer of riak_kv_vnode from 'riak@10.21.136.81' 296867520082839655260123481645494988367611297792 to 'riak@10.21.136.92' 296867520082839655260123481645494988367611297792 failed because of enotconn 2015-08-04 01:04:03.883 [error] <0.195.0>@riak_core_handoff_manager:handle_info:289 An outbound handoff of partition riak_kv_vnode 296867520082839655260123481645494988367611297792 was terminated for reason: {shutdown,{error,enotconn}} 2015-08-04 04:38:39.512 [error] <0.15702.1> Trailing data, discarding (2753 bytes) 2015-08-04 08:06:39.080 [error] <0.26581.272>@riak_core_handoff_sender:start_fold:262 ownership_transfer transfer of riak_kv_vnode from 'riak@10.21.136.81' 411047335499316445744786359201454599278231027712 to 'riak@10.21.136.93' 411047335499316445744786359201454599278231027712 failed because of enotconn 2015-08-04 08:06:39.081 [error] <0.195.0>@riak_core_handoff_manager:handle_info:289 An outbound handoff of partition riak_kv_vnode 411047335499316445744786359201454599278231027712 was terminated for reason: {shutdown,{error,enotconn}} 2015-08-04 18:07:53.040 [error] <0.6973.342>@riak_api_pb_server:handle_info:141 Unrecognized message {63685090,{error,timeout}} Today I detached the network bond(we used the bond mode 6/0) in several nodes of the riak cluster. After that, "ifconfig" command show decreased "dropped" packages, There're only 1~2 dropped network packages during the last four hours. However, I want to know what's the status of the Riak cluster? Currently I started riak-cs service on four new nodes. That's ok, they can service. how many hours those four node will take to leave the cluster? On Tue, Aug 4, 2015 at 4:54 AM, Matthew Brender wrote: > Hey Amao, > > Excuse the delay on this message being posted. You hit our upper limit on > size (see below). When possible, please link to a gist or another external > service to prevent this from being filtered. > > Thanks for your patience! > Matt > > *Matt Brender | Developer Advocacy Lead* > Basho Technologies > t: @mjbrender <https://twitter.com/mjbrender> > > > On Thu, Jul 30, 2015 at 5:35 AM, wrote: > >> As list administrator, your authorization is requested for the >> following mailing list posting: >> >> List:riak-users@lists.basho.com >> From:wang.chang...@gmail.com >> Subject: riak handoff of partition error >> Reason: Message body is too big: 3105110 bytes with a limit of 40 KB >> >> At your convenience, visit: >> >> http://lists.basho.com/mailman/admindb/riak-users_lists.basho.com >> >> to approve or deny the request. >> >> >> -- Forwarded message -- >> From: changmao wang >> To: riak-users@lists.basho.com >> Cc: >> Date: Thu, 30 Jul 2015 17:35:13 +0800 >> Subject: riak handoff of partition error >> Hi Riak-users group, >> >> I have found some errors related to handoff of partition in >> /etc/riak/log/errors. >> Details are as below: >> >> 2015-07-30 16:04:33.643 [error] >> <0.12872.15>@riak_core_handoff_sender:start_fold:262 ownership_transfer >> transfer of riak_kv_vnode from 'riak@10.21.136.76' >> 45671926166590716193865151022383844364247891968 to 'riak@10.21.136.93' >> 45671926166590716193865151022383844364247891968 failed because of enotconn >> 2015-07-30 16:04:33.643 [error] >> <0.197.0>@riak_core_ha
Re: why leaving riak cluster so slowly and how to accelerate the speed
Is there any ideas to fix this issue? Amao On Fri, Aug 7, 2015 at 6:55 AM, changmao wang wrote: > Dmitri, > > Thanks for your quick reply. > my question are as below: > 1. what's the current status of the whole cluster? Is't doing data balance? > 2. there's so many errors during one of the node error log. how to handle > it? > 2015-08-05 01:38:59.717 [error] > <0.23000.298>@riak_core_handoff_sender:start_fold:262 ownership_transfer > transfer of riak_kv_vnode from 'riak@10.21.136.81' > 525227150915793236229449236757414210188850757632 to 'riak@10.21.136.94' > 525227150915793236229449236757414210188850757632 failed because of enotconn > 2015-08-05 01:38:59.718 [error] > <0.195.0>@riak_core_handoff_manager:handle_info:289 An outbound handoff of > partition riak_kv_vnode 525227150915793236229449236757414210188850757632 > was terminated for reason: {shutdown,{error,enotconn}} > > During the last 5 days, there's no changes of the "riak-admin member > status" output. > 3. how to accelerate the data balance? > > > On Fri, Aug 7, 2015 at 6:41 AM, Dmitri Zagidulin > wrote: > >> Ok, I think I understand so far. So what's the question? >> >> On Thursday, August 6, 2015, Changmao.Wang >> wrote: >> >>> Hi Riak users, >>> >>> Before adding new nodes, the cluster only have five nodes. The member >>> list are as below: >>> 10.21.136.66,10.21.136.71,10.21.136.76,10.21.136.81,10.21.136.86. >>> We did not setup http proxy for the cluster, only one node of the >>> cluster provide the http service. so the CPU load is always high on this >>> node. >>> >>> After that, I added four nodes (10.21.136.[91-94]) to those cluster. >>> During the ring/data balance progress, each node failed(riak stopped) >>> because of disk 100% full. >>> I used multi-disk path to "data_root" parameter in >>> '/etc/riak/app.config'. Each disk is only 580MB size. >>> As you know, bitcask storage engine did not support multi-disk path. >>> After one of the disks is 100% full, it can not switch next idle disk. So >>> the "riak" service is down. >>> >>> After that, I removed the new add four nodes at active nodes with >>> "riak-admin cluster leave riak@'10.21.136.91'". >>> and then stop "riak" service on other active new nodes, reformat the >>> above new nodes with LVM disk management (bind 6 disk with virtual disk >>> group). >>> Replace the "data-root" parameter with one folder, and then start "riak" >>> service again. After that, the cluster began the data balance again. >>> That's the whole story. >>> >>> >>> Amao >>> >>> -- >>> *From: *"Dmitri Zagidulin" >>> *To: *"Changmao.Wang" >>> *Sent: *Thursday, August 6, 2015 10:46:59 PM >>> *Subject: *Re: why leaving riak cluster so slowly and how to accelerate >>> the speed >>> >>> Hi Amao, >>> >>> Can you explain a bit more which steps you've taken, and what the >>> problem is? >>> >>> Which nodes have been added, and which nodes are leaving the cluster? >>> >>> On Tue, Jul 28, 2015 at 11:03 PM, Changmao.Wang < >>> changmao.w...@datayes.com> wrote: >>> >>>> Hi Raik user group, >>>> >>>> I'm using riak and riak-cs 1.4.2. Last weekend, I added four nodes to >>>> cluster with 5 nodes. However, it's failed with one of disks 100% full. >>>> As you know bitcask storage engine can not support multifolders. >>>> >>>> After that, I restarted the "riak" and leave the cluster with the >>>> command "riak-admin cluster leave" and "riak-admin cluster plan", and the >>>> commit. >>>> However, riak is always doing KV balance after my submit leaving >>>> command. I guess that it's doing join cluster progress. >>>> >>>> Could you show us how to accelerate the leaving progress? I have tuned >>>> the "transfer-limit" parameters on 9 nodes. >>>> >>>> below is some commands output: >>>> riak-admin member-status >>>> = Membership >>>> == >>>> Status RingPendingNode >>>> >>>> ---
riak-admin diag errors
Hi Riak-users, when I run riak-admin diag command, sometimes it's ok, sometimes it found run time errors like below: root@cluster1-hd13:~# riak-admin diag RPC to 'riak@10.21.136.81' failed: {'EXIT', {function_clause, [{lists,zip, [['riak@10.21.136.94'],[]], [{file,"lists.erl"},{line,321}]}, {lists,zip,2, [{file,"lists.erl"},{line,321}]}, {lists,zip,2, [{file,"lists.erl"},{line,321}]}, {riaknostic_check_search,check,0, [{file, "src/riaknostic_check_search.erl"}, {line,49}]}, {riaknostic_check,check,1, [{file,"src/riaknostic_check.erl"}, {line,74}]}, {riaknostic,'-run/1-fun-1-',2, [{file,"src/riaknostic.erl"}, {line,106}]}, {lists,foldl,3, [{file,"lists.erl"},{line,1197}]}, {riaknostic,run,1, [{file,"src/riaknostic.erl"}, {line,105}]}]}} By the way, I used riak 1.4.2 version on ubuntu with 9 nodes. (four nodes have been added last month). Any more background, do u want to know? -- Amao Wang Best & Regards ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: why leaving riak cluster so slowly and how to accelerate the speed
1. About backuping new nodes of four and then using 'riak-admin force-replace'. what's the status of new added nodes? as you know, we want to replace one of leaving nodes. 2. what's the risk of 'riak-admin force-remove' 'riak@10.21.136.91' without backup? As you know, now the node(riak@10.21.136.91) is a member of the cluster, and keeping almost 2.5TB data, maybe 10 percent of the whole cluster. On Tue, Aug 11, 2015 at 7:32 PM, Dmitri Zagidulin wrote: > 1. How to force leave "leaving"'s nodes without data loss? > > This depends on - did you back up the data directory of the 4 new nodes, > before you reformatted them? > If you backed them up (and then restored the data directory once you > reformatted them), you can try: > > riak-admin force-replace 'riak@10.21.136.91' 'riak@ address is for that node>' > (same for the other 3) > > If you did not back up those nodes, the only thing you can do is force > them to leave, and then join the new ones. So, for each of the 4: > > riak-admin force-remove 'riak@10.21.136.91' 'riak@10.21.136.66' > (same for the other 3) > > In either case, after force-replacing or force-removing, you have to join > the new nodes to the cluster, before you commit. > > riak-admin join 'riak@new node' 'riak@10.21.136.66' > (same for the other 3) > and finally: > riak-cluster plan > riak-cluster commit > > As for the error, the reason you're seeing it, is because the other nodes > can't contact the 4 that are supposed to be leaving. (Since you wiped them). > The amount of time that passed doesn't matter, the cluster will be waiting > for those nodes to leave indefinitely, unless you force-remove or > force-replace. > > > > On Tue, Aug 11, 2015 at 1:32 AM, changmao wang > wrote: > >> HI Dmitri, >> >> For your question, >> 3) Re-formatted those four nodes and re-installed Riak. Here is where it >> gets tricky though. Several questions for you: >> - Did you attempt to re-join those 4 reinstalled nodes into the cluster? >> What was the output of the cluster join and cluster plan commands? >> - Did the IP address change, after they were reformatted? If so, you >> probably need to use something like 'reip' at this point: >> http://docs.basho.com/riak/latest/ops/running/tools/riak-admin/#reip >> >> I did NOT try to re-join those 4 re-join those 4 reinstalled nodes into >> the cluster. As you know, member-status shows 'they're leaving" as below: >> riak-admin member-status >> = Membership >> == >> Status RingPendingNode >> >> --- >> leaving10.9% 10.9%'riak@10.21.136.91' >> leaving 9.4% 10.9%'riak@10.21.136.92' >> leaving 7.8% 10.9%'riak@10.21.136.93' >> leaving 7.8% 10.9%'riak@10.21.136.94' >> valid 10.9% 10.9%'riak@10.21.136.66' >> valid 10.9% 10.9%'riak@10.21.136.71' >> valid 14.1% 10.9%'riak@10.21.136.76' >> valid 17.2% 12.5%'riak@10.21.136.81' >> valid 10.9% 10.9%'riak@10.21.136.86' >> >> --- >> Valid:5 / Leaving:4 / Exiting:0 / Joining:0 / Down:0 >> >> two weeks elapsed, 'riak-admin member-status' shows same result. I don't >> know which step ring hand off? >> >> I did not changed the IP address of four newly adding nodes. >> >> My questions: >> >> 1. How to force leave "leaving"'s nodes without data loss? >> 2. I have found some errors related to handoff of partition in >> /etc/riak/log/errors. >> Details are as below: >> >> 2015-07-30 16:04:33.643 [error] >> <0.12872.15>@riak_core_handoff_sender:start_fold:262 ownership_transfer >> transfer of riak_kv_vnode from 'riak@10.21.136.76' >> 45671926166590716193865151022383844364247891968 to 'riak@10.21.136.93' >> 45671926166590716193865151022383844364247891968 failed because of enotconn >> 2015-07-30 16:04:33.643 [error] >> <0.197.0>@riak_core_handoff_manager:handle_info:289 An outbound handoff of >> partition riak_kv_vnode 45671926166590716193865151022383844364247891968 was >> terminated for reason: {shutdown,{error,enotconn}} >> >> >> >> I have search
Re: why leaving riak cluster so slowly and how to accelerate the speed
During last three days, I setup a developing riak cluster with five nodes, and used "s3cmd" to upload 18GB testing data(maybe 20 thousands of files). After that, I tried to let one node leaving the cluster, and then shutdown and mark down it. Replacing the IP address and joining the cluster again. The above whole processes were successful. However, I'm not sure whether no not it can be done on production environment. I followed below the docs to do above steps: http://docs.basho.com/riak/latest/ops/running/nodes/renaming/ After I run "riak-admin cluster leave riak@'x.x.x.x'" ,"riak-admin cluster plan", "riak-admin cluster commit", then checked the member-status, the main difference of leaving cluster on production and developing environment are as below: root@cluster-s3-dev-hd1:~# riak-admin member-status = Membership == Status RingPendingNode --- leaving18.8% 0.0%'riak@10.21.236.185' valid 21.9% 25.0%'riak@10.21.236.181' valid 21.9% 25.0%'riak@10.21.236.182' valid 18.8% 25.0%'riak@10.21.236.183' valid 18.8% 25.0%'riak@10.21.236.184' --- several minutes elapsed, the then checking the status as below: root@cluster-s3-dev-hd1:~# riak-admin member-status = Membership == Status RingPendingNode --- leaving12.5% 0.0%'riak@10.21.236.185' valid 21.9% 25.0%'riak@10.21.236.181' valid 28.1% 25.0%'riak@10.21.236.182' valid 18.8% 25.0%'riak@10.21.236.183' valid 18.8% 25.0%'riak@10.21.236.184' --- Valid:4 / Leaving:1 / Exiting:0 / Joining:0 / Down:0 After that, I shutdown riak with "riak stop", and mark down it on active nodes. My question is what's the meaning ot "Pending 0.0%"? On production cluster, the status are as below: root@cluster1-hd12:/root/scripts# riak-admin transfers 'riak@10.21.136.94' waiting to handoff 5 partitions 'riak@10.21.136.93' waiting to handoff 5 partitions 'riak@10.21.136.92' waiting to handoff 5 partitions 'riak@10.21.136.91' waiting to handoff 5 partitions 'riak@10.21.136.86' waiting to handoff 5 partitions 'riak@10.21.136.81' waiting to handoff 2 partitions 'riak@10.21.136.76' waiting to handoff 3 partitions 'riak@10.21.136.71' waiting to handoff 5 partitions 'riak@10.21.136.66' waiting to handoff 5 partitions And there're active transfers. On developing environment, there're no active transfers after my running of "riak-admin cluster commit". Can I follow the same steps as developing environment to run it on production cluster? On Wed, Aug 12, 2015 at 10:39 PM, Dmitri Zagidulin wrote: > Responses inline. > > > On Tue, Aug 11, 2015 at 12:53 PM, changmao wang > wrote: > >> 1. About backuping new nodes of four and then using 'riak-admin >> force-replace'. what's the status of new added nodes? >> as you know, we want to replace one of leaving nodes. >> > > I don't understand the question. Doing 'riak-admin force-replace' on one > of the nodes that's leaving should overwrite the leave request and tell it > to change its node id / ip address. (If that doesn't work, stop the leaving > node, and do a 'riak-admin reip' command instead). > > > >> 2. what's the risk of 'riak-admin force-remove' 'riak@10.21.136.91' >> without backup? >> As you know, now the node(riak@10.21.136.91) is a member of the cluster, >> and keeping almost 2.5TB data, maybe 10 percent of the whole cluster. >> > > The only reason I asked about backup is because it sounded like you > cleared the disk on it. If it currently has the data, then it'll be fine. > Force-remove just changes the IP address, and doesn't delete the data or > anything. > > > On Tue, Aug 11, 2015 at 7:32 PM, Dmitri Zagidulin > wrote: > >> 1. How to force leave "leaving"'s nodes without data loss? >> >> This depends on - did you back up the data directory of the 4 new nodes, >> before you reformatted them? >> If you backed them up (and then restored the data directory once you >> reformatted them), you can try: >>
Re: why leaving riak cluster so slowly and how to accelerate the speed
Dmitri, I got it. Just following the steps tested on developing environment, I finished the changing on production environment. Now the production cluster status is as below: root@cluster1-hd13:~# riak-admin member-status = Membership == Status RingPendingNode --- valid 10.9% 10.9%'riak@10.21.136.66' valid 10.9% 10.9%'riak@10.21.136.71' valid 14.1% 10.9%'riak@10.21.136.76' valid 17.2% 12.5%'riak@10.21.136.81' valid 10.9% 10.9%'riak@10.21.136.86' valid 7.8% 10.9%'riak@10.21.136.95' valid 7.8% 10.9%'riak@10.21.136.96' valid 9.4% 10.9%'riak@10.21.136.97' valid 10.9% 10.9%'riak@10.21.136.98' --- Valid:9 / Leaving:0 / Exiting:0 / Joining:0 / Down:0 Thanks for your great help. we can close this ticket. several issues related with transferring partition, I'll open it in anther email. On Mon, Aug 17, 2015 at 11:01 PM, Dmitri Zagidulin wrote: > Amao, > > As I've mentioned, those pending transfers are going to stay there > indefinitely. They will keep showing up on the 'status' list, until you do > a 'force-replace' or 'force-remove'. > > > -- Amao Wang Best & Regards ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
s3cmd error: access to bucket was denied
Matthew, I used s3cmd --configure to generate ".s3cfg" config file and then access RIAK service by s3cmd. The access_key and secret_key from ".s3cfg" is same as admin_key and admin_secret from "/etc/riak-cs/app.config". However, I got error as below using s3cmd to access one bucket. root@cluster-s3-hd1:~# s3cmd -c /root/.s3cfg ls s3://pipeline/article/111.pdf ERROR: Access to bucket 'pipeline' was denied By the way, I used Riak and Riak-CS 1.4.2 on Ubuntu. Current production cluster is a legacy system without documents for co-workers. Attached file is "s3cfg" generated by "s3cmd --configure". -- Amao Wang Best & Regards .s3cfg Description: Binary data ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: s3cmd error: access to bucket was denied
somebody watching on this? On Wed, Aug 19, 2015 at 9:01 AM, changmao wang wrote: > Matthew, > > I used s3cmd --configure to generate ".s3cfg" config file and then access > RIAK service by s3cmd. > The access_key and secret_key from ".s3cfg" is same as admin_key > and admin_secret from "/etc/riak-cs/app.config". > > However, I got error as below using s3cmd to access one bucket. > > root@cluster-s3-hd1:~# s3cmd -c /root/.s3cfg ls > s3://pipeline/article/111.pdf > ERROR: Access to bucket 'pipeline' was denied > > By the way, I used Riak and Riak-CS 1.4.2 on Ubuntu. Current production > cluster is a legacy system without documents for co-workers. > > Attached file is "s3cfg" generated by "s3cmd --configure". > -- > Amao Wang > Best & Regards > -- Amao Wang Best & Regards ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: s3cmd error: access to bucket was denied
stanislav, what's your meaning of domain name of /etc/riak-cs/app.config and ~/.s3cfg? I guess it's cs_root_host parameter from /etc/riak-cs/app.config and host_base from '~/.s3cfg'. If so, there're same as "api2.cloud-datayes.com". However, I can not ping this host from localhost. On Thu, Aug 20, 2015 at 5:23 PM, Stanislav Vlasov wrote: > 2015-08-20 13:57 GMT+05:00 changmao wang : > > somebody watching on this? > > Do you set up same domain in riak-cs.conf and in .s3cfg? > I got such error in this case. > > > On Wed, Aug 19, 2015 at 9:01 AM, changmao wang > > wrote: > >> > >> Matthew, > >> > >> I used s3cmd --configure to generate ".s3cfg" config file and then > access > >> RIAK service by s3cmd. > >> The access_key and secret_key from ".s3cfg" is same as admin_key and > >> admin_secret from "/etc/riak-cs/app.config". > >> > >> However, I got error as below using s3cmd to access one bucket. > >> > >> root@cluster-s3-hd1:~# s3cmd -c /root/.s3cfg ls > >> s3://pipeline/article/111.pdf > >> ERROR: Access to bucket 'pipeline' was denied > >> > >> By the way, I used Riak and Riak-CS 1.4.2 on Ubuntu. Current production > >> cluster is a legacy system without documents for co-workers. > >> > >> Attached file is "s3cfg" generated by "s3cmd --configure". > >> -- > >> Amao Wang > >> Best & Regards > > > > > > > > > > -- > > Amao Wang > > Best & Regards > > > > ___ > > riak-users mailing list > > riak-users@lists.basho.com > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > > > > -- > Stanislav > -- Amao Wang Best & Regards ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: s3cmd error: access to bucket was denied
Just now, I used "admin_key" and "admin_secret" from /etc/riak-cs/app.config to run "s3cmd -c s3-stock ls s3://stock/XSHE/0/000600" and I got the below error: ERROR: Access to bucket 'stock' was denied Below is abstract from "/var/log/riak-cs/console.log" 2015-08-20 18:40:22.790 [debug] <0.28085.18>@riak_cs_s3_auth:calculate_signature:129 STS: ["GET","\n",[],"\n",[],"\n","\n",[["x-amz-date",":",<<"Thu, 20 Aug 2015 10:40:22 +">>,"\n"]],["/stock/",[]]] 2015-08-20 18:40:32.861 [error] <0.28153.18>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: no_user_key 2015-08-20 18:40:32.861 [debug] <0.28153.18>@riak_cs_wm_common:post_authentication:452 No user key 2015-08-20 18:40:32.969 [debug] <0.28189.18>@riak_cs_get_fsm:prepare:406 Manifest: {lfs_manifest_v3,3,1048576,{<<"pipeline">>,<<100,97,116,97,121,101,115,47,112,105,112,101,108,105,110,101,47,100,97,116,97,47,114,101,112,111,114,116,47,115,122,47,83,90,48,48,50,49,55,53,67,78,47,50,48,49,48,95,50,48,49,48,45,48,52,45,50,52,95,229,133,172,229,143,184,231,171,160,231,168,139,239,188,136,50,48,49,48,229,185,180,52,230,156,136,239,188,137,46,80,68,70>>},[],"2013-12-16T23:01:12.000Z",<<192,71,150,153,181,181,77,61,186,41,100,32,5,91,197,166>>,255387,<<"application/pdf">>,<<55,41,141,170,187,226,47,223,183,95,105,129,155,154,210,202>>,active,{1387,234872,598819},{1387,234872,918555},[],undefined,undefined,undefined,undefined,{acl_v2,{"pipelinewrite","ef38ca69e145a40c1f8378633994192dace4539339315e6b42d7d1e6e2d2de51","AVG2DHZ4UNUYFAZ8F4WR"},[{{"pipelinewrite","ef38ca69e145a40c1f8378633994192dace4539339315e6b42d7d1e6e2d2de51"},['FULL_CONTROL']},{'AllUsers',['READ']}],{1387,234872,598546}},[],undefined} 2015-08-20 18:40:33.043 [debug] <0.28189.18>@riak_cs_lfs_utils:range_blocks:118 InitialBlock: 0, FinalBlock: 0 2015-08-20 18:40:33.043 [debug] <0.28189.18>@riak_cs_lfs_utils:range_blocks:120 SkipInitial: 0, KeepFinal: 255387 2015-08-20 18:40:33.050 [debug] <0.28189.18>@riak_cs_get_fsm:waiting_continue_or_stop:229 Block Servers: [<0.28191.18>] 2015-08-20 18:40:33.079 [debug] <0.28189.18>@riak_cs_get_fsm:waiting_chunks:307 Retrieved block {<<192,71,150,153,181,181,77,61,186,41,100,32,5,91,197,166>>,0} 2015-08-20 18:40:33.079 [debug] <0.28189.18>@riak_cs_get_fsm:perhaps_send_to_user:280 Returning block {<<192,71,150,153,181,181,77,61,186,41,100,32,5,91,197,166>>,0} to client 2015-08-20 18:40:38.218 [error] <0.28086.18>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: no_user_key 2015-08-20 18:40:38.218 [debug] <0.28086.18>@riak_cs_wm_common:post_authentication:452 No user key 2015-08-20 18:40:38.226 [debug] <0.28210.18>@riak_cs_get_fsm:prepare:406 Manifest: {lfs_manifest_v3,3,1048576,{<<"pipeline">>,<<100,97,116,97,121,101,115,47,112,105,112,101,108,105,110,101,47,100,97,116,97,47,114,101,112,111,114,116,47,115,104,47,83,72,54,48,48,55,53,48,67,78,47,50,48,48,55,95,50,48,48,55,45,49,49,45,50,49,95,230,177,159,228,184,173,232,141,175,228,184,154,229,133,179,228,186,142,73,66,69,95,53,232,141,175,229,147,129,232,142,183,229,190,151,228,186,140,230,156,159,228,184,180,229,186,138,230,137,185,230,150,135,231,154,132,229,133,172,229,145,138,229,143,138,233,163,142,233,153,169,230,143,144,231,164,186,46,112,100,102>>},[],"2013-12-15T09:04:48.000Z",<<201,247,249,158,95,22,64,242,161,118,253,64,120,187,205,105>>,89863,<<"application/pdf">>,<<139,151,203,173,6,111,222,48,17,81,102,170,216,66,193,77>>,active,{1387,98288,545827},{1387,98288,618409},[],undefined,undefined,undefined,undefined,{acl_v2,{"pipelinewrite","ef38ca69e145a40c1f8378633994192dace4539339315e6b42d7d1e6e2d2de51","AVG2DHZ4UNUYFAZ8F4WR"},[{{"pipelinewrite","ef38ca69e145a40c1f8378633994192dace4539339315e6b42d7d1e6e2d2de51"},['FULL_CONTROL']},{'AllUsers',['READ']}],{1387,98288,545618}},[],undefined} 2015-08-20 18:40:38.280 [debug] <0.28210.18>@riak_cs_lfs_utils:range_blocks:118 InitialBlock: 0, FinalBlock: 0 2015-08-20 18:40:38.280 [debug] <0.28210.18>@riak_cs_lfs_utils:range_blocks:120 SkipInitial: 0, KeepFinal: 89863 2015-08-20 18:40:38.280 [debug] <0.28210.18>@riak_cs_get_fsm:waiting_continue_or_stop:229 Block Servers: [<0.28212.18>] 2015-08-20 18:40:38.343 [debug] <0.28210.18>@riak_cs_get_fsm:waiting_chunks:307 Retrieved block {<<201,247,249,158,95,22,64,242,161,118,253,64,120,187,205,105>>,0
Re: s3cmd error: access to bucket was denied
Kazuhiro, Maybe that's not the key point. I'm using riak 1.4.2 and follow below docs to configure "s3cfg" file. http://docs.basho.com/riakcs/1.4.2/cookbooks/configuration/Configuring-an-S3-Client/#Sample-s3cmd-Configuration-File-for-Production-Use There's no "signature_v2" parameter in "s3cfg". However, I added this parameter to "s3cfg" and tried again with same errors. On Thu, Aug 20, 2015 at 10:31 PM, Kazuhiro Suzuki wrote: > Hi Changmao, > > It seems your s3cmd config should include 2 items: > > signature_v2 = True > host_base = api2.cloud-datayes.com > > Riak CS requires "signature_v2 = True" since Riak CS has not supported > s3 authentication version 4 yet. > You can find a sample configuration of s3cmd here to interact with Riak CS > [1]. > > [1]: > http://docs.basho.com/riakcs/2.0.1/cookbooks/configuration/Configuring-an-S3-Client/#Sample-s3cmd-Configuration-File-for-Production-Use > > Thanks, > > On Thu, Aug 20, 2015 at 7:44 PM, changmao wang > wrote: > > Just now, I used "admin_key" and "admin_secret" from > /etc/riak-cs/app.config > > to run "s3cmd -c s3-stock ls s3://stock/XSHE/0/000600" > > and I got the below error: > > ERROR: Access to bucket 'stock' was denied > > > > Below is abstract from "/var/log/riak-cs/console.log" > > 2015-08-20 18:40:22.790 [debug] > > <0.28085.18>@riak_cs_s3_auth:calculate_signature:129 STS: > > ["GET","\n",[],"\n",[],"\n","\n",[["x-amz-date",":",<<"Thu, 20 Aug 2015 > > 10:40:22 +">>,"\n"]],["/stock/",[]]] > > 2015-08-20 18:40:32.861 [error] > > <0.28153.18>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user > > record for s3 failed. Reason: no_user_key > > 2015-08-20 18:40:32.861 [debug] > > <0.28153.18>@riak_cs_wm_common:post_authentication:452 No user key > > 2015-08-20 18:40:32.969 [debug] <0.28189.18>@riak_cs_get_fsm:prepare:406 > > Manifest: > > > {lfs_manifest_v3,3,1048576,{<<"pipeline">>,<<100,97,116,97,121,101,115,47,112,105,112,101,108,105,110,101,47,100,97,116,97,47,114,101,112,111,114,116,47,115,122,47,83,90,48,48,50,49,55,53,67,78,47,50,48,49,48,95,50,48,49,48,45,48,52,45,50,52,95,229,133,172,229,143,184,231,171,160,231,168,139,239,188,136,50,48,49,48,229,185,180,52,230,156,136,239,188,137,46,80,68,70>>},[],"2013-12-16T23:01:12.000Z",<<192,71,150,153,181,181,77,61,186,41,100,32,5,91,197,166>>,255387,<<"application/pdf">>,<<55,41,141,170,187,226,47,223,183,95,105,129,155,154,210,202>>,active,{1387,234872,598819},{1387,234872,918555},[],undefined,undefined,undefined,undefined,{acl_v2,{"pipelinewrite","ef38ca69e145a40c1f8378633994192dace4539339315e6b42d7d1e6e2d2de51","AVG2DHZ4UNUYFAZ8F4WR"},[{{"pipelinewrite","ef38ca69e145a40c1f8378633994192dace4539339315e6b42d7d1e6e2d2de51"},['FULL_CONTROL']},{'AllUsers',['READ']}],{1387,234872,598546}},[],undefined} > > 2015-08-20 18:40:33.043 [debug] > > <0.28189.18>@riak_cs_lfs_utils:range_blocks:118 InitialBlock: 0, > FinalBlock: > > 0 > > 2015-08-20 18:40:33.043 [debug] > > <0.28189.18>@riak_cs_lfs_utils:range_blocks:120 SkipInitial: 0, > KeepFinal: > > 255387 > > 2015-08-20 18:40:33.050 [debug] > > <0.28189.18>@riak_cs_get_fsm:waiting_continue_or_stop:229 Block Servers: > > [<0.28191.18>] > > 2015-08-20 18:40:33.079 [debug] > > <0.28189.18>@riak_cs_get_fsm:waiting_chunks:307 Retrieved block > > {<<192,71,150,153,181,181,77,61,186,41,100,32,5,91,197,166>>,0} > > 2015-08-20 18:40:33.079 [debug] > > <0.28189.18>@riak_cs_get_fsm:perhaps_send_to_user:280 Returning block > > {<<192,71,150,153,181,181,77,61,186,41,100,32,5,91,197,166>>,0} to client > > 2015-08-20 18:40:38.218 [error] > > <0.28086.18>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user > > record for s3 failed. Reason: no_user_key > > 2015-08-20 18:40:38.218 [debug] > > <0.28086.18>@riak_cs_wm_common:post_authentication:452 No user key > > 2015-08-20 18:40:38.226 [debug] <0.28210.18>@riak_cs_get_fsm:prepare:406 > > Manifest: > > > {lfs_manifest_v3,3,1048576,{<<"pipeline">>,<<100,97,116,97,121,101,115,47,112,105,112,101,108,105,110,101,47,100,97,116,97,47,114,101,112,111,114,116,47,115,104,47,83,72,54,48,48,55,53,48,67,78,47,50,48,48,55,95,50,48,48,55,45,49,49,45,50,49,95,230,177,159,228,184,173,23
Re: s3cmd error: access to bucket was denied
Shunichi, Thanks for your reply. Below is my command result: root@cluster-s3-hd1:~# s3cmd ls 2013-12-01 06:45 s3://test root@cluster-s3-hd1:~# s3cmd info s3://stock ERROR: Access to bucket 'stock' was denied root@cluster-s3-hd1:~# s3cmd info s3://stock -d DEBUG: ConfigParser: Reading file '/root/.s3cfg' DEBUG: ConfigParser: access_key->M2...17_chars...K DEBUG: ConfigParser: bucket_location->US DEBUG: ConfigParser: cloudfront_host->cloudfront.amazonaws.com DEBUG: ConfigParser: cloudfront_resource->/2010-07-15/distribution DEBUG: ConfigParser: default_mime_type->binary/octet-stream DEBUG: ConfigParser: delete_removed->False DEBUG: ConfigParser: dry_run->False DEBUG: ConfigParser: encoding->UTF-8 DEBUG: ConfigParser: encrypt->False DEBUG: ConfigParser: follow_symlinks->False DEBUG: ConfigParser: force->False DEBUG: ConfigParser: get_continue->False DEBUG: ConfigParser: gpg_command->/usr/bin/gpg DEBUG: ConfigParser: gpg_decrypt->%(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s DEBUG: ConfigParser: gpg_encrypt->%(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s DEBUG: ConfigParser: gpg_passphrase->...-3_chars... DEBUG: ConfigParser: guess_mime_type->True DEBUG: ConfigParser: host_base->api2.cloud-datayes.com DEBUG: ConfigParser: host_bucket->%(bucket)s.api2.cloud-datayes.com DEBUG: ConfigParser: human_readable_sizes->False DEBUG: ConfigParser: list_md5->False DEBUG: ConfigParser: log_target_prefix-> DEBUG: ConfigParser: preserve_attrs->True DEBUG: ConfigParser: progress_meter->True DEBUG: ConfigParser: proxy_host->10.21.136.81 DEBUG: ConfigParser: proxy_port->8080 DEBUG: ConfigParser: recursive->False DEBUG: ConfigParser: recv_chunk->4096 DEBUG: ConfigParser: reduced_redundancy->False DEBUG: ConfigParser: secret_key->1u...37_chars...= DEBUG: ConfigParser: send_chunk->4096 DEBUG: ConfigParser: simpledb_host->sdb.amazonaws.com DEBUG: ConfigParser: skip_existing->False DEBUG: ConfigParser: socket_timeout->10 DEBUG: ConfigParser: urlencoding_mode->normal DEBUG: ConfigParser: use_https->False DEBUG: ConfigParser: verbosity->WARNING DEBUG: Updating Config.Config encoding -> UTF-8 DEBUG: Updating Config.Config follow_symlinks -> False DEBUG: Updating Config.Config verbosity -> 10 DEBUG: Unicodising 'info' using UTF-8 DEBUG: Unicodising 's3://stock' using UTF-8 DEBUG: Command: info DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Mon, 24 Aug 2015 01:11:09 +\n/stock/?location' DEBUG: CreateRequest: resource[uri]=/?location DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Mon, 24 Aug 2015 01:11:09 +\n/stock/?location' DEBUG: Processing request, please wait... DEBUG: get_hostname(stock): stock.api2.cloud-datayes.com DEBUG: format_uri(): http://stock.api2.cloud-datayes.com/?location DEBUG: Response: {'status': 403, 'headers': {'date': 'Mon, 24 Aug 2015 01:11:09 GMT', 'content-length': '160', 'content-type': 'application/xml', 'server': 'Riak CS'}, 'reason': 'Forbidden', 'data': 'AccessDeniedAccess Denied/stock'} DEBUG: S3Error: 403 (Forbidden) DEBUG: HttpHeader: date: Mon, 24 Aug 2015 01:11:09 GMT DEBUG: HttpHeader: content-length: 160 DEBUG: HttpHeader: content-type: application/xml DEBUG: HttpHeader: server: Riak CS DEBUG: ErrorXML: Code: 'AccessDenied' DEBUG: ErrorXML: Message: 'Access Denied' DEBUG: ErrorXML: Resource: '/stock' DEBUG: ErrorXML: RequestId: None ERROR: Access to bucket 'stock' was denied On Mon, Aug 24, 2015 at 9:04 AM, Shunichi Shinohara wrote: > The error message in console.log shows no user with access_key in your > s3cfg. > Could you provide resutls following commands? > > - s3cmd ls > - s3cmd info s3://stock > > If error happens, debug print switch "-d" of s3cmd might help. > > [1] > http://docs.basho.com/riakcs/latest/cookbooks/Account-Management/#Creating-a-User-Account > > -- > Shunichi Shinohara > Basho Japan KK > > > On Fri, Aug 21, 2015 at 10:00 AM, changmao wang > wrote: > > Kazuhiro, > > > > Maybe that's not the key point. I'm using riak 1.4.2 and follow below > docs > > to configure "s3cfg" file. > > > http://docs.basho.com/riakcs/1.4.2/cookbooks/configuration/Configuring-an-S3-Client/#Sample-s3cmd-Configuration-File-for-Production-Use > > > > There's no "signature_v2" parameter in "s3cfg". However, I added this > > parameter to "s3cfg" and tried again with same errors. > > > > > > > > &
Re: s3cmd error: access to bucket was denied
1. root@cluster1-hd10:~# grep cs_root_host /etc/riak-cs/app.config {cs_root_host, "api2.cloud-datayes.com"}, root@cluster1-hd10:~# grep host_base .s3cfg host_base = api2.cloud-datayes.com root@cluster1-hd10:~# grep host_base s3cfg1 host_base = api2.cloud-datayes.com 2. please check attached file for "s3cmd -d" output and '/etc/riak-cs/console.log'. On Mon, Aug 24, 2015 at 9:54 AM, Shunichi Shinohara wrote: > What is "api2.cloud-datayes.com"? Your s3cfg attached at the first one > in this email thread > does not include it. Please make sure you provide correct / consistent > information to > debug the issue. > > - What is your riak cs config "cs_root_host"? > - What is your host_base in s3cfg that you USE? > - What is your host_bucket in s3cfg? > > Also, please attach s3cmd debug output AND riak cs console log at the same > time > interval. > -- > Shunichi Shinohara > Basho Japan KK > > > On Mon, Aug 24, 2015 at 10:42 AM, changmao wang > wrote: > > I'm not sure who created it. This's a legacy production system. > > > > Just now, I used another "s3cfg" file to access it. Below is my output: > > root@cluster1-hd10:~# s3cmd -c s3cfg1 info > > s3://stock/XSHE/0/50/2008/XSHE-50-20080102 > > s3://stock/XSHE/0/50/2008/XSHE-50-20080102 (object): > >File size: 397535 > >Last mod: Thu, 05 Dec 2013 03:19:00 GMT > >MIME type: binary/octet-stream > >MD5 sum: feb2609ecfc9bb21549f2401a5c9477d > >ACL: stockwrite: FULL_CONTROL > >ACL: *anon*: READ > >URL: > > http://stock.s3.amazonaws.com/XSHE/0/50/2008/XSHE-50-20080102 > > root@cluster1-hd10:~# s3cmd -c s3cfg1 ls > > s3://stock/XSHE/0/50/2008/XSHE-50-20080102 -d > > DEBUG: ConfigParser: Reading file 's3cfg1' > > DEBUG: ConfigParser: access_key->TE...17_chars...0 > > DEBUG: ConfigParser: bucket_location->US > > DEBUG: ConfigParser: cloudfront_host->cloudfront.amazonaws.com > > DEBUG: ConfigParser: cloudfront_resource->/2010-07-15/distribution > > DEBUG: ConfigParser: default_mime_type->binary/octet-stream > > DEBUG: ConfigParser: delete_removed->False > > DEBUG: ConfigParser: dry_run->False > > DEBUG: ConfigParser: encoding->UTF-8 > > DEBUG: ConfigParser: encrypt->False > > DEBUG: ConfigParser: follow_symlinks->False > > DEBUG: ConfigParser: force->False > > DEBUG: ConfigParser: get_continue->False > > DEBUG: ConfigParser: gpg_command->/usr/bin/gpg > > DEBUG: ConfigParser: gpg_decrypt->%(gpg_command)s -d --verbose > > --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o > > %(output_file)s %(input_file)s > > DEBUG: ConfigParser: gpg_encrypt->%(gpg_command)s -c --verbose > > --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o > > %(output_file)s %(input_file)s > > DEBUG: ConfigParser: gpg_passphrase->...-3_chars... > > DEBUG: ConfigParser: guess_mime_type->True > > DEBUG: ConfigParser: host_base->api2.cloud-datayes.com > > DEBUG: ConfigParser: host_bucket->%(bucket)s.api2.cloud-datayes.com > > DEBUG: ConfigParser: human_readable_sizes->False > > DEBUG: ConfigParser: list_md5->False > > DEBUG: ConfigParser: log_target_prefix-> > > DEBUG: ConfigParser: preserve_attrs->True > > DEBUG: ConfigParser: progress_meter->True > > DEBUG: ConfigParser: proxy_host->10.21.136.81 > > DEBUG: ConfigParser: proxy_port->8080 > > DEBUG: ConfigParser: recursive->False > > DEBUG: ConfigParser: recv_chunk->4096 > > DEBUG: ConfigParser: reduced_redundancy->False > > DEBUG: ConfigParser: secret_key->Hk...37_chars...= > > DEBUG: ConfigParser: send_chunk->4096 > > DEBUG: ConfigParser: simpledb_host->sdb.amazonaws.com > > DEBUG: ConfigParser: skip_existing->False > > DEBUG: ConfigParser: socket_timeout->100 > > DEBUG: ConfigParser: urlencoding_mode->normal > > DEBUG: ConfigParser: use_https->False > > DEBUG: ConfigParser: verbosity->WARNING > > DEBUG: Updating Config.Config encoding -> UTF-8 > > DEBUG: Updating Config.Config follow_symlinks -> False > > DEBUG: Updating Config.Config verbosity -> 10 > > DEBUG: Unicodising 'ls' using UTF-8 > > DEBUG: Unicodising 's3://stock/XSHE/0/50/2008/XSHE-50-20080102' > > using UTF-8 > > DEBUG: Command: ls > > DEBUG: Bucket 's3://stock': > > DEBUG: String 'XSHE/0/50/2008/XSHE-50-20080102' encoded to > > 'XSHE/0/50/2008
Re: s3cmd error: access to bucket was denied
I'm not sure who created it. This's a legacy production system. Just now, I used another "s3cfg" file to access it. Below is my output: root@cluster1-hd10:~# s3cmd -c s3cfg1 info s3://stock/XSHE/0/50/2008/XSHE-50-20080102 s3://stock/XSHE/0/50/2008/XSHE-50-20080102 (object): File size: 397535 Last mod: Thu, 05 Dec 2013 03:19:00 GMT MIME type: binary/octet-stream MD5 sum: feb2609ecfc9bb21549f2401a5c9477d ACL: stockwrite: FULL_CONTROL ACL: *anon*: READ URL: http://stock.s3.amazonaws.com/XSHE/0/50/2008/XSHE-50-20080102 root@cluster1-hd10:~# s3cmd -c s3cfg1 ls s3://stock/XSHE/0/50/2008/XSHE-50-20080102 -d DEBUG: ConfigParser: Reading file 's3cfg1' DEBUG: ConfigParser: access_key->TE...17_chars...0 DEBUG: ConfigParser: bucket_location->US DEBUG: ConfigParser: cloudfront_host->cloudfront.amazonaws.com DEBUG: ConfigParser: cloudfront_resource->/2010-07-15/distribution DEBUG: ConfigParser: default_mime_type->binary/octet-stream DEBUG: ConfigParser: delete_removed->False DEBUG: ConfigParser: dry_run->False DEBUG: ConfigParser: encoding->UTF-8 DEBUG: ConfigParser: encrypt->False DEBUG: ConfigParser: follow_symlinks->False DEBUG: ConfigParser: force->False DEBUG: ConfigParser: get_continue->False DEBUG: ConfigParser: gpg_command->/usr/bin/gpg DEBUG: ConfigParser: gpg_decrypt->%(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s DEBUG: ConfigParser: gpg_encrypt->%(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s DEBUG: ConfigParser: gpg_passphrase->...-3_chars... DEBUG: ConfigParser: guess_mime_type->True DEBUG: ConfigParser: host_base->api2.cloud-datayes.com DEBUG: ConfigParser: host_bucket->%(bucket)s.api2.cloud-datayes.com DEBUG: ConfigParser: human_readable_sizes->False DEBUG: ConfigParser: list_md5->False DEBUG: ConfigParser: log_target_prefix-> DEBUG: ConfigParser: preserve_attrs->True DEBUG: ConfigParser: progress_meter->True DEBUG: ConfigParser: proxy_host->10.21.136.81 DEBUG: ConfigParser: proxy_port->8080 DEBUG: ConfigParser: recursive->False DEBUG: ConfigParser: recv_chunk->4096 DEBUG: ConfigParser: reduced_redundancy->False DEBUG: ConfigParser: secret_key->Hk...37_chars...= DEBUG: ConfigParser: send_chunk->4096 DEBUG: ConfigParser: simpledb_host->sdb.amazonaws.com DEBUG: ConfigParser: skip_existing->False DEBUG: ConfigParser: socket_timeout->100 DEBUG: ConfigParser: urlencoding_mode->normal DEBUG: ConfigParser: use_https->False DEBUG: ConfigParser: verbosity->WARNING DEBUG: Updating Config.Config encoding -> UTF-8 DEBUG: Updating Config.Config follow_symlinks -> False DEBUG: Updating Config.Config verbosity -> 10 DEBUG: Unicodising 'ls' using UTF-8 DEBUG: Unicodising 's3://stock/XSHE/0/50/2008/XSHE-50-20080102' using UTF-8 DEBUG: Command: ls DEBUG: Bucket 's3://stock': DEBUG: String 'XSHE/0/50/2008/XSHE-50-20080102' encoded to 'XSHE/0/50/2008/XSHE-50-20080102' DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Mon, 24 Aug 2015 01:36:01 +\n/stock/' DEBUG: CreateRequest: resource[uri]=/ DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Mon, 24 Aug 2015 01:36:01 +\n/stock/' DEBUG: Processing request, please wait... DEBUG: get_hostname(stock): stock.api2.cloud-datayes.com DEBUG: format_uri(): http://stock.api2.cloud-datayes.com/?prefix=XSHE/0/50/2008/XSHE-50-20080102&delimiter=/ WARNING: Retrying failed request: /?prefix=XSHE/0/50/2008/XSHE-50-20080102&delimiter=/ ('') WARNING: Waiting 3 sec... DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Mon, 24 Aug 2015 01:37:05 +\n/stock/' DEBUG: Processing request, please wait... DEBUG: get_hostname(stock): stock.api2.cloud-datayes.com DEBUG: format_uri(): http://stock.api2.cloud-datayes.com/?prefix=XSHE/0/50/2008/XSHE-50-20080102&delimiter=/ WARNING: Retrying failed request: /?prefix=XSHE/0/50/2008/XSHE-50-20080102&delimiter=/ ('') WARNING: Waiting 6 sec... On Mon, Aug 24, 2015 at 9:17 AM, Shunichi Shinohara wrote: > The result of "s3cmd ls" (aka, GET Service API) indicates there > is no bucket with name "stock": > > > root@cluster-s3-hd1:~# s3cmd ls > > 2013-12-01 06:45 s3://test > > Have you created it? > > -- > Shunichi Shinohara > Basho Japan KK > > > On Mon, Aug 24, 2015 at 10:14 AM, changmao wang > wrote: > > Shunichi, > > > > Thanks for your reply. Below is my command result: > > root@cluster-s3-hd1:~# s3cmd ls > > 2013-12-01 06:45 s3://test > > root@cluster-s3-hd1:~# s3cmd info s3://stock > > ERROR: Access to
Re: s3cmd error: access to bucket was denied
Please check attached file for details. On Mon, Aug 24, 2015 at 4:48 PM, Shunichi Shinohara wrote: > Then, back to my first questions: > Could you provide results following commands with s3cfg1? > - s3cmd ls > - s3cmd info s3://stock > > From log file, gc index queries timed out again and again. > Not sure but it may be subtle situation... > > -- > Shunichi Shinohara > Basho Japan KK > > > On Mon, Aug 24, 2015 at 11:03 AM, changmao wang > wrote: > > 1. root@cluster1-hd10:~# grep cs_root_host /etc/riak-cs/app.config > > {cs_root_host, "api2.cloud-datayes.com"}, > > root@cluster1-hd10:~# grep host_base .s3cfg > > host_base = api2.cloud-datayes.com > > root@cluster1-hd10:~# grep host_base s3cfg1 > > host_base = api2.cloud-datayes.com > > > > 2. please check attached file for "s3cmd -d" output and > > '/etc/riak-cs/console.log'. > > > > > > On Mon, Aug 24, 2015 at 9:54 AM, Shunichi Shinohara > wrote: > >> > >> What is "api2.cloud-datayes.com"? Your s3cfg attached at the first one > >> in this email thread > >> does not include it. Please make sure you provide correct / consistent > >> information to > >> debug the issue. > >> > >> - What is your riak cs config "cs_root_host"? > >> - What is your host_base in s3cfg that you USE? > >> - What is your host_bucket in s3cfg? > >> > >> Also, please attach s3cmd debug output AND riak cs console log at the > same > >> time > >> interval. > >> -- > >> Shunichi Shinohara > >> Basho Japan KK > >> > >> > >> On Mon, Aug 24, 2015 at 10:42 AM, changmao wang < > wang.chang...@gmail.com> > >> wrote: > >> > I'm not sure who created it. This's a legacy production system. > >> > > >> > Just now, I used another "s3cfg" file to access it. Below is my > output: > >> > root@cluster1-hd10:~# s3cmd -c s3cfg1 info > >> > s3://stock/XSHE/0/50/2008/XSHE-50-20080102 > >> > s3://stock/XSHE/0/50/2008/XSHE-50-20080102 (object): > >> >File size: 397535 > >> >Last mod: Thu, 05 Dec 2013 03:19:00 GMT > >> >MIME type: binary/octet-stream > >> >MD5 sum: feb2609ecfc9bb21549f2401a5c9477d > >> >ACL: stockwrite: FULL_CONTROL > >> >ACL: *anon*: READ > >> >URL: > >> > http://stock.s3.amazonaws.com/XSHE/0/50/2008/XSHE-50-20080102 > >> > root@cluster1-hd10:~# s3cmd -c s3cfg1 ls > >> > s3://stock/XSHE/0/50/2008/XSHE-50-20080102 -d > >> > DEBUG: ConfigParser: Reading file 's3cfg1' > >> > DEBUG: ConfigParser: access_key->TE...17_chars...0 > >> > DEBUG: ConfigParser: bucket_location->US > >> > DEBUG: ConfigParser: cloudfront_host->cloudfront.amazonaws.com > >> > DEBUG: ConfigParser: cloudfront_resource->/2010-07-15/distribution > >> > DEBUG: ConfigParser: default_mime_type->binary/octet-stream > >> > DEBUG: ConfigParser: delete_removed->False > >> > DEBUG: ConfigParser: dry_run->False > >> > DEBUG: ConfigParser: encoding->UTF-8 > >> > DEBUG: ConfigParser: encrypt->False > >> > DEBUG: ConfigParser: follow_symlinks->False > >> > DEBUG: ConfigParser: force->False > >> > DEBUG: ConfigParser: get_continue->False > >> > DEBUG: ConfigParser: gpg_command->/usr/bin/gpg > >> > DEBUG: ConfigParser: gpg_decrypt->%(gpg_command)s -d --verbose > >> > --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o > >> > %(output_file)s %(input_file)s > >> > DEBUG: ConfigParser: gpg_encrypt->%(gpg_command)s -c --verbose > >> > --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o > >> > %(output_file)s %(input_file)s > >> > DEBUG: ConfigParser: gpg_passphrase->...-3_chars... > >> > DEBUG: ConfigParser: guess_mime_type->True > >> > DEBUG: ConfigParser: host_base->api2.cloud-datayes.com > >> > DEBUG: ConfigParser: host_bucket->%(bucket)s.api2.cloud-datayes.com > >> > DEBUG: ConfigParser: human_readable_sizes->False > >> > DEBUG: ConfigParser: list_md5->False > >> > DEBUG: ConfigParser: log_target_prefix-> > >> > DEBUG: ConfigParser: preserve_attrs->True > >> > DEBUG: ConfigParser: progress_meter->True > >> >
Re: s3cmd error: access to bucket was denied
Any ideas on this issue? On Mon, Aug 24, 2015 at 5:09 PM, changmao wang wrote: > Please check attached file for details. > > On Mon, Aug 24, 2015 at 4:48 PM, Shunichi Shinohara > wrote: > >> Then, back to my first questions: >> Could you provide results following commands with s3cfg1? >> - s3cmd ls >> - s3cmd info s3://stock >> >> From log file, gc index queries timed out again and again. >> Not sure but it may be subtle situation... >> >> -- >> Shunichi Shinohara >> Basho Japan KK >> >> >> On Mon, Aug 24, 2015 at 11:03 AM, changmao wang >> wrote: >> > 1. root@cluster1-hd10:~# grep cs_root_host /etc/riak-cs/app.config >> > {cs_root_host, "api2.cloud-datayes.com"}, >> > root@cluster1-hd10:~# grep host_base .s3cfg >> > host_base = api2.cloud-datayes.com >> > root@cluster1-hd10:~# grep host_base s3cfg1 >> > host_base = api2.cloud-datayes.com >> > >> > 2. please check attached file for "s3cmd -d" output and >> > '/etc/riak-cs/console.log'. >> > >> > >> > On Mon, Aug 24, 2015 at 9:54 AM, Shunichi Shinohara >> wrote: >> >> >> >> What is "api2.cloud-datayes.com"? Your s3cfg attached at the first one >> >> in this email thread >> >> does not include it. Please make sure you provide correct / consistent >> >> information to >> >> debug the issue. >> >> >> >> - What is your riak cs config "cs_root_host"? >> >> - What is your host_base in s3cfg that you USE? >> >> - What is your host_bucket in s3cfg? >> >> >> >> Also, please attach s3cmd debug output AND riak cs console log at the >> same >> >> time >> >> interval. >> >> -- >> >> Shunichi Shinohara >> >> Basho Japan KK >> >> >> >> >> >> On Mon, Aug 24, 2015 at 10:42 AM, changmao wang < >> wang.chang...@gmail.com> >> >> wrote: >> >> > I'm not sure who created it. This's a legacy production system. >> >> > >> >> > Just now, I used another "s3cfg" file to access it. Below is my >> output: >> >> > root@cluster1-hd10:~# s3cmd -c s3cfg1 info >> >> > s3://stock/XSHE/0/50/2008/XSHE-50-20080102 >> >> > s3://stock/XSHE/0/50/2008/XSHE-50-20080102 (object): >> >> >File size: 397535 >> >> >Last mod: Thu, 05 Dec 2013 03:19:00 GMT >> >> >MIME type: binary/octet-stream >> >> >MD5 sum: feb2609ecfc9bb21549f2401a5c9477d >> >> >ACL: stockwrite: FULL_CONTROL >> >> >ACL: *anon*: READ >> >> >URL: >> >> > >> http://stock.s3.amazonaws.com/XSHE/0/50/2008/XSHE-50-20080102 >> >> > root@cluster1-hd10:~# s3cmd -c s3cfg1 ls >> >> > s3://stock/XSHE/0/50/2008/XSHE-50-20080102 -d >> >> > DEBUG: ConfigParser: Reading file 's3cfg1' >> >> > DEBUG: ConfigParser: access_key->TE...17_chars...0 >> >> > DEBUG: ConfigParser: bucket_location->US >> >> > DEBUG: ConfigParser: cloudfront_host->cloudfront.amazonaws.com >> >> > DEBUG: ConfigParser: cloudfront_resource->/2010-07-15/distribution >> >> > DEBUG: ConfigParser: default_mime_type->binary/octet-stream >> >> > DEBUG: ConfigParser: delete_removed->False >> >> > DEBUG: ConfigParser: dry_run->False >> >> > DEBUG: ConfigParser: encoding->UTF-8 >> >> > DEBUG: ConfigParser: encrypt->False >> >> > DEBUG: ConfigParser: follow_symlinks->False >> >> > DEBUG: ConfigParser: force->False >> >> > DEBUG: ConfigParser: get_continue->False >> >> > DEBUG: ConfigParser: gpg_command->/usr/bin/gpg >> >> > DEBUG: ConfigParser: gpg_decrypt->%(gpg_command)s -d --verbose >> >> > --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o >> >> > %(output_file)s %(input_file)s >> >> > DEBUG: ConfigParser: gpg_encrypt->%(gpg_command)s -c --verbose >> >> > --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o >> >> > %(output_file)s %(input_file)s >> >> > DEBUG: ConfigParser: gpg_passphrase->...-3_chars... >> >> > DEBUG: ConfigParser: guess_mime_type->True >> >> > DEBUG: Confi
Re: s3cmd error: access to bucket was denied
Shunichi, Just now, I followed your direction to change fold_objects_for_list_keys to true, and restart riak-cs service. sed -i '/fold_objects_for_list_keys/ s/false/true/g' /etc/riak-cs/app.config; riak-cs restart After that, I run below command and got same error. s3cmd -c s3cfg1 ls s3://stock/XSHE/0/50/2008/XSHE-50-20080102 -d ... DEBUG: Unicodising 'ls' using UTF-8 DEBUG: Unicodising 's3://stock/XSHE/0/50/2008/XSHE-50-20080102' using UTF-8 DEBUG: Command: ls DEBUG: Bucket 's3://stock': DEBUG: String 'XSHE/0/50/2008/XSHE-50-20080102' encoded to 'XSHE/0/50/2008/XSHE-50-20080102' DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Thu, 27 Aug 2015 09:35:51 +\n/stock/' DEBUG: CreateRequest: resource[uri]=/ DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Thu, 27 Aug 2015 09:35:51 +\n/stock/' DEBUG: Processing request, please wait... DEBUG: get_hostname(stock): stock.api2.cloud-datayes.com DEBUG: format_uri(): http://stock.api2.cloud-datayes.com/?prefix=XSHE/0/50/2008/XSHE-50-20080102&delimiter=/ WARNING: Retrying failed request: /?prefix=XSHE/0/50/2008/XSHE-50-20080102&delimiter=/ (timed out) WARNING: Waiting 3 sec... DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Thu, 27 Aug 2015 09:37:34 +\n/stock/' DEBUG: Processing request, please wait... DEBUG: get_hostname(stock): stock.api2.cloud-datayes.com DEBUG: format_uri(): http://stock.api2.cloud-datayes.com/?prefix=XSHE/0/50/2008/XSHE-50-20080102&delimiter=/ below was last 10 errors from "/var/log/riak-cs/console.log" 2015-08-27 17:35:40.085 [error] <0.27146.26>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: no_user_key 2015-08-27 17:37:34.744 [error] <0.27147.26>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: no_user_key 2015-08-27 17:37:49.356 [error] <0.27146.26>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: no_user_key 2015-08-27 17:39:49.249 [error] <0.27147.26>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: no_user_key 2015-08-27 17:39:54.811 [error] <0.27146.26>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: no_user_key On Wed, Aug 26, 2015 at 5:35 PM, Shunichi Shinohara wrote: > Sorry for delay and thanks for further information. > The log of Riak CS shows timeout in list query to Riak: >2015-08-24 16:59:49.717 [error] <0.18027.7> gen_fsm <0.18027.7> in state >waiting_list_keys terminated with reason: <<"timeout">> > > There are some kinds of possibility for timeout, e.g. hardware resource. > But if the timeout is occured by large number of objects (manifests) in > your > bucket, improvement from 1.4.0 may help. > It can be used by setting >{fold_objects_for_list_keys, true} > in your app.config's riak_cs section or executing >application:set_env(riak_cs, fold_objects_for_list_keys, true). > by attaching to riak-cs node. > For more information about it, please refer the original PR [1]. > > [1] https://github.com/basho/riak_cs/pull/600 > -- > Shunichi Shinohara > Basho Japan KK > > > On Wed, Aug 26, 2015 at 2:04 PM, Stanislav Vlasov > wrote: > > 2015-08-25 11:03 GMT+05:00 changmao wang : > >> Any ideas on this issue? > > > > Can you check credentials with another client? > > s3curl, for example? > > > > I got some bugs in s3cmd after debian upgrade, so if another client > > works, than s3cmd has bug. > > > >> On Mon, Aug 24, 2015 at 5:09 PM, changmao wang > > >> wrote: > >>> > >>> Please check attached file for details. > >>> > >>> On Mon, Aug 24, 2015 at 4:48 PM, Shunichi Shinohara > >>> wrote: > >>>> > >>>> Then, back to my first questions: > >>>> Could you provide results following commands with s3cfg1? > >>>> - s3cmd ls > >>>> - s3cmd info s3://stock > >>>> > >>>> From log file, gc index queries timed out again and again. > >>>> Not sure but it may be subtle situation... > >>>> > >>>> -- > >>>> Shunichi Shinohara > >>>> Basho Japan KK > >>>> > >>>> > >>>> On Mon, Aug 24, 2015 at 11:03 AM, changmao wang < > wang.chang...@gmail.com> > >>>> wrote: > >>>> > 1. root@cluster1-hd10:~# grep cs_root_host /etc/riak-cs/app.config > >>>> > {cs_root_host, "api2.cloud-datayes.com&
Re: s3cmd error: access to bucket was denied
root@cluster1-hd10:~# s3cmd -c s3cfg1 info s3://stock/XSHE/0/50/2008/XSHE-50-20080102 s3://stock/XSHE/0/50/2008/XSHE-50-20080102 (object): File size: 397535 Last mod: Thu, 05 Dec 2013 03:19:00 GMT MIME type: binary/octet-stream MD5 sum: feb2609ecfc9bb21549f2401a5c9477d ACL: stockwrite: FULL_CONTROL ACL: *anon*: READ URL: http://stock.s3.amazonaws.com/XSHE/0/50/2008/XSHE-50-20080102 s3cmd -c s3cfg1 ls s3://stock/XSHE/0/50/2008/XSHE-50-20080102 -d After running above command, I checked "/var/log/riak-cs/console.log" on all nodes, I got below error: 2015-09-02 09:45:03.405 [error] <0.22108.130>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: no_user_key 2015-09-02 09:45:03.581 [error] <0.31347.130>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: no_user_key root@cluster1-hd10:~# .. root@cluster1-hd10:~# s3cmd -c s3cfg1 ls 2013-12-03 08:18 s3://stock root@cluster1-hd10:~# s3cmd -c s3cfg1 mb s3://logfile ! An unexpected error has occurred. Please report the following lines to: s3tools-b...@lists.sourceforge.net ! Problem: AttributeError: 'NoneType' object has no attribute 'getchildren' S3cmd: 1.0.0 On Tue, Sep 1, 2015 at 8:57 AM, Shunichi Shinohara wrote: > Hmm.. it seems that situation is not good. > > GET Bucket (aka List Objects) of Riak CS uses a kind of coverage > operation of Riak (KV), > and it shows timeout. There may be some log at riak nodes. You have to > look into logs in > all nodes because all nodes participate in coverage operations. > > Could you try some further commands? > 1. for sanity check, >s3cmd -c s3cfg1 ls > 2. To try list objects for completely empty buckets, >s3cmd -c s3cfg1 mb s3:// >s3cmd -c s3cfg1 ls s3:// > 3. Another coverage operation, without riak cs >curl -v 'http://127.0.0.1:8098/buckets/foobar/keys?keys=true' ># 127.0.0.1 and 8098 should be changed to HTTP host and port ># (*NOT* PB host/port) of one of riak nodes > > Thanks, > Shino > > On Thu, Aug 27, 2015 at 6:41 PM, changmao wang > wrote: > > Shunichi, > > > > Just now, I followed your direction to change fold_objects_for_list_keys > to > > true, and restart riak-cs service. > > > > > > sed -i '/fold_objects_for_list_keys/ s/false/true/g' > > /etc/riak-cs/app.config; riak-cs restart > > > > After that, I run below command and got same error. > > > > s3cmd -c s3cfg1 ls s3://stock/XSHE/0/50/2008/XSHE-50-20080102 -d > > > > ... > > DEBUG: Unicodising 'ls' using UTF-8 > > DEBUG: Unicodising 's3://stock/XSHE/0/50/2008/XSHE-50-20080102' > > using UTF-8 > > DEBUG: Command: ls > > DEBUG: Bucket 's3://stock': > > DEBUG: String 'XSHE/0/50/2008/XSHE-50-20080102' encoded to > > 'XSHE/0/50/2008/XSHE-50-20080102' > > DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Thu, 27 Aug 2015 09:35:51 > > +\n/stock/' > > DEBUG: CreateRequest: resource[uri]=/ > > DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Thu, 27 Aug 2015 09:35:51 > > +\n/stock/' > > DEBUG: Processing request, please wait... > > DEBUG: get_hostname(stock): stock.api2.cloud-datayes.com > > DEBUG: format_uri(): > > > http://stock.api2.cloud-datayes.com/?prefix=XSHE/0/50/2008/XSHE-50-20080102&delimiter=/ > > WARNING: Retrying failed request: > > /?prefix=XSHE/0/50/2008/XSHE-50-20080102&delimiter=/ (timed out) > > WARNING: Waiting 3 sec... > > DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Thu, 27 Aug 2015 09:37:34 > > +\n/stock/' > > DEBUG: Processing request, please wait... > > DEBUG: get_hostname(stock): stock.api2.cloud-datayes.com > > DEBUG: format_uri(): > > > http://stock.api2.cloud-datayes.com/?prefix=XSHE/0/50/2008/XSHE-50-20080102&delimiter=/ > > > > below was last 10 errors from "/var/log/riak-cs/console.log" > > 2015-08-27 17:35:40.085 [error] > > <0.27146.26>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user > > record for s3 failed. Reason: no_user_key > > 2015-08-27 17:37:34.744 [error] > > <0.27147.26>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user > > record for s3 failed. Reason: no_user_key > > 2015-08-27 17:37:49.356 [error] > > <0.27146.26>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user > > record for s3 failed. Reason: no_user_key > > 2015-08-27 1
Re: s3cmd error: access to bucket was denied
A little strange thing happened as below: root@cluster1-hd10:~# s3cmd -c s3cfg1 del s3://stock/XSHE/0/50/2008/XSHE-50-20080102 s3://stock/XSHE/0/50/2008/XSHE-50-20080102 File s3://stock/XSHE/0/50/2008/XSHE-50-20080102 deleted On Wed, Sep 2, 2015 at 9:50 AM, changmao wang wrote: > root@cluster1-hd10:~# s3cmd -c s3cfg1 info > s3://stock/XSHE/0/50/2008/XSHE-50-20080102 > s3://stock/XSHE/0/50/2008/XSHE-50-20080102 (object): >File size: 397535 >Last mod: Thu, 05 Dec 2013 03:19:00 GMT >MIME type: binary/octet-stream >MD5 sum: feb2609ecfc9bb21549f2401a5c9477d >ACL: stockwrite: FULL_CONTROL >ACL: *anon*: READ >URL: > http://stock.s3.amazonaws.com/XSHE/0/50/2008/XSHE-50-20080102 > > s3cmd -c s3cfg1 ls s3://stock/XSHE/0/50/2008/XSHE-50-20080102 -d > > After running above command, I checked "/var/log/riak-cs/console.log" on > all nodes, I got below error: > 2015-09-02 09:45:03.405 [error] > <0.22108.130>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user > record for s3 failed. Reason: no_user_key > 2015-09-02 09:45:03.581 [error] > <0.31347.130>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user > record for s3 failed. Reason: no_user_key > root@cluster1-hd10:~# > .. > > root@cluster1-hd10:~# s3cmd -c s3cfg1 ls > 2013-12-03 08:18 s3://stock > root@cluster1-hd10:~# s3cmd -c s3cfg1 mb s3://logfile > > ! > An unexpected error has occurred. > Please report the following lines to: >s3tools-b...@lists.sourceforge.net > ! > > Problem: AttributeError: 'NoneType' object has no attribute 'getchildren' > S3cmd: 1.0.0 > > On Tue, Sep 1, 2015 at 8:57 AM, Shunichi Shinohara > wrote: > >> Hmm.. it seems that situation is not good. >> >> GET Bucket (aka List Objects) of Riak CS uses a kind of coverage >> operation of Riak (KV), >> and it shows timeout. There may be some log at riak nodes. You have to >> look into logs in >> all nodes because all nodes participate in coverage operations. >> >> Could you try some further commands? >> 1. for sanity check, >>s3cmd -c s3cfg1 ls >> 2. To try list objects for completely empty buckets, >>s3cmd -c s3cfg1 mb s3:// >>s3cmd -c s3cfg1 ls s3:// >> 3. Another coverage operation, without riak cs >>curl -v 'http://127.0.0.1:8098/buckets/foobar/keys?keys=true' >># 127.0.0.1 and 8098 should be changed to HTTP host and port >># (*NOT* PB host/port) of one of riak nodes >> >> Thanks, >> Shino >> >> On Thu, Aug 27, 2015 at 6:41 PM, changmao wang >> wrote: >> > Shunichi, >> > >> > Just now, I followed your direction to change >> fold_objects_for_list_keys to >> > true, and restart riak-cs service. >> > >> > >> > sed -i '/fold_objects_for_list_keys/ s/false/true/g' >> > /etc/riak-cs/app.config; riak-cs restart >> > >> > After that, I run below command and got same error. >> > >> > s3cmd -c s3cfg1 ls s3://stock/XSHE/0/50/2008/XSHE-50-20080102 -d >> > >> > ... >> > DEBUG: Unicodising 'ls' using UTF-8 >> > DEBUG: Unicodising 's3://stock/XSHE/0/50/2008/XSHE-50-20080102' >> > using UTF-8 >> > DEBUG: Command: ls >> > DEBUG: Bucket 's3://stock': >> > DEBUG: String 'XSHE/0/50/2008/XSHE-50-20080102' encoded to >> > 'XSHE/0/50/2008/XSHE-50-20080102' >> > DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Thu, 27 Aug 2015 09:35:51 >> > +\n/stock/' >> > DEBUG: CreateRequest: resource[uri]=/ >> > DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Thu, 27 Aug 2015 09:35:51 >> > +\n/stock/' >> > DEBUG: Processing request, please wait... >> > DEBUG: get_hostname(stock): stock.api2.cloud-datayes.com >> > DEBUG: format_uri(): >> > >> http://stock.api2.cloud-datayes.com/?prefix=XSHE/0/50/2008/XSHE-50-20080102&delimiter=/ >> > WARNING: Retrying failed request: >> > /?prefix=XSHE/0/50/2008/XSHE-50-20080102&delimiter=/ (timed out) >> > WARNING: Waiting 3 sec... >> > DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Thu, 27 Aug 2015 09:37:34 >> > +\n/stock/' >> > DEBUG: Processing request, please wait... >> > DEBUG: get_hostname(stock): stock.api2.cloud-datayes.com >> > DEBUG: format_uri(): >>