Re: Riak CS: Undeletable broken buckets
Hi Shunichi,
Enabling that temporarily then enabled me to remove the buckets by
first creating (s3cmd mb) and then removing (s3cmd rb) the problematic
buckets.
Thanks,
Toby
On 7 July 2014 19:38, Shunichi Shinohara wrote:
> Hi Toby,
>
> There is a rarely used option "disable_local_bucket_check" in Riak CS.
> I don't know it solves your case, please let me mention it.
>
> To use it, first set it in app.config of riak-cs (or
> application:set_env/3 in shell),
> {riak_cs, [...
>{disable_local_bucket_check, true},
>...]},
> Then create bucket as usual (e.g. s3cmd mb ...).
>
> These steps solves one of partial update patterns that Andrew mentioned.
>
> Thanks,
> Shino
>
> On Mon, Jul 7, 2014 at 2:12 PM, Toby Corkindale wrote:
>> Hi Andrew,
>> Thanks for the details.
>> The Puppet config should never have let it be setup with
>> allow_mult=false, but as this is a test cluster, it's possible
>> something went awry there at some point.
>>
>> If it's not really a bug that needs reporting then I can let it go.
>> Thanks,,
>> Toby
>>
>> On 7 July 2014 12:21, Andrew Stone wrote:
>>> Hi Toby,
>>>
>>> We've seen this scenario before. It occurs because riak-cs stores bucket
>>> information in 2 places on disk:
>>> 1) Inside the user record (for bucket permissions)
>>> 2) Inside a global list of buckets, since each bucket must be unique
>>>
>>> What has happened most likely is that the bucket is no longer stored for the
>>> given user, but still in the global list of bucket. It shows up in bucket
>>> lists, but the current user doesn't have permission to actually do anything
>>> with it. Essentially you have partially written (or partially deleted) data.
>>> I believe the only time we saw this was when Riak was configured with
>>> {allow_mult, false} which is an invalid setting when used with riak-cs.
>>> Riak-cs uses siblings intelligently to merge conflicting data, and without
>>> that it's possible to end up in these types of scenarios. Later versions of
>>> riak-cs should refuse to run with {allow_mult, false}. I'd check your riak
>>> config to see if that is the case here.
>>>
>>> We actually have scripts to detect and remove the bad buckets that we've
>>> used in support. We can probably get you a copy if you want. Just let me
>>> know. And make sure when running in production that allow_mult = true.
>>>
>>> -Andrew
>>>
>>>
>>>
>>> On Sun, Jul 6, 2014 at 9:59 PM, Toby Corkindale wrote:
Hi,
At some point we've managed to create a couple of buckets that don't
work and can't be deleted (in a development/testing cluster, not
production).
They show up with both 's3cmd ls' or by querying the HTTP API for a
user's buckets.
However attempting to list files in the bucket, or removing the
bucket, or recreating the bucket, fails.
It's not in a production cluster so it's not a huge concern to me, but
thought I'd report the bug here in case it's of interest to you.
Riak 1.4.9-1 and Riak-CS 1.4.5-1 on Ubuntu 12.04 LTS.
$ s3cmd ls
2014-02-07 00:07 s3://test5403
2013-12-13 07:25 s3://test9857
$ s3cmd ls s3://test5403
ERROR: Bucket 'test5403' does not exist
tobyc@adonai:~$ s3cmd ls s3://test9857
ERROR: Bucket 'test9857' does not exist
$ s3cmd rb s3://test5403
ERROR: Bucket 'test5403' does not exist
Bucket 's3://test5403/' removed
$ s3cmd ls
2014-02-07 00:07 s3://test5403
2013-12-13 07:25 s3://test9857
$ s3cmd mb s3://test5403
Bucket 's3://test5403/' created
$ s3cmd ls s3://test5403
ERROR: Bucket 'test5403' does not exist
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>>
>>
>>
>>
>> --
>> Turning and turning in the widening gyre
>> The falcon cannot hear the falconer
>> Things fall apart; the center cannot hold
>> Mere anarchy is loosed upon the world
>>
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
--
Turning and turning in the widening gyre
The falcon cannot hear the falconer
Things fall apart; the center cannot hold
Mere anarchy is loosed upon the world
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak CS: Undeletable broken buckets
Hi Toby,
There is a rarely used option "disable_local_bucket_check" in Riak CS.
I don't know it solves your case, please let me mention it.
To use it, first set it in app.config of riak-cs (or
application:set_env/3 in shell),
{riak_cs, [...
{disable_local_bucket_check, true},
...]},
Then create bucket as usual (e.g. s3cmd mb ...).
These steps solves one of partial update patterns that Andrew mentioned.
Thanks,
Shino
On Mon, Jul 7, 2014 at 2:12 PM, Toby Corkindale wrote:
> Hi Andrew,
> Thanks for the details.
> The Puppet config should never have let it be setup with
> allow_mult=false, but as this is a test cluster, it's possible
> something went awry there at some point.
>
> If it's not really a bug that needs reporting then I can let it go.
> Thanks,,
> Toby
>
> On 7 July 2014 12:21, Andrew Stone wrote:
>> Hi Toby,
>>
>> We've seen this scenario before. It occurs because riak-cs stores bucket
>> information in 2 places on disk:
>> 1) Inside the user record (for bucket permissions)
>> 2) Inside a global list of buckets, since each bucket must be unique
>>
>> What has happened most likely is that the bucket is no longer stored for the
>> given user, but still in the global list of bucket. It shows up in bucket
>> lists, but the current user doesn't have permission to actually do anything
>> with it. Essentially you have partially written (or partially deleted) data.
>> I believe the only time we saw this was when Riak was configured with
>> {allow_mult, false} which is an invalid setting when used with riak-cs.
>> Riak-cs uses siblings intelligently to merge conflicting data, and without
>> that it's possible to end up in these types of scenarios. Later versions of
>> riak-cs should refuse to run with {allow_mult, false}. I'd check your riak
>> config to see if that is the case here.
>>
>> We actually have scripts to detect and remove the bad buckets that we've
>> used in support. We can probably get you a copy if you want. Just let me
>> know. And make sure when running in production that allow_mult = true.
>>
>> -Andrew
>>
>>
>>
>> On Sun, Jul 6, 2014 at 9:59 PM, Toby Corkindale wrote:
>>>
>>> Hi,
>>> At some point we've managed to create a couple of buckets that don't
>>> work and can't be deleted (in a development/testing cluster, not
>>> production).
>>> They show up with both 's3cmd ls' or by querying the HTTP API for a
>>> user's buckets.
>>> However attempting to list files in the bucket, or removing the
>>> bucket, or recreating the bucket, fails.
>>>
>>> It's not in a production cluster so it's not a huge concern to me, but
>>> thought I'd report the bug here in case it's of interest to you.
>>> Riak 1.4.9-1 and Riak-CS 1.4.5-1 on Ubuntu 12.04 LTS.
>>>
>>> $ s3cmd ls
>>> 2014-02-07 00:07 s3://test5403
>>> 2013-12-13 07:25 s3://test9857
>>>
>>> $ s3cmd ls s3://test5403
>>> ERROR: Bucket 'test5403' does not exist
>>> tobyc@adonai:~$ s3cmd ls s3://test9857
>>> ERROR: Bucket 'test9857' does not exist
>>>
>>> $ s3cmd rb s3://test5403
>>> ERROR: Bucket 'test5403' does not exist
>>> Bucket 's3://test5403/' removed
>>>
>>> $ s3cmd ls
>>> 2014-02-07 00:07 s3://test5403
>>> 2013-12-13 07:25 s3://test9857
>>>
>>> $ s3cmd mb s3://test5403
>>> Bucket 's3://test5403/' created
>>>
>>> $ s3cmd ls s3://test5403
>>> ERROR: Bucket 'test5403' does not exist
>>>
>>> ___
>>> riak-users mailing list
>>> riak-users@lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
>
>
> --
> Turning and turning in the widening gyre
> The falcon cannot hear the falconer
> Things fall apart; the center cannot hold
> Mere anarchy is loosed upon the world
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak CS: Undeletable broken buckets
Hi Andrew,
Thanks for the details.
The Puppet config should never have let it be setup with
allow_mult=false, but as this is a test cluster, it's possible
something went awry there at some point.
If it's not really a bug that needs reporting then I can let it go.
Thanks,,
Toby
On 7 July 2014 12:21, Andrew Stone wrote:
> Hi Toby,
>
> We've seen this scenario before. It occurs because riak-cs stores bucket
> information in 2 places on disk:
> 1) Inside the user record (for bucket permissions)
> 2) Inside a global list of buckets, since each bucket must be unique
>
> What has happened most likely is that the bucket is no longer stored for the
> given user, but still in the global list of bucket. It shows up in bucket
> lists, but the current user doesn't have permission to actually do anything
> with it. Essentially you have partially written (or partially deleted) data.
> I believe the only time we saw this was when Riak was configured with
> {allow_mult, false} which is an invalid setting when used with riak-cs.
> Riak-cs uses siblings intelligently to merge conflicting data, and without
> that it's possible to end up in these types of scenarios. Later versions of
> riak-cs should refuse to run with {allow_mult, false}. I'd check your riak
> config to see if that is the case here.
>
> We actually have scripts to detect and remove the bad buckets that we've
> used in support. We can probably get you a copy if you want. Just let me
> know. And make sure when running in production that allow_mult = true.
>
> -Andrew
>
>
>
> On Sun, Jul 6, 2014 at 9:59 PM, Toby Corkindale wrote:
>>
>> Hi,
>> At some point we've managed to create a couple of buckets that don't
>> work and can't be deleted (in a development/testing cluster, not
>> production).
>> They show up with both 's3cmd ls' or by querying the HTTP API for a
>> user's buckets.
>> However attempting to list files in the bucket, or removing the
>> bucket, or recreating the bucket, fails.
>>
>> It's not in a production cluster so it's not a huge concern to me, but
>> thought I'd report the bug here in case it's of interest to you.
>> Riak 1.4.9-1 and Riak-CS 1.4.5-1 on Ubuntu 12.04 LTS.
>>
>> $ s3cmd ls
>> 2014-02-07 00:07 s3://test5403
>> 2013-12-13 07:25 s3://test9857
>>
>> $ s3cmd ls s3://test5403
>> ERROR: Bucket 'test5403' does not exist
>> tobyc@adonai:~$ s3cmd ls s3://test9857
>> ERROR: Bucket 'test9857' does not exist
>>
>> $ s3cmd rb s3://test5403
>> ERROR: Bucket 'test5403' does not exist
>> Bucket 's3://test5403/' removed
>>
>> $ s3cmd ls
>> 2014-02-07 00:07 s3://test5403
>> 2013-12-13 07:25 s3://test9857
>>
>> $ s3cmd mb s3://test5403
>> Bucket 's3://test5403/' created
>>
>> $ s3cmd ls s3://test5403
>> ERROR: Bucket 'test5403' does not exist
>>
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
--
Turning and turning in the widening gyre
The falcon cannot hear the falconer
Things fall apart; the center cannot hold
Mere anarchy is loosed upon the world
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak CS: Undeletable broken buckets
Hi Toby,
We've seen this scenario before. It occurs because riak-cs stores bucket
information in 2 places on disk:
1) Inside the user record (for bucket permissions)
2) Inside a global list of buckets, since each bucket must be unique
What has happened most likely is that the bucket is no longer stored for
the given user, but still in the global list of bucket. It shows up in
bucket lists, but the current user doesn't have permission to actually do
anything with it. Essentially you have partially written (or partially
deleted) data. I believe the only time we saw this was when Riak was
configured with {allow_mult, false} which is an invalid setting when used
with riak-cs. Riak-cs uses siblings intelligently to merge conflicting
data, and without that it's possible to end up in these types of scenarios.
Later versions of riak-cs should refuse to run with {allow_mult, false}.
I'd check your riak config to see if that is the case here.
We actually have scripts to detect and remove the bad buckets that we've
used in support. We can probably get you a copy if you want. Just let me
know. And make sure when running in production that allow_mult = true.
-Andrew
On Sun, Jul 6, 2014 at 9:59 PM, Toby Corkindale wrote:
> Hi,
> At some point we've managed to create a couple of buckets that don't
> work and can't be deleted (in a development/testing cluster, not
> production).
> They show up with both 's3cmd ls' or by querying the HTTP API for a
> user's buckets.
> However attempting to list files in the bucket, or removing the
> bucket, or recreating the bucket, fails.
>
> It's not in a production cluster so it's not a huge concern to me, but
> thought I'd report the bug here in case it's of interest to you.
> Riak 1.4.9-1 and Riak-CS 1.4.5-1 on Ubuntu 12.04 LTS.
>
> $ s3cmd ls
> 2014-02-07 00:07 s3://test5403
> 2013-12-13 07:25 s3://test9857
>
> $ s3cmd ls s3://test5403
> ERROR: Bucket 'test5403' does not exist
> tobyc@adonai:~$ s3cmd ls s3://test9857
> ERROR: Bucket 'test9857' does not exist
>
> $ s3cmd rb s3://test5403
> ERROR: Bucket 'test5403' does not exist
> Bucket 's3://test5403/' removed
>
> $ s3cmd ls
> 2014-02-07 00:07 s3://test5403
> 2013-12-13 07:25 s3://test9857
>
> $ s3cmd mb s3://test5403
> Bucket 's3://test5403/' created
>
> $ s3cmd ls s3://test5403
> ERROR: Bucket 'test5403' does not exist
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
