Re: Invalid hintfiles

[Andrew Stone]

Good to hear. :)


On Wed, Sep 18, 2013 at 11:52 PM, Toby Corkindale <
toby.corkind...@strategicdata.com.au> wrote:

> Hi Andrew,
> Thanks for that -- so far things are looking stable after making that
> change.
>
> -T
>
>
> On 19/09/13 13:27, Andrew Stone wrote:
>
>> Hi Toby,
>>
>> Can you try raising the pb_backlog to 128 in your riak app.config on
>> each node. It's likely those disconnect errors are left over from the
>> stampede of connections from the CS connection pool on startup. For one
>> reason or another the resets don't come through and the hanging
>> disconnected socket isn't discovered until the next send attempt.
>>
>> -Andrew
>>
>>
>> On Wed, Sep 18, 2013 at 10:31 PM, Toby Corkindale
>> 
>> >
>> wrote:
>>
>> On 19/09/13 11:17, Luke Bakken wrote:
>>
>> The "error: disconnected" message is a good clue. If you can
>> provide
>> log files that may point to the cause.
>>
>>
>> See below for logs from riak and riak cs, for a roughly five minute
>> window during which I'd fiddled with the load balancer and DNS to
>> try and isolate this server from other requests, and only send mine
>> to it.
>>
>> Unfortunately I really don't see much in there apart from the
>> disconnected messages. I did note that there were warnings about
>> system hitting high watermarks for memory at one point though.
>> That's a bit odd, as the machine wasn't near max capacity at any
>> point - it's only 8GB, but 'free' was reporting that most was being
>> used just for buffers and cache at the time; as follows:
>>
>> ie.
>>   total   used   free sharedbuffers
>> cached
>> Mem:   817641238034044373008  0  34112
>>   3052340
>> -/+ buffers/cache: 7169527459460
>> Swap:  3903484  03903484
>>
>>
>>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Invalid hintfiles

[Toby Corkindale]

Hi Andrew,
Thanks for that -- so far things are looking stable after making that 
change.


-T

On 19/09/13 13:27, Andrew Stone wrote:

Hi Toby,

Can you try raising the pb_backlog to 128 in your riak app.config on
each node. It's likely those disconnect errors are left over from the
stampede of connections from the CS connection pool on startup. For one
reason or another the resets don't come through and the hanging
disconnected socket isn't discovered until the next send attempt.

-Andrew


On Wed, Sep 18, 2013 at 10:31 PM, Toby Corkindale
mailto:toby.corkind...@strategicdata.com.au>> wrote:

On 19/09/13 11:17, Luke Bakken wrote:

The "error: disconnected" message is a good clue. If you can provide
log files that may point to the cause.


See below for logs from riak and riak cs, for a roughly five minute
window during which I'd fiddled with the load balancer and DNS to
try and isolate this server from other requests, and only send mine
to it.

Unfortunately I really don't see much in there apart from the
disconnected messages. I did note that there were warnings about
system hitting high watermarks for memory at one point though.
That's a bit odd, as the machine wasn't near max capacity at any
point - it's only 8GB, but 'free' was reporting that most was being
used just for buffers and cache at the time; as follows:

ie.
  total   used   free sharedbuffers
cached
Mem:   817641238034044373008  0  34112
  3052340
-/+ buffers/cache: 7169527459460
Swap:  3903484  03903484





___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: ring resizing feature?

[Alexander Sicular]

That's about the rights of it. As I was reading your message I did some quick 
math and sure enough you already put that same number down. I've heard told 
that the max size for any Erlang network, of which Riak is most assuredly, is 
200 Erlang VM's. Extrapolating from there, considering that I've also heard 
told that a good sizing rule of thumb is 10 vnodes per node, you find yourself 
with ~2000 vnodes. Accounting for another wives' tale that the ring size should 
be powers of 2, 2048 looks to be the upper bound. 

Wives' tales, rumors and general misinformation aside, I've not tested those 
numbers in any way whatsoever but I would love to see an EC2 script give it a 
shot. 

-Alexander

@siculars
http://siculars.posthaven.com

Sent from my iRotaryPhone

On Sep 18, 2013, at 3:44, pseudo-n...@laposte.net wrote:

> I have seen mentioned that 2048 and higher could importantly increase the 
> volume of gossiping, what is your experience with it?

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Invalid hintfiles

[Andrew Stone]

Hi Toby,

Can you try raising the pb_backlog to 128 in your riak app.config on each
node. It's likely those disconnect errors are left over from the stampede
of connections from the CS connection pool on startup. For one reason or
another the resets don't come through and the hanging disconnected socket
isn't discovered until the next send attempt.

-Andrew


On Wed, Sep 18, 2013 at 10:31 PM, Toby Corkindale <
toby.corkind...@strategicdata.com.au> wrote:

> On 19/09/13 11:17, Luke Bakken wrote:
>
>> The "error: disconnected" message is a good clue. If you can provide
>> log files that may point to the cause.
>>
>
> See below for logs from riak and riak cs, for a roughly five minute window
> during which I'd fiddled with the load balancer and DNS to try and isolate
> this server from other requests, and only send mine to it.
>
> Unfortunately I really don't see much in there apart from the disconnected
> messages. I did note that there were warnings about system hitting high
> watermarks for memory at one point though. That's a bit odd, as the machine
> wasn't near max capacity at any point - it's only 8GB, but 'free' was
> reporting that most was being used just for buffers and cache at the time;
> as follows:
>
> ie.
>  total   used   free sharedbuffers cached
> Mem:   817641238034044373008  0  341123052340
> -/+ buffers/cache: 7169527459460
> Swap:  3903484  03903484
>
>
> === erlang.log (riak cs) =
>
> = Thu Sep 19 11:58:46 EST 2013
> 11:58:46.287 [info] alarm_handler: {set,{system_memory_high_**
> watermark,[]}}^M
> 12:00:47.864 [info] webmachine_log_handler: closing log file:
> "/var/log/riak-cs/access.log"
> ^M
> 12:00:47.865 [info] opening log file: "/var/log/riak-cs/access.log.**
> 2013_09_19_02"
> ^M
> 12:00:50.081 [error] Retrieval of user record for s3 failed. Reason:
> disconnected^M
> 12:00:51.013 [error] Retrieval of user record for s3 failed. Reason:
> disconnected^M
> 12:00:51.406 [error] Retrieval of user record for s3 failed. Reason:
> disconnected^M
> 12:01:49.320 [error] Retrieval of user record for s3 failed. Reason:
> disconnected^M
> 12:02:04.662 [error] Retrieval of user record for s3 failed. Reason:
> disconnected^M
> 12:03:37.330 [error] Retrieval of user record for s3 failed. Reason:
> disconnected^M
> 12:06:24.345 [error] Retrieval of user record for s3 failed. Reason:
> disconnected^M
> 12:06:41.670 [error] Retrieval of user record for s3 failed. Reason:
> disconnected^M
> 12:06:46.554 [info] Finished garbage collection: 0 seconds, 0 batch_count,
> 0 batch_skips, 0 manif_count, 0 block_count^M
> 12:11:46.300 [info] alarm_handler: {clear,system_memory_high_**
> watermark}^M
> 12:16:46.305 [info] alarm_handler: {set,{system_memory_high_**
> watermark,[]}}^M
> 12:18:46.307 [info] alarm_handler: {clear,system_memory_high_**
> watermark}^M
> ==**==
>
>
> = console.log (riak cs) ===
> 2013-09-19 12:00:47.865 [info] <0.94.0> opening log file:
> "/var/log/riak-cs/access.log.**2013_09_19_02"
>
> 2013-09-19 12:00:50.081 [error] 
> <0.7092.0>@riak_cs_wm_common:**maybe_create_user:223
> Retrieval of user record for s3 failed. Reason: disconnected
> 2013-09-19 12:00:51.013 [error] 
> <0.10406.0>@riak_cs_wm_common:**maybe_create_user:223
> Retrieval of user record for s3 failed. Reason: disconnected
> 2013-09-19 12:00:51.406 [error] 
> <0.11179.0>@riak_cs_wm_common:**maybe_create_user:223
> Retrieval of user record for s3 failed. Reason: disconnected
> 2013-09-19 12:01:49.320 [error] 
> <0.11197.0>@riak_cs_wm_common:**maybe_create_user:223
> Retrieval of user record for s3 failed. Reason: disconnected
> 2013-09-19 12:02:04.662 [error] 
> <0.9667.0>@riak_cs_wm_common:**maybe_create_user:223
> Retrieval of user record for s3 failed. Reason: disconnected
> 2013-09-19 12:03:37.330 [error] 
> <0.11343.0>@riak_cs_wm_common:**maybe_create_user:223
> Retrieval of user record for s3 failed. Reason: disconnected
> 2013-09-19 12:06:24.345 [error] 
> <0.11589.0>@riak_cs_wm_common:**maybe_create_user:223
> Retrieval of user record for s3 failed. Reason: disconnected
> 2013-09-19 12:06:41.670 [error] 
> <0.11215.0>@riak_cs_wm_common:**maybe_create_user:223
> Retrieval of user record for s3 failed. Reason: disconnected
> 2013-09-19 12:06:46.554 [info] 
> <0.274.0>@riak_cs_gc_d:**fetching_next_fileset:162
> Finished garbage collection: 0 seconds, 0 batch_count, 0 batch_skips, 0
> manif_count, 0 block_count
> 2013-09-19 12:11:46.300 [info] <0.41.0> alarm_handler:
> {clear,system_memory_high_**watermark}
> 2013-09-19 12:16:46.305 [info] <0.41.0> alarm_handler:
> {set,{system_memory_high_**watermark,[]}}
> 2013-09-19 12:18:46.307 [info] <0.41.0> alarm_handler:
> {clear,system_memory_high_**watermark}
> ==**
>
>
>
> === error.log (riak cs) 
> 2013-09-19 12:00:50.081 [error] 
> <0.7092.0>@riak_cs

Re: Invalid hintfiles

[Toby Corkindale]

On 19/09/13 11:17, Luke Bakken wrote:

The "error: disconnected" message is a good clue. If you can provide
log files that may point to the cause.


See below for logs from riak and riak cs, for a roughly five minute 
window during which I'd fiddled with the load balancer and DNS to try 
and isolate this server from other requests, and only send mine to it.


Unfortunately I really don't see much in there apart from the 
disconnected messages. I did note that there were warnings about system 
hitting high watermarks for memory at one point though. That's a bit 
odd, as the machine wasn't near max capacity at any point - it's only 
8GB, but 'free' was reporting that most was being used just for buffers 
and cache at the time; as follows:


ie.
 total   used   free sharedbuffers cached
Mem:   817641238034044373008  0  341123052340
-/+ buffers/cache: 7169527459460
Swap:  3903484  03903484


=== erlang.log (riak cs) =

= Thu Sep 19 11:58:46 EST 2013
11:58:46.287 [info] alarm_handler: {set,{system_memory_high_watermark,[]}}^M
12:00:47.864 [info] webmachine_log_handler: closing log file: 
"/var/log/riak-cs/access.log"

^M
12:00:47.865 [info] opening log file: 
"/var/log/riak-cs/access.log.2013_09_19_02"

^M
12:00:50.081 [error] Retrieval of user record for s3 failed. Reason: 
disconnected^M
12:00:51.013 [error] Retrieval of user record for s3 failed. Reason: 
disconnected^M
12:00:51.406 [error] Retrieval of user record for s3 failed. Reason: 
disconnected^M
12:01:49.320 [error] Retrieval of user record for s3 failed. Reason: 
disconnected^M
12:02:04.662 [error] Retrieval of user record for s3 failed. Reason: 
disconnected^M
12:03:37.330 [error] Retrieval of user record for s3 failed. Reason: 
disconnected^M
12:06:24.345 [error] Retrieval of user record for s3 failed. Reason: 
disconnected^M
12:06:41.670 [error] Retrieval of user record for s3 failed. Reason: 
disconnected^M
12:06:46.554 [info] Finished garbage collection: 0 seconds, 0 
batch_count, 0 batch_skips, 0 manif_count, 0 block_count^M

12:11:46.300 [info] alarm_handler: {clear,system_memory_high_watermark}^M
12:16:46.305 [info] alarm_handler: {set,{system_memory_high_watermark,[]}}^M
12:18:46.307 [info] alarm_handler: {clear,system_memory_high_watermark}^M



= console.log (riak cs) ===
2013-09-19 12:00:47.865 [info] <0.94.0> opening log file: 
"/var/log/riak-cs/access.log.2013_09_19_02"


2013-09-19 12:00:50.081 [error] 
<0.7092.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user 
record for s3 failed. Reason: disconnected
2013-09-19 12:00:51.013 [error] 
<0.10406.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user 
record for s3 failed. Reason: disconnected
2013-09-19 12:00:51.406 [error] 
<0.11179.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user 
record for s3 failed. Reason: disconnected
2013-09-19 12:01:49.320 [error] 
<0.11197.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user 
record for s3 failed. Reason: disconnected
2013-09-19 12:02:04.662 [error] 
<0.9667.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user 
record for s3 failed. Reason: disconnected
2013-09-19 12:03:37.330 [error] 
<0.11343.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user 
record for s3 failed. Reason: disconnected
2013-09-19 12:06:24.345 [error] 
<0.11589.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user 
record for s3 failed. Reason: disconnected
2013-09-19 12:06:41.670 [error] 
<0.11215.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user 
record for s3 failed. Reason: disconnected
2013-09-19 12:06:46.554 [info] 
<0.274.0>@riak_cs_gc_d:fetching_next_fileset:162 Finished garbage 
collection: 0 seconds, 0 batch_count, 0 batch_skips, 0 manif_count, 0 
block_count
2013-09-19 12:11:46.300 [info] <0.41.0> alarm_handler: 
{clear,system_memory_high_watermark}
2013-09-19 12:16:46.305 [info] <0.41.0> alarm_handler: 
{set,{system_memory_high_watermark,[]}}
2013-09-19 12:18:46.307 [info] <0.41.0> alarm_handler: 
{clear,system_memory_high_watermark}

==



=== error.log (riak cs) 
2013-09-19 12:00:50.081 [error] 
<0.7092.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user 
record for s3 failed. Reason: disconnected
2013-09-19 12:00:51.013 [error] 
<0.10406.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user 
record for s3 failed. Reason: disconnected
2013-09-19 12:00:51.406 [error] 
<0.11179.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user 
record for s3 failed. Reason: disconnected
2013-09-19 12:01:49.320 [error] 
<0.11197.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user 
record for s3 failed. Reason: disconnected
2013-09-19 12:02:04.662 [error] 
<0.9667.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user 
record for s3 failed. Reason:

Re: Invalid hintfiles

[Seth Thomas]

Toby,Although I haven't traced a root cause just yet I've had several reports in IRC of odd behavior when using nginx as a load balancer for Riak CS. In almost every case, removing nginx from the equation and switching to haproxy or pointing directly at the nodes in the application has resolved issues. It's anecdotal until I can actually determine a root cause but it's worth testing out if you're experiencing trouble. -- Seth ThomasOn September 18, 2013 at 8:48:27 PM, Toby Corkindale (toby.corkind...@strategicdata.com.au) wrote: On 19/09/13 11:17, Luke Bakken wrote:
> Hi Toby,
>
> Invalid hint files won't cause Riak to fail requests - there must have
> been something else happening. Hint files are used by Riak to speed
> start time when loading a large key set.
>
> You mentioned a "load-balancer pool" - are you using something like
> HAProxy to load-balance requests to your Riak CS cluster?
>
> The "error: disconnected" message is a good clue. If you can provide
> log files that may point to the cause.

Hi Luke,
I'm still seeing quite a few failed requests. I've been chasing the  
hintfiles but I guess that was a red herring.

We're using nginx to load balance requests to Riak CS.
I tried going directly to each node in turn, and it didn't show that any  
one node was reliably failing every request.

Hitting one server just now came up with OK/403/403/OK/OK.
Trying another was OK/OK/OK/OK/403 though.

Here's some logs from riak-cs:

error.log:2013-09-19 11:37:02.242 [error]  
<0.5105.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user  
record for s3 failed. Reason: disconnected

There wasn't anything immediately either side of that. The riak logs for  
the same minute on that server likewise do not have anything.

There's quite a lot of free memory on the servers; they have 32000 file  
handles available.

Toby


> On Wed, Sep 18, 2013 at 4:29 PM, Toby Corkindale
>  wrote:
>> I found one Riak server was reporting a lot of errors like
>> [error] <0.808.0> Hintfile
>> '/var/lib/riak/bitcask/68507889249886074290797726533575766546371837952/3.bitcask.hint'
>> invalid
>>
>> And the Riak CS logs contained a lot of messages about being unable to
>> retrieve s3 user details because "error: disconnected"
>>
>> I think I've blown away the bad hintfiles and have had them repaired from
>> other replicas now, and I haven't seen any more errors for a little while.
>>
>> I'm not sure what caused those to become invalid.
>> Just a thought, but would be good if Riak could automatically repair them
>> rather than failing requests.
>>
>> Cheer,s
>> Toby
>>
>> On 19/09/13 08:42, Toby Corkindale wrote:
>>>
>>> Ah, hold on.. have just discovered that rather than it being deletion
>>> calls, it seems to just be every X calls of any sort.. sounds like one
>>> of the servers in the load-balancer pool must be misconfigured somehow,
>>> but the rest are OK.
>>>
>>> On 19/09/13 08:34, Toby Corkindale wrote:

 I've just upgraded from Riak CS 1.3.1 to 1.4.1

 Using s3cmd to test a few things, I've found some odd behaviour.
 Creating a bucket and putting a file works just fine, eg:

 s3cmd mb s3://test
 s3cmd put README s3://test
 s3cmd get s3://test/README

 However if I try to delete a file or bucket, it throws an error:

 s3cmd del s3://test/README
 ERROR: S3 error: 403 (InvalidAccessKeyId): The AWS Access Key Id you
 provided does not exist in our records.

 s3cmd rb s3://test
 ERROR: S3 error: 403 (InvalidAccessKeyId): The AWS Access Key Id you
 provided does not exist in our records.


 Have I messed something up during the upgrade, or is this a bug in 1.4.1?


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Invalid hintfiles

[Toby Corkindale]

On 19/09/13 11:17, Luke Bakken wrote:

Hi Toby,

Invalid hint files won't cause Riak to fail requests - there must have
been something else happening. Hint files are used by Riak to speed
start time when loading a large key set.

You mentioned a "load-balancer pool" - are you using something like
HAProxy to load-balance requests to your Riak CS cluster?

The "error: disconnected" message is a good clue. If you can provide
log files that may point to the cause.


Hi Luke,
I'm still seeing quite a few failed requests. I've been chasing the 
hintfiles but I guess that was a red herring.


We're using nginx to load balance requests to Riak CS.
I tried going directly to each node in turn, and it didn't show that any 
one node was reliably failing every request.


Hitting one server just now came up with OK/403/403/OK/OK.
Trying another was OK/OK/OK/OK/403 though.

Here's some logs from riak-cs:

error.log:2013-09-19 11:37:02.242 [error] 
<0.5105.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user 
record for s3 failed. Reason: disconnected


There wasn't anything immediately either side of that. The riak logs for 
the same minute on that server likewise do not have anything.


There's quite a lot of free memory on the servers; they have 32000 file 
handles available.


Toby



On Wed, Sep 18, 2013 at 4:29 PM, Toby Corkindale
 wrote:

I found one Riak server was reporting a lot of errors like
[error] <0.808.0> Hintfile
'/var/lib/riak/bitcask/68507889249886074290797726533575766546371837952/3.bitcask.hint'
invalid

And the Riak CS logs contained a lot of messages about being unable to
retrieve s3 user details because "error: disconnected"

I think I've blown away the bad hintfiles and have had them repaired from
other replicas now, and I haven't seen any more errors for a little while.

I'm not sure what caused those to become invalid.
Just a thought, but would be good if Riak could automatically repair them
rather than failing requests.

Cheer,s
Toby

On 19/09/13 08:42, Toby Corkindale wrote:


Ah, hold on.. have just discovered that rather than it being deletion
calls, it seems to just be every X calls of any sort.. sounds like one
of the servers in the load-balancer pool must be misconfigured somehow,
but the rest are OK.

On 19/09/13 08:34, Toby Corkindale wrote:


I've just upgraded from Riak CS 1.3.1 to 1.4.1

Using s3cmd to test a few things, I've found some odd behaviour.
Creating a bucket and putting a file works just fine, eg:

s3cmd mb s3://test
s3cmd put README s3://test
s3cmd get s3://test/README

However if I try to delete a file or bucket, it throws an error:

s3cmd del s3://test/README
ERROR: S3 error: 403 (InvalidAccessKeyId): The AWS Access Key Id you
provided does not exist in our records.

s3cmd rb s3://test
ERROR: S3 error: 403 (InvalidAccessKeyId): The AWS Access Key Id you
provided does not exist in our records.


Have I messed something up during the upgrade, or is this a bug in 1.4.1?



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Invalid hintfiles (was: Riak CS 1.4.1 bug? InvalidAccessKeyId reported for deletion calls)

[Luke Bakken]

Hi Toby,

Invalid hint files won't cause Riak to fail requests - there must have
been something else happening. Hint files are used by Riak to speed
start time when loading a large key set.

You mentioned a "load-balancer pool" - are you using something like
HAProxy to load-balance requests to your Riak CS cluster?

The "error: disconnected" message is a good clue. If you can provide
log files that may point to the cause.

--
Luke Bakken
CSE
lbak...@basho.com


On Wed, Sep 18, 2013 at 4:29 PM, Toby Corkindale
 wrote:
> I found one Riak server was reporting a lot of errors like
> [error] <0.808.0> Hintfile
> '/var/lib/riak/bitcask/68507889249886074290797726533575766546371837952/3.bitcask.hint'
> invalid
>
> And the Riak CS logs contained a lot of messages about being unable to
> retrieve s3 user details because "error: disconnected"
>
> I think I've blown away the bad hintfiles and have had them repaired from
> other replicas now, and I haven't seen any more errors for a little while.
>
> I'm not sure what caused those to become invalid.
> Just a thought, but would be good if Riak could automatically repair them
> rather than failing requests.
>
> Cheer,s
> Toby
>
> On 19/09/13 08:42, Toby Corkindale wrote:
>>
>> Ah, hold on.. have just discovered that rather than it being deletion
>> calls, it seems to just be every X calls of any sort.. sounds like one
>> of the servers in the load-balancer pool must be misconfigured somehow,
>> but the rest are OK.
>>
>> On 19/09/13 08:34, Toby Corkindale wrote:
>>>
>>> I've just upgraded from Riak CS 1.3.1 to 1.4.1
>>>
>>> Using s3cmd to test a few things, I've found some odd behaviour.
>>> Creating a bucket and putting a file works just fine, eg:
>>>
>>> s3cmd mb s3://test
>>> s3cmd put README s3://test
>>> s3cmd get s3://test/README
>>>
>>> However if I try to delete a file or bucket, it throws an error:
>>>
>>> s3cmd del s3://test/README
>>> ERROR: S3 error: 403 (InvalidAccessKeyId): The AWS Access Key Id you
>>> provided does not exist in our records.
>>>
>>> s3cmd rb s3://test
>>> ERROR: S3 error: 403 (InvalidAccessKeyId): The AWS Access Key Id you
>>> provided does not exist in our records.
>>>
>>>
>>> Have I messed something up during the upgrade, or is this a bug in 1.4.1?
>>>
>>> ta
>>> Toby
>>>
>>> ___
>>> riak-users mailing list
>>> riak-users@lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>>
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Invalid hintfiles (was: Riak CS 1.4.1 bug? InvalidAccessKeyId reported for deletion calls)

[Toby Corkindale]

I found one Riak server was reporting a lot of errors like
[error] <0.808.0> Hintfile 
'/var/lib/riak/bitcask/68507889249886074290797726533575766546371837952/3.bitcask.hint' 
invalid


And the Riak CS logs contained a lot of messages about being unable to 
retrieve s3 user details because "error: disconnected"


I think I've blown away the bad hintfiles and have had them repaired 
from other replicas now, and I haven't seen any more errors for a little 
while.


I'm not sure what caused those to become invalid.
Just a thought, but would be good if Riak could automatically repair 
them rather than failing requests.


Cheer,s
Toby

On 19/09/13 08:42, Toby Corkindale wrote:

Ah, hold on.. have just discovered that rather than it being deletion
calls, it seems to just be every X calls of any sort.. sounds like one
of the servers in the load-balancer pool must be misconfigured somehow,
but the rest are OK.

On 19/09/13 08:34, Toby Corkindale wrote:

I've just upgraded from Riak CS 1.3.1 to 1.4.1

Using s3cmd to test a few things, I've found some odd behaviour.
Creating a bucket and putting a file works just fine, eg:

s3cmd mb s3://test
s3cmd put README s3://test
s3cmd get s3://test/README

However if I try to delete a file or bucket, it throws an error:

s3cmd del s3://test/README
ERROR: S3 error: 403 (InvalidAccessKeyId): The AWS Access Key Id you
provided does not exist in our records.

s3cmd rb s3://test
ERROR: S3 error: 403 (InvalidAccessKeyId): The AWS Access Key Id you
provided does not exist in our records.


Have I messed something up during the upgrade, or is this a bug in 1.4.1?

ta
Toby

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: 503 Error

[Luke Bakken]

Hi Fred -

Did you try out "curl" to delete an object, rather than using the
python library?

Also - are you using Riak search? I'm trying to figure out some of the
errors I see in the logs. If you're on Riak 1.4 or later, could you
run "riak-debug" on each of your nodes and send me the archives that
are produced?

--
Luke Bakken
CSE
lbak...@basho.com


On Wed, Sep 18, 2013 at 8:57 AM, Wolfinger, Fred
 wrote:
> Hey Luke,
>
> Thanks so much for the reply. I have had a few people here looking at this
> and have never seen anything like it. When I run the delete script, it
> does fine for a while, then seems to crap out on an entry for whatever
> reason. However, the errors you will see in these logs seem to happen even
> during successful delete calls.
>
> I look forward to your feedback,
>
> Fred
>
> On 9/18/13 11:48 AM, "Luke Bakken"  wrote:
>
>>Hi Fred,
>>
>>Could you try deleting an object using an HTTP client app like "curl"?
>>Instructions for doing so are located here:
>>http://docs.basho.com/riak/latest/dev/using/basics/
>>
>>If that is not successful could you attach an archive of all of the
>>console.log* files from each node in your cluster? They should be
>>located in /var/log/riak.
>>
>>Thanks,
>>Luke Bakken
>>CSE
>>lbak...@basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak CS 1.4.1 bug? InvalidAccessKeyId reported for deletion calls

[Toby Corkindale]

Ah, hold on.. have just discovered that rather than it being deletion 
calls, it seems to just be every X calls of any sort.. sounds like one 
of the servers in the load-balancer pool must be misconfigured somehow, 
but the rest are OK.


On 19/09/13 08:34, Toby Corkindale wrote:

I've just upgraded from Riak CS 1.3.1 to 1.4.1

Using s3cmd to test a few things, I've found some odd behaviour.
Creating a bucket and putting a file works just fine, eg:

s3cmd mb s3://test
s3cmd put README s3://test
s3cmd get s3://test/README

However if I try to delete a file or bucket, it throws an error:

s3cmd del s3://test/README
ERROR: S3 error: 403 (InvalidAccessKeyId): The AWS Access Key Id you
provided does not exist in our records.

s3cmd rb s3://test
ERROR: S3 error: 403 (InvalidAccessKeyId): The AWS Access Key Id you
provided does not exist in our records.


Have I messed something up during the upgrade, or is this a bug in 1.4.1?

ta
Toby

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Riak CS 1.4.1 bug? InvalidAccessKeyId reported for deletion calls

[Toby Corkindale]

I've just upgraded from Riak CS 1.3.1 to 1.4.1

Using s3cmd to test a few things, I've found some odd behaviour.
Creating a bucket and putting a file works just fine, eg:

s3cmd mb s3://test
s3cmd put README s3://test
s3cmd get s3://test/README

However if I try to delete a file or bucket, it throws an error:

s3cmd del s3://test/README
ERROR: S3 error: 403 (InvalidAccessKeyId): The AWS Access Key Id you 
provided does not exist in our records.


s3cmd rb s3://test
ERROR: S3 error: 403 (InvalidAccessKeyId): The AWS Access Key Id you 
provided does not exist in our records.



Have I messed something up during the upgrade, or is this a bug in 1.4.1?

ta
Toby

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re:

[Alex Moore]

Hi Markus,With Riak 1.4, 2i results are sorted in ascending order. So a range query from 0 to  or  to 0 with a max result of 1 will always return the lowest index value. In your case you may want to store the largest key # somewhere, or if you need to find it infrequently you may also try a map reduce job.  How often will you need to know this max key?Thanks,Alex MooreOn September 18, 2013 at 5:56:22 AM, Markus Doppelbauer (doppelba...@gmx.net) wrote:Hello,Is there a chance to get the biggest key of a bucket or secondary index?E.g.: A bucket (or 2i) contains the keys: 10, 20, 30, 40, 50The result should be: 50 Could this problem be solved, by quering a secondary index from back, e.g.:curl http://localhost:8098/buckets/mybucket/index/field_int/999/0?max_results=1 Thanks a lotMarkus  


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: riak control and memory usage

[Thomas Stein]

On Wednesday 18 September 2013 11:55:49 Christopher Meiklejohn wrote:
> On Wednesday, September 18, 2013 at 11:47 AM, Thomas Stein wrote:
> > hello.
> > 
> > I'm playing around with riak control. Nice tool. One problem i have right
> > now. Memory Usage of the nodes is not shown. See screenshot at
> > 
> > http://meine-oma.de/riakcontrol.png
> > 
> > Someone has an idea whats causing this? Error logs are empty.
> 
> This bug has been fixed on the 1.4 and master branches:
> 
> https://github.com/basho/riak_control/pull/140

Ah thanks. 

cheers
t.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: riak control and memory usage

[Christopher Meiklejohn]

On Wednesday, September 18, 2013 at 11:47 AM, Thomas Stein wrote:
> hello.
> 
> I'm playing around with riak control. Nice tool. One problem i have right 
> now. 
> Memory Usage of the nodes is not shown. See screenshot at
> 
> http://meine-oma.de/riakcontrol.png
> 
> Someone has an idea whats causing this? Error logs are empty. 
This bug has been fixed on the 1.4 and master branches:

https://github.com/basho/riak_control/pull/140 

- Chris

-- 
Christopher Meiklejohn
Software Engineer
Basho Technologies, Inc.





___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Riak control on HTTP (not HTTPS)

[Guido Medina]

Hi,

Is it possible to have Riak control running at HTTP port on localhost? 
Assuming security is provided by SSH tunnels.
If so what is needed to be done at app.config? I enabled Riak control 
but it is redirecting me to HTTPS.


Thanks,

Guido.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

ring resizing feature?

[pseudo-nyme]

Dear all,
After a quick glimpse at Riak, I must say I like it: DHT with masterless design 
and gossiping protocol, that is appealing. More than maintainability or 
configurability, I am interested in scalability. I have looked at the node 
adding process for fine scaling, but for scaling big (over the long term) the 
necessity of resizing the ring (in order to double the number of partitions) 
seems unavoidable to me.
On Github, the v1.4 changelog seems to indicate that the "ring resizing 
feature" is now merged in the release (notably from riak-core#301); however, 
the official documentation for v1.4 (on docs.basho.com) still states that 
"ring_creation_size" is only configured a install time. I understand that ring 
resizing is not by itself a scaling mechanism, but
if you keep scaling out, sooner or later you will reach the limit. So I would 
like a bit of clarification on it:
- is this feature officially released as part of 1.4?
- if yes, is it a stable feature, is it ready for production? if not, is
it still under ongoing development?
- if it was released as unstable, what is your experience with it?
- in your opinion, can the problem be simplified if we only consider scaling 
out by doubling the ring size?
Also, independently of ring resizing, what do you consider the current limits 
of ring_creation_size? I have seen mentioned that 2048 and higher could 
importantly increase the volume of gossiping, what is your experience with it?
I hope you will be kind enough to provide me with some enlightenments. Kind 
regards.
PN


Une messagerie gratuite, garantie à vie et des services en plus, ça vous tente ?
Je crée ma boîte mail www.laposte.net

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

riak control and memory usage

[Thomas Stein]

hello.

I'm playing around with riak control. Nice tool. One problem i have right now. 
Memory Usage of the nodes is not shown. See screenshot at

http://meine-oma.de/riakcontrol.png

Someone has an idea whats causing this? Error logs are empty.

thanks and best regards
t.



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: ring resizing feature?

[Jordan West]

Hey PN,

I'm the engineer at Basho working on Ring Resizing, primarily. I've left
some comments inline.


On Wed, Sep 18, 2013 at 12:46 AM,  wrote:

>
> Dear all,
> After a quick glimpse at Riak, I must say I like it: DHT with masterless
> design and gossiping protocol, that is appealing. More than maintainability
> or configurability, I am interested in scalability. I have looked at the
> node adding process for fine scaling, but for scaling big (over the long
> term) the necessity of resizing the ring (in order to double the number of
> partitions) seems unavoidable to me.
>

This is certainly one of the use cases the feature is meant to address.
However, we suggest that it is your "last resort" scaling feature -- if
your cluster can still handle adding nodes that's a better option. But at
some point, yes, doubling or quadrupling the ring size can be useful,
especially if you started with a small cluster. Its, not useful, however,
for e.g. doubling the number of partitions during peak hours for more
concurrency and then shrinking back down again.


> - is this feature officially released as part of 1.4?
>

The feature was released as experimental in Riak 1.4. The move out of
experimental is slated for the next release.


> - if yes, is it a stable feature, is it ready for production? if not, is
> it still under ongoing development?
>
- if it was released as unstable, what is your experience with it?
>


For open source, using this feature w/ the 1.4 release is not suggested for
production use. That should change with the next release. This is why there
is no documentation.

We are currently working with a few customers, who have clusters suited for
resizing, at various levels of testing/production.

It is currently under further development. Immediate work involves bug
fixes and minor improvements. Longer term improvements will focus on
reducing the overhead/cost of the operation.


> - in your opinion, can the problem be simplified if we only consider
> scaling out by doubling the ring size?
>

The problem is pretty much the same whether you are expanding by a factor
of 2 or 4 (same for shrinking). The larger the growth/shrink factor the
slower the operation will be.

Also, independently of ring resizing, what do you consider the current
> limits of ring_creation_size? I have seen mentioned that 2048 and higher
> could importantly increase the volume of gossiping, what is your experience
> with it?
> I hope you will be kind enough to provide me with some enlightenments.
> Kind regards.
> PN
>

There we some changes recently to riak_core to improve its handling of
large rings [1]. The benchmarks there tested up to 16384, however, i
believe that was for illustration purposes. Most large clusters we see use
1024. The theoretical maximum most likely depends on many things.

Cheers,
Jordan


>
>
>
> Une messagerie gratuite, garantie à vie et des services en plus, ça vous
> tente ?
> Je crée ma boîte mail www.laposte.net
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak control on HTTP (not HTTPS)

[Christopher Meiklejohn]

I've opened an issue for this: 

https://github.com/basho/riak_control/issues/156

Thanks!

- Chris 

-- 
Christopher Meiklejohn
Software Engineer
Basho Technologies, Inc.



On Wednesday, September 18, 2013 at 10:32 AM, Guido Medina wrote:

> Now it is working, maybe to be added to a wish list, a flag to
> "enforce_secure_transport" and by default be true, I'm quite sure many
> users use SSH tunnels for those things, no point on loading certificates
> and stuff if your servers are well protected and there is a different
> transport for sensitive information.
> 
> Regards,
> 
> Guido. 



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak control on HTTP (not HTTPS)

[Guido Medina]

Thanks for the quick response,

Now it is working, maybe to be added to a wish list, a flag to 
"enforce_secure_transport" and by default be true, I'm quite sure many 
users use SSH tunnels for those things, no point on loading certificates 
and stuff if your servers are well protected and there is a different 
transport for sensitive information.


Regards,

Guido.

On 18/09/13 16:15, Christopher Meiklejohn wrote:

On Wednesday, September 18, 2013 at 10:12 AM, Guido Medina wrote:

Hi,

Is it possible to have Riak control running at HTTP port on localhost?
Assuming security is provided by SSH tunnels.
If so what is needed to be done at app.config? I enabled Riak control
but it is redirecting me to HTTPS.

Yes, this is possible if you disable the authentication.

It will only redirect to SSL when the authentication is set to 'user list'
instead of 'none' to ensure you don't expose credentials over an
unencrypted channel.

- Chris




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak control on HTTP (not HTTPS)

[Christopher Meiklejohn]

On Wednesday, September 18, 2013 at 10:12 AM, Guido Medina wrote:
> Hi,
> 
> Is it possible to have Riak control running at HTTP port on localhost? 
> Assuming security is provided by SSH tunnels.
> If so what is needed to be done at app.config? I enabled Riak control 
> but it is redirecting me to HTTPS.

Yes, this is possible if you disable the authentication.  

It will only redirect to SSL when the authentication is set to 'user list' 
instead of 'none' to ensure you don't expose credentials over an 
unencrypted channel.

- Chris

-- 
Christopher Meiklejohn
Software Engineer
Basho Technologies, Inc.





___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

503 Error

[Wolfinger, Fred]

Good morning.

For the past 24 hours, I have been trying to delete some data from Riak and 
reingest. I am using the 2.0.1 python-riak-client (http interface) and keep 
getting a 503 error. I have tried restarting the nodes, everything seems to be 
happy. This is a 3-node virtual cluster that we are using for prototyping. And 
I am not at this time dealing with large amounts of data either. I tried using 
the GPB interface but got an error that it doesn't appear to support search, 
which we are using.

Any thoughts on how to troubleshoot would be greatly appreciated.

Fred Wolfinger
Senior Data Scientist, CyberPoint Labs
direct +1 410 779 6741
mobile +1 443 655 3322


CyberPoint International

621 East Pratt Street, Suite 300

Baltimore MD 21202-3140

phone +1 410 779 6700

www.cyberpointllc.com





If you believe you received this e-mail in error, please notify the sender 
immediately, delete the e-mail from your computer and do not copy or disclose 
it to anyone else.


The information in this email constitutes the proprietary information of Cyber 
Point International, LLC (DBA CyberPoint), and should be accessed only by the 
individual to whom it is addressed. The information in this email and any 
attachments may not be used, copied or disclosed without the consent of 
CyberPoint. CyberPoint is not responsible for any damages caused by your 
unauthorized use of the materials in this email.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: riak-cs_1.4.1-1 Debian package

[Jared Morrow]

Sebastian,

There is no newer package.  Can you tell me the following:

* What debian version are you on?
* What user did you run the above command on?  Do they have root access?

>From that same user can you do this:

mkdir -p /etc/riak-cs && touch /etc/riak-cs/testfile && echo "success"

* If you are running in a virtualized environment, what type is it?

The package script should be adding files to /etc/riak-cs before the chmod
command gets there, so something is fishy about the files not being added.
 Either the user or system has weird permissions on adding files to /etc/
or there is a bug in the package.  Once you get back to me, I'll try to
reproduce on my end.

-Jared


On Wed, Sep 18, 2013 at 6:08 AM, Sebastian Müller wrote:

> Hello,
> I tired to install riak-cs on Debian, but installation fails due to an
> error in the post-install script.
>
> # dpkg -i riak-cs_1.4.1-1_amd64.deb
> Selecting previously unselected package riak-cs.
> (Reading database ... 51331 files and directories currently installed.)
> Unpacking riak-cs (from riak-cs_1.4.1-1_amd64.deb) ...
> Setting up riak-cs (1.4.1-1) ...
> chmod: cannot access `/etc/riak-cs/*': No such file or directory
> dpkg: error processing riak-cs (--install):
>  subprocess installed post-installation script returned error exit status 1
> Processing triggers for man-db ...
> Errors were encountered while processing:
>  riak-cs
>
> The VM has no internet connection (has access to the university's Debian
> repo), so I downloaded them from the Basho website. I expected the config
> files for riak-cs in /etc/riak-cs, but the directory is empty.
>
> Is there a newer version of the package or is it a bug?
>
> Regards,
> Sebastian
>
>
> __**_
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/**mailman/listinfo/riak-users_**lists.basho.com
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

riak-cs_1.4.1-1 Debian package

[Sebastian Müller]

Hello,
I tired to install riak-cs on Debian, but installation fails due to an 
error in the post-install script.


# dpkg -i riak-cs_1.4.1-1_amd64.deb
Selecting previously unselected package riak-cs.
(Reading database ... 51331 files and directories currently installed.)
Unpacking riak-cs (from riak-cs_1.4.1-1_amd64.deb) ...
Setting up riak-cs (1.4.1-1) ...
chmod: cannot access `/etc/riak-cs/*': No such file or directory
dpkg: error processing riak-cs (--install):
 subprocess installed post-installation script returned error exit status 1
Processing triggers for man-db ...
Errors were encountered while processing:
 riak-cs

The VM has no internet connection (has access to the university's Debian 
repo), so I downloaded them from the Basho website. I expected the 
config files for riak-cs in /etc/riak-cs, but the directory is empty.


Is there a newer version of the package or is it a bug?

Regards,
Sebastian


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

[no subject]

[Markus Doppelbauer]

Hello,


Is there a chance to get the biggest key of a bucket or secondary index?

E.g.: A bucket (or 2i) contains the keys: 10, 20, 30, 40, 50

The result should be: 50

 

Could this problem be solved, by quering a secondary index from back, e.g.:

curl http://localhost:8098/buckets/mybucket/index/field_int/999/0?max_results=1

 

Thanks a lot

Markus

 

 


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

ring resizing feature?

[pseudo-nyme]

Dear all,
After a quick glimpse at Riak, I must say I like it: DHT with masterless design 
and gossiping protocol, that is appealing. More than maintainability or 
configurability, I am interested in scalability. I have looked at the node 
adding process for fine scaling, but for scaling big (over the long term) the 
necessity of resizing the ring (in order to double the number of partitions) 
seems unavoidable to me.
On Github, the v1.4 changelog seems to indicate that the "ring resizing 
feature" is now merged in the release (notably from riak-core#301); however, 
the official documentation for v1.4 (on docs.basho.com) still states that 
"ring_creation_size" is only configured a install time. I understand that ring 
resizing is not by itself a scaling mechanism, but
if you keep scaling out, sooner or later you will reach the limit. So I would 
like a bit of clarification on it:
- is this feature officially released as part of 1.4?
- if yes, is it a stable feature, is it ready for production? if not, is
it still under ongoing development?
- if it was released as unstable, what is your experience with it?
- in your opinion, can the problem be simplified if we only consider scaling 
out by doubling the ring size?
Also, independently of ring resizing, what do you consider the current limits 
of ring_creation_size? I have seen mentioned that 2048 and higher could 
importantly increase the volume of gossiping, what is your experience with it?
I hope you will be kind enough to provide me with some enlightenments. Kind 
regards.
PN



Une messagerie gratuite, garantie à vie et des services en plus, ça vous tente ?
Je crée ma boîte mail www.laposte.net

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: member-status pending section

[Thomas Stein]

On Wednesday 18 September 2013 08:38:41 Thomas Stein wrote:

> host / # riak-admin transfer-limit
> === Transfer Limit
> 
> LimitNode
> 
> --- 2'ri...@10.xxx'
> 2'ri...@10.xxx'
> 1'ri...@10.xxx'
> 
> --- Note: You can change transfer limits with 'riak-admin transfer_limit
> ' and 'riak-admin transfer_limit  '
> 
> Do i have to raise the transfer limit? Last time i did this the impact was
> pretty much high.

After setting riak7 limit to 2, everything started to working again. Now i 
have to add riak4 and riak8 back to the cluster. Let's see how this works out.

thanks and best regards
t.


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com