Re: Status and maturity of riak-cs

[Toby Corkindale]

Riak CS works, but has some long-standing issues that have never been
resolved.
It doesn't support AWSv4 auth signatures, so some client libraries won't
talk to it at all.
It's quite fragile in respect to its connections to Riak KV -- if KV is
slow to start up, then Riak CS dies. If KV is restarted, then CS just error
messages to client for as many requests as you have internal worker
connections. (ie. many across your cluster)

Performance isn't particularly good, and disk usage ends up far higher than
you'd expect.

I've been actively trying to migrate over to Minio for S3 storage, instead.
Worth a look, and unlike riak CS, has had lots of active development in
recent times.

-Toby

On Thu, 20 Jul 2017 at 02:28 Stefan Funk  wrote:

> Hi everybody,
>
> I'm new to Riak-CS and just joined the group.
>
> We've been exploring Riak-CS now for a couple of days and consider it as a
> potential inhouse-alternative to external S3-based storage providers.
>
> Given the last commit was in January 2016, the question rose as to how
> well the project is supported and how mature the solution is.
>
> I'd be very thankful for any comments from the community on this.
>
>
> Best regards
>
> Stefan
>
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Bug: Stanchion 1.5 in repo along riak cs 2.0

[Toby Corkindale]

Hey all,
I was just bootstrapping a fresh Riak CS cluster, and to my disappointment
ran into the same problem! Stanchion 1.5.0 gets installed, because that's
the one you have in the packagecloud repositories.
It's been two years since I reported this issue!

Toby

On Tue, 16 Jun 2015 at 17:41 Toby Corkindale <t...@dryft.net> wrote:

> Glad we got to the bottom of it.
> You should probably look at updating your docs for Riak-CS and removing
> stanchion 1.5 from the repo to avoid future confusion?
>
> On Wed, 10 Jun 2015 at 00:46 Kota Uenishi <k...@basho.com> wrote:
>
>> I got it! Good catch, Chris.
>>
>> Toby, you should follow these instructions to get the latest Stanchion:
>>
>> https://packagecloud.io/basho/stanchion/install
>>
>> as it looks like we have different repository since 2.0. I did follow
>> these instructions prior to try my test earlier in this loop.
>>
>> For Riak CS package, this instructions might be more precise because
>> it's automatically generated:
>>
>> https://packagecloud.io/basho/riak-cs/install
>>
>> On Tue, Jun 9, 2015 at 11:02 PM, Christopher Mancini <cmanc...@basho.com>
>> wrote:
>> > Kota,
>> >
>> > I think Toby is right, if you look at packagecloud.io, there is no
>> reference
>> > to stanchion 2, only 1.5.
>> >
>> > https://packagecloud.io/basho/riak-cs?filter=debs
>> >
>> > Not sure who can fix this, but its certainly not aligned with our
>> > recommendations.
>> >
>> >
>> > On Mon, Jun 8, 2015 at 9:04 PM, Toby Corkindale <t...@dryft.net> wrote:
>> >>
>> >> Hi Kota,
>> >> I was installing these on fresh 14.04 VMs, so I don't know how I could
>> >> have had stale apt-cache info re Stanchion, but still had up to date
>> riak-cs
>> >> package info.
>> >> Oh well, if you couldn't reproduce it, never mind. Thanks for trying.
>> >>
>> >> Cheers
>> >> Toby
>> >>
>> >> On Mon, 8 Jun 2015 at 11:54 Kota Uenishi <k...@basho.com> wrote:
>> >>>
>> >>> Toby,
>> >>>
>> >>> I tried to reproduce your problem in my clean ubuntu 14.04 which is a
>> >>> fresh docker image, but failed. No recommendation was made. Maybe
>> state of
>> >>> your apt repository or something is stale?
>> >>>
>> >>> # apt-get install riak-cs=2.0.1-1
>> >>>
>> >>> Reading package lists... Done
>> >>>
>> >>> Building dependency tree
>> >>>
>> >>> Reading state information... Done
>> >>>
>> >>> The following NEW packages will be installed:
>> >>>
>> >>>   riak-cs
>> >>>
>> >>> 0 upgraded, 1 newly installed, 0 to remove and 25 not upgraded.
>> >>>
>> >>> Need to get 20.1 MB of archives.
>> >>>
>> >>> After this operation, 36.8 MB of additional disk space will be used.
>> >>>
>> >>> Fetched 20.1 MB in 6s (3322 kB/s)
>> >>>
>> >>> Selecting previously unselected package riak-cs.
>> >>>
>> >>> (Reading database ... 12002 files and directories currently
>> installed.)
>> >>>
>> >>> Preparing to unpack .../riak-cs_2.0.1-1_amd64.deb ...
>> >>>
>> >>> Unpacking riak-cs (2.0.1-1) ...
>> >>>
>> >>> Processing triggers for ureadahead (0.100.0-16) ...
>> >>>
>> >>> Setting up riak-cs (2.0.1-1) ...
>> >>>
>> >>> Adding group `riak' (GID 105) ...
>> >>>
>> >>> Done.
>> >>>
>> >>> Adding system user `riakcs' (UID 102) ...
>> >>>
>> >>> Adding new user `riakcs' (UID 102) with group `riak' ...
>> >>>
>> >>> Not creating home directory `/var/lib/riak-cs'.
>> >>>
>> >>> Processing triggers for ureadahead (0.100.0-16) ...
>> >>>
>> >>> # apt-get install stanchion=2.0.0-1
>> >>>
>> >>> Reading package lists... Done
>> >>>
>> >>> Building dependency tree
>> >>>
>> >>> Reading state information... Done
>> >>>
>> >>> The following NEW packages will be installed:
>> >>>
>> >>>   stanchion
>> >>>
>> >>

Re: Long delays when trying to recover from errors in Java client at startup

[Toby Corkindale]

Hi Magnus,
I can't use riak-admin in a script like this, because Riak KV is not
installed on the client containers.

Ah, I forgot that bucket listing was the same as key listing. I'll switch
that to a different test.
In this use case, though, it's not too important -- I'm trying to deal with
unit tests that start up Riak fixtures via Docker.

So there is literally no content in them to start up. Unfortunately, doing
a simple HTTP request to /ping isn't sufficient to really tell if the Riak
fixture has spun up properly, as sometimes it'll return OK to /ping, but
still return errors on other requests.

Do you know why there is this eight minute delay before RiakClient sees the
node as good?
It's not the Riak KV instance -- because if I cheat and put a ten second
delay in, so that the very first attempt to scan for buckets succeeds, then
things proceed instantly.

On Mon, 22 May 2017 at 17:14 Magnus Kessler <mkess...@basho.com> wrote:

> On 22 May 2017 at 07:02, Toby Corkindale <t...@dryft.net> wrote:
>
>> Hi,
>> I've been trying to make a JVM-based app have better error recovery when
>> the Riak cluster is still in a starting-up state.
>> I have a fairly naive wait-loop that tries to connect and list buckets,
>> and if there's an exception, retry again after a short delay.
>>
>> However once the Riak cluster comes good, the java client hangs on the
>> first operation it makes, for a really long time. Minutes.
>>  -- in particular, at
>> com.basho.riak.client.core.RiakCluster.retryOperation(RiakCluster.java:479)
>>
>> I've tried shutting down and recreating the RiakClient between attempts,
>> but this doesn't seem to help.
>> I guess the node manager has its own back-offs and delays.. Is there a
>> way to reduce these timeouts?
>>
>> Thanks,
>> Toby
>>
>>
> Hi Toby,
>
> Using bucket listing as a method to determine live-ness is a really bad
> idea. Bucket-listing, just as key-listing, requires a coverage query across
> ALL objects stored in the cluster, and will take a really long time if the
> cluster contains many objects.
>
> A better alternative would be to have a canary object with a known key,
> that can be read quickly.
>
> In startup scripts, that need to wait until Riak KV is operational, we
> recommend using `riak-admin wait-for-service riak_kv`.
>
> Kind Regards,
>
> Magnus
>
> --
> Magnus Kessler
> Client Services Engineer
> Basho Technologies Limited
>
> Registered Office - 8 Lincoln’s Inn Fields London WC2A 3BP Reg 07970431
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Long delays (8min) when trying to recover from errors in Java client at startup

[Toby Corkindale]

Thought I'd fill in some extra info in case it helps

$ time ./target/pack/bin/launcher
WARNING: Riak connection bad, retrying... 30 tries remaining
Node state: HEALTH_CHECKING
WARNING: Riak connection bad, retrying... 29 tries remaining
Node state: HEALTH_CHECKING
WARNING: Riak connection bad, retrying... 28 tries remaining
Node state: RUNNING
.
Then we hang for almost exactly eight minutes!
Then things continue fine!

It's that eight minutes of sleeping that I really want to fix -- as
timeouts go it is just way too long!

Thanks for any advice,
Toby

On Mon, 22 May 2017 at 16:01 Toby Corkindale <t...@dryft.net> wrote:

> Hi,
> I've been trying to make a JVM-based app have better error recovery when
> the Riak cluster is still in a starting-up state.
> I have a fairly naive wait-loop that tries to connect and list buckets,
> and if there's an exception, retry again after a short delay.
>
> However once the Riak cluster comes good, the java client hangs on the
> first operation it makes, for a really long time. Minutes.
>  -- in particular, at
> com.basho.riak.client.core.RiakCluster.retryOperation(RiakCluster.java:479)
>
> I've tried shutting down and recreating the RiakClient between attempts,
> but this doesn't seem to help.
> I guess the node manager has its own back-offs and delays.. Is there a way
> to reduce these timeouts?
>
> Thanks,
> Toby
>
> "pool-4-thread-1" #102 prio=5 os_prio=0 tid=0x7f813478e000 nid=0x5517
> waiting on condition [0x0
> 0007f8110e2d000]
>   java.lang.Thread.State: WAITING (parking)
>at sun.misc.Unsafe.park(Native Method)
>- parking to wait for  <0x00076d707b38> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>at
> com.basho.riak.client.core.RiakCluster.retryOperation(RiakCluster.java:479)
>at
> com.basho.riak.client.core.RiakCluster.access$1000(RiakCluster.java:44)
>at
> com.basho.riak.client.core.RiakCluster$RetryTask.run(RiakCluster.java:580)
>at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>
>at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>
>at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
>at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
>at java.lang.Thread.run(Thread.java:748)
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Long delays when trying to recover from errors in Java client at startup

[Toby Corkindale]

Hi,
I've been trying to make a JVM-based app have better error recovery when
the Riak cluster is still in a starting-up state.
I have a fairly naive wait-loop that tries to connect and list buckets, and
if there's an exception, retry again after a short delay.

However once the Riak cluster comes good, the java client hangs on the
first operation it makes, for a really long time. Minutes.
 -- in particular, at
com.basho.riak.client.core.RiakCluster.retryOperation(RiakCluster.java:479)

I've tried shutting down and recreating the RiakClient between attempts,
but this doesn't seem to help.
I guess the node manager has its own back-offs and delays.. Is there a way
to reduce these timeouts?

Thanks,
Toby

"pool-4-thread-1" #102 prio=5 os_prio=0 tid=0x7f813478e000 nid=0x5517
waiting on condition [0x0
0007f8110e2d000]
  java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00076d707b38> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

   at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   at
com.basho.riak.client.core.RiakCluster.retryOperation(RiakCluster.java:479)
   at
com.basho.riak.client.core.RiakCluster.access$1000(RiakCluster.java:44)
   at
com.basho.riak.client.core.RiakCluster$RetryTask.run(RiakCluster.java:580)
   at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

   at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

   at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

   at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

   at java.lang.Thread.run(Thread.java:748)
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: 5-node low-cost, low-power Riak KV cluster -- this sound feasible?

[Toby Corkindale]

On Thu, 27 Apr 2017 at 13:02 Lloyd R. Prentice 
wrote:

> Hello,
>
> I'd like to build a five-node low-cost, low-power Riak KV cluster. Use
> case: learning Riak,  development, testing, storage of personal data.
>
> -- Based on pure hand-waving, I've plugged numbers that seem more than
> adequate for my purposes into the Riak Cluster Capacity Planning
> Calculator. Here's the recommendation:
>
> To manage your estimated 100.0 thousand key/bucket pairs where bucket
> names are ~10 bytes, keys are ~36 bytes, values are ~97.7 KiB and you are
> setting aside 2.0 GiB of RAM per-node for in-memory data management within
> a cluster that is configured to maintain 3 replicas per key (N = 3) then
> Riak, using the Bitcask storage engine, will require at least:
>
> 5 nodes
> 6.4 MiB of RAM per node (31.9 MiB total across all nodes)
> 5.6 GiB of storage space per node (28.0 GiB total storage space used
> across all nodes)
> Based on this, I'm considering the following hardware:
>
> 5 ODROID XU-4
> http://www.hardkernel.com/main/products/prdt_info.php?g_code=G143452239825
>
> Each with 8 Gb eMMC storage
>
> This provides a 64-bit 2 GHz processor and 2 Gb RAM per day node running
> Ubuntu 16.04 at total cost of something under total $65000 for the cluster.
>
> Does this sound like a feasible way to go? Any downsides?
>


I'm not sure where you get the $65000 total figure, but that sounds
extremely high!! Unless you're talking Vietnamese Dong currency or
something.

Either way, why not consider running 5x Amazon AWS EC2 nodes?
Or run up a bunch of virtual machines on your own PC?

I think both of those will give you more realistic Riak results than using
a bunch of ARM bigLITTLE machines which are pretty different to your
typical Linux servers.

Toby
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Error folding keys - incomplete_hint

[Toby Corkindale]

I thought I'd follow up on this again, a long time later.
We gave up on the Dockerised version of Riak.
But I notice we're getting an awful lot of these incomplete_hint errors on
the regular, non-docker, cluster now.

We had a sudden power failure in that server room recently, so there would
have been unclean Riak shutdowns. I guessed those were the cause of the
issues with the Docker version, years ago, so I'm wondering if the same
thing happened here?

Should Riak be better at recovering from rough shutdowns? Or is this
another issue altogether?

-Toby

On Fri, 23 Oct 2015 at 11:10 Toby Corkindale <t...@dryft.net> wrote:

> Quick follow-up: As a bit of a hack, deleting all the .hint files prior to
> each start-up does resolve the errors, and immediately results in a whole
> lot of Bitcask merges happening.
> But that doesn't strike me as a good long-term fix.
>
> On Fri, 23 Oct 2015 at 10:52 Toby Corkindale <t...@dryft.net> wrote:
>
> Hi Hector,
> You can see the Dockerfile here:
> https://gist.github.com/TJC/cb3184705bc0eacde885
>
> It's a work in progress, but also, not that involved.
>
> Ubuntu 14.04 is used as both the docker host, and the docker container.
> It's on the btrfs storage driver. (I've had too many issues with the other
> two)
> The Riak data directory is a volume, and is mounted to an external,
> persistent location. (Which is also btrfs)
>
> I suspect there's an issue around Riak shutting down uncleanly when the
> docker container is stopped.
> I have already had to add this to the start-up each time:
> find /var/lib/riak -name "bitcask.*.lock" -delete
>
> So it's clear that Riak is getting killed rather than shutting down
> cleanly; but even so, I'd hope that Riak would cope with that, rather than
> getting into a permanent state of throwing errors.
>
> Toby
>
>
> On Fri, 23 Oct 2015 at 00:01 Hector Castro <hectcas...@gmail.com> wrote:
>
> Can't say I've paid enough attention to the logs in my single-machine
> Riak within Docker setups to confirm.
>
> Do you have the container image definitions somewhere public? That may
> help someone reproduce the issue. Also, did you ensure that the Riak
> data directory is setup as a Docker volume?
>
> Other things that come to mind:
>
> - What OS is the Docker host running?
> - What storage driver are you using for Docker?
> - What file system is the Docker data directory using?
>
> --
> Hector
>
>
> On Thu, Oct 22, 2015 at 2:27 AM, Toby Corkindale <t...@dryft.net> wrote:
> > Anyone?
> >
> > I note that after 24 hours (on a very lightly loaded test cluster) I'm
> still
> > seeing these scroll by a lot - 600 an hour per node.
> > Really curious to know if this is expected behaviour or if this is
> resulting
> > from some kind of node corruption.
> >
> > Cheers
> > Toby
> >
> >
> >
> > On Wed, 21 Oct 2015 at 12:23 Toby Corkindale <t...@dryft.net> wrote:
> >>
> >> Hi,
> >> I've been working on getting Riak to run inside Docker containers - in a
> >> multi-machine cluster. (Previous work I've seen has only run Riak as a
> >> cluster all on the same machine.)
> >> I thought I had it cracked, although I tripped up on the existing issue
> >> with Riak and lockfiles[1]. But the nodes have been generating an awful
> lot
> >> of errors like the below, and I wondered if anyone here can give me an
> >> explanation? (And, is it a problem?)
> >>
> >> 2015-10-21 01:19:23.567 [error] <0.24495.0> Error folding keys for
> >> "/var/lib/riak/bitcask.1h/2283596
> >> 30832953580969325755111919221821239459840/2.bitcask.data":
> >> {incomplete_hint,4}
> >>
> >> 1: Related issues to the lockfiles --
> >> I note that many are closed, but the problem still exists, and is
> >> particularly triggered by using Docker and stopping/killing Riak more
> >> violently than it likes.
> >> https://github.com/basho/bitcask/issues/163 (closed)
> >> https://github.com/basho/riak/issues/535 (open)
> >> https://github.com/basho/bitcask/issues/167 (closed)
> >> https://github.com/basho/bitcask/issues/99 (closed)
> >
> >
> > ___
> > riak-users mailing list
> > riak-users@lists.basho.com
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak CS race condition at start-up (was: Riak-CS issues when Riak endpoint fails-over to new server)

[Toby Corkindale]

Thanks Luke.

We're using the official Basho Ubuntu packages for Ubuntu 14.04 LTS.
These don't seem to perform that step; they just start Riak CS
simultaneously with Riak KV.

I'm surprised if we're the only people who hit this. Don't people hit these
issues in your other production deployments, or are they running on another
platforms that have more finessed start-up scripts?

Toby


On Sat, 21 Jan 2017 at 03:19 Luke Bakken <lbak...@basho.com> wrote:

> Hi Toby,
>
> The process you use to run "riak-cs start" could use the "riak-admin
> wait-for-service riak_kv" command to ensure Riak is ready first:
>
>
> http://docs.basho.com/riak/kv/2.2.0/using/admin/riak-admin/#wait-for-service
>
> --
> Luke Bakken
> Engineer
> lbak...@basho.com
>
>
> On Thu, Jan 19, 2017 at 5:38 PM, Toby Corkindale <t...@dryft.net> wrote:
> > Hi guys,
> > I've switched our configuration around, so that Riak CS now talks to
> > 127.0.0.1:8087 instead of the haproxy version.
> >
> > We have immediately re-encountered the problems that caused us to move to
> > haproxy.
> > On start-up, riak takes slightly longer than riak-cs to get ready, and so
> > riak-cs logs the following then exits.
> > Restarting riak-cs again (so now 15 seconds after Riak started) results
> in a
> > successful start-up, but obviously this is really annoying for our ops
> guys
> > to have to remember to do after restarting riak or rebooting a machine.
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Riak CS race condition at start-up (was: Riak-CS issues when Riak endpoint fails-over to new server)

[Toby Corkindale]

Hi guys,
I've switched our configuration around, so that Riak CS now talks to
127.0.0.1:8087 instead of the haproxy version.

We have immediately re-encountered the problems that caused us to move to
haproxy.
On start-up, riak takes slightly longer than riak-cs to get ready, and so
riak-cs logs the following then exits.
Restarting riak-cs again (so now 15 seconds after Riak started) results in
a successful start-up, but obviously this is really annoying for our ops
guys to have to remember to do after restarting riak or rebooting a machine.

How do other people avoid this issue in production?

```
2017-01-20 12:23:12.937 [warning]
<0.150.0>@riak_cs_app:check_bucket_props:187 Unable to verify moss.users
bucket settings (disconnected).
2017-01-20 12:23:12.937 [warning]
<0.150.0>@riak_cs_app:check_bucket_props:187 Unable to verify moss.access
bucket settings (disconnected).
2017-01-20 12:23:12.937 [warning]
<0.150.0>@riak_cs_app:check_bucket_props:187 Unable to verify moss.storage
bucket settings (disconnected).
2017-01-20 12:23:12.937 [warning]
<0.150.0>@riak_cs_app:check_bucket_props:187 Unable to verify moss.buckets
bucket settings (disconnected).
2017-01-20 12:23:12.937 [error] <0.150.0>@riak_cs_app:sanity_check:125
Could not verify bucket properties. Error was disconnected.
2017-01-20 12:23:12.938 [error] <0.149.0> CRASH REPORT Process <0.149.0>
with 0 neighbours exited with reason:
{error_verifying_props,{riak_cs_app,start,[normal,[]]}} in
application_master:init/4 line 133
2017-01-20 12:23:12.938 [info] <0.7.0> Application riak_cs exited with
reason: {error_verifying_props,{riak_cs_app,start,[normal,[]]}}
```

On Fri, 6 Jan 2017 at 12:33 Toby Corkindale <t...@dryft.net> wrote:

> Hi Shaun,
> We've been running Riak CS since its early days, so it's possible best
> practice has changed.. but I'm sure at some point it was suggested to put a
> haproxy between CS and KV to guard against the issues of start-up race
> condition, individual KV losses, and brief KV restarts.
> I'm sure we used to have continual issues with CS being dead on nodes
> before we moved to the haproxy solution. That was probably on Debian
> Squeeze though, and these days we're on Ubuntu LTS, and so if CS is
> launched from Upstart at least it can retry to start, whereas on old-school
> init systems it just gets one attempt and then dies.
>
> Moving on though--
>
> Even if it's not the majority practice, shouldn't CS still be able to
> withstand dropping and reconnecting it's protocol buffer TCP connections?
>
> CS still has a problem with not handling the case when its long-standing
> idle PBC connections to KV get reset. Regardless of whether that's because
> the local KV process is restarted, or because we've failed over to a new
> haproxy.
> The errors get pushed back to the S3 client software, but even if they
> retry, they get repeated errors because, I think, CS has such a large pool
> of PBC connections. You have to work through a large portion of this pool
> before you finally get to one that's reconnected and is good.
>
> In our case, the pool size is multiplied by the number of CS instances, so
> quite a large number.
> Most client software has retry limits built in, at much lower values.
>
> While it will come good eventually, there's a significant period of time
> where everything fails, all our monitoring goes red, etc. which we'd like
> to avoid!
>
> I'm surprised this problem doesn't come up more for other users; I don't
> feel like we're running at a large scale.. but maybe we're using a more
> dynamic architecture than major users.
>
> Toby
>
> On Thu, 5 Jan 2017 at 20:04 Shaun McVey <smc...@basho.com> wrote:
>
> Hi Toby,
>
> > I thought that it was recommended AGAINST talking to a co-located Riak
> on the same host?
>
> I'm not sure where you heard that from, but that's not the case.  We do
> discourage running other parts of an application on the same hosts, such as
> client front-ends for example.  From the top of my head (which means
> there's probably an exception to the rule), all our customers have nodes
> set up in the way Magnus described: one CS instance talking locally to its
> KV instance directly on the same node.  The load balancing comes between
> the CS node and the client.
>
> Riak shouldn't take a particularly long time to start at all.  We have
> customers that have terabytes of data per node and a KV node can be
> restarted in just a minute or two.  As long as you have valid bitcask hint
> files in place (which requires a proper shutdown beforehand), then a node
> should come up quickly.  If you have nodes that you feel are taking a
> particularly long time to start up, that may be a symptom of another issue
> unrelated to this discussion.
>
> If, fo

Re: Riak CS - admin keys changing

[Toby Corkindale]

Hi,
I have a follow-up question around this security aspect.

If the riak-cs.conf and stanchion.conf files are changed so that their
admin.key and admin.secret match a different user (eg. not that
first-created admin user) then will that user now have admin-like
privileges?

Or are the admin-abilities determined by something set in the admin user's
data in Riak?

Thanks,
Toby

On Fri, 13 Jan 2017 at 16:38 Toby Corkindale <t...@dryft.net> wrote:

> Thanks, Luke!
>
> On Fri, 13 Jan 2017 at 12:10 Luke Bakken <lbak...@basho.com> wrote:
>
> Hi Toby,
>
> When you create the user, the data is stored in Riak (and is the
> authoritative location). The values must match in the config files to
> provide credentials used when connecting to various parts of your CS
> cluster.
>
> --
> Luke Bakken
> Engineer
> lbak...@basho.com
>
> On Thu, Jan 12, 2017 at 3:47 PM, Toby Corkindale <t...@dryft.net> wrote:
> > Hi,
> > In Riak CS, the admin key and secret is in the config files for both CS
> and
> > Stanchion.
> > Is that the authoritative location for the secrets, or is the
> > initially-created admin user the source, and those just have to match?
> >
> > I tried to figure this out from the source code, but my Erlang really
> isn't
> > up to scratch :(
> >
> > Toby
> >
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak CS - admin keys changing

[Toby Corkindale]

Thanks, Luke!

On Fri, 13 Jan 2017 at 12:10 Luke Bakken <lbak...@basho.com> wrote:

Hi Toby,

When you create the user, the data is stored in Riak (and is the
authoritative location). The values must match in the config files to
provide credentials used when connecting to various parts of your CS
cluster.

--
Luke Bakken
Engineer
lbak...@basho.com

On Thu, Jan 12, 2017 at 3:47 PM, Toby Corkindale <t...@dryft.net> wrote:
> Hi,
> In Riak CS, the admin key and secret is in the config files for both CS
and
> Stanchion.
> Is that the authoritative location for the secrets, or is the
> initially-created admin user the source, and those just have to match?
>
> I tried to figure this out from the source code, but my Erlang really
isn't
> up to scratch :(
>
> Toby
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Riak CS - admin keys changing

[Toby Corkindale]

Hi,
In Riak CS, the admin key and secret is in the config files for both CS and
Stanchion.
Is that the authoritative location for the secrets, or is the
initially-created admin user the source, and those just have to match?

I tried to figure this out from the source code, but my Erlang really isn't
up to scratch :(

Toby
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak-CS issues when Riak endpoint fails-over to new server

[Toby Corkindale]

Hi Shaun,
We've been running Riak CS since its early days, so it's possible best
practice has changed.. but I'm sure at some point it was suggested to put a
haproxy between CS and KV to guard against the issues of start-up race
condition, individual KV losses, and brief KV restarts.
I'm sure we used to have continual issues with CS being dead on nodes
before we moved to the haproxy solution. That was probably on Debian
Squeeze though, and these days we're on Ubuntu LTS, and so if CS is
launched from Upstart at least it can retry to start, whereas on old-school
init systems it just gets one attempt and then dies.

Moving on though--

Even if it's not the majority practice, shouldn't CS still be able to
withstand dropping and reconnecting it's protocol buffer TCP connections?

CS still has a problem with not handling the case when its long-standing
idle PBC connections to KV get reset. Regardless of whether that's because
the local KV process is restarted, or because we've failed over to a new
haproxy.
The errors get pushed back to the S3 client software, but even if they
retry, they get repeated errors because, I think, CS has such a large pool
of PBC connections. You have to work through a large portion of this pool
before you finally get to one that's reconnected and is good.

In our case, the pool size is multiplied by the number of CS instances, so
quite a large number.
Most client software has retry limits built in, at much lower values.

While it will come good eventually, there's a significant period of time
where everything fails, all our monitoring goes red, etc. which we'd like
to avoid!

I'm surprised this problem doesn't come up more for other users; I don't
feel like we're running at a large scale.. but maybe we're using a more
dynamic architecture than major users.

Toby

On Thu, 5 Jan 2017 at 20:04 Shaun McVey <smc...@basho.com> wrote:

> Hi Toby,
>
> > I thought that it was recommended AGAINST talking to a co-located Riak
> on the same host?
>
> I'm not sure where you heard that from, but that's not the case.  We do
> discourage running other parts of an application on the same hosts, such as
> client front-ends for example.  From the top of my head (which means
> there's probably an exception to the rule), all our customers have nodes
> set up in the way Magnus described: one CS instance talking locally to its
> KV instance directly on the same node.  The load balancing comes between
> the CS node and the client.
>
> Riak shouldn't take a particularly long time to start at all.  We have
> customers that have terabytes of data per node and a KV node can be
> restarted in just a minute or two.  As long as you have valid bitcask hint
> files in place (which requires a proper shutdown beforehand), then a node
> should come up quickly.  If you have nodes that you feel are taking a
> particularly long time to start up, that may be a symptom of another issue
> unrelated to this discussion.
>
> If, for any reason, you need to shut down KV, you would then just remove
> the CS node from the HAproxy configuration so it doesn't deal with any
> requests.  The other CS nodes then take the additional load.  There
> shouldn't be a need to restart CS if you remove it from the load balancer.
> Having said that, you shouldn't have to worry about restarting CS as far as
> I'm aware.  You might see failures if KV is down, but once it's up and
> running again, CS will continue to deal with new requests without
> problems.  Any failures to connect to its KV node should be passed to the
> client/front-end, which should have all the proper logic for re-attempts or
> error reporting.
>
> > I'm surprised more people with highly-available Riak CS installations
> haven't hit the same issues.
>
> As I mentioned, our customers go with the setup Magnus described.  I can't
> speak for setups like yours as I've not seen them in the wild.
>
> Kind Regards,
> Shaun
>
> On Wed, Jan 4, 2017 at 11:26 PM, Toby Corkindale <t...@dryft.net> wrote:
>
> Hi Magnus,
> I thought that it was recommended AGAINST talking to a co-located Riak on
> the same host?
> The reason being, the local Riak will take longer to start up than Riak
> CS, once you have a sizeable amount of data. This means Riak CS starts up,
> fails to connect to Riak, and exits.
> You also end up in a situation where you must always restart Riak CS if
> you restart the co-located Riak. (Otherwise the Riak CS PBC connections
> suffer the same problem as I described in my earlier email, where Riak CS
> doesn't realise it needs to reconnect them and returns errors).
>
> Putting haproxy between Riak CS and Riak solved the problem of needing the
> local Riak to be started first.
> But it seems we just were putting the core problem off, rather than
> solving it. ie. That Riak CS doesn't 

Re: Riak-CS issues when Riak endpoint fails-over to new server

[Toby Corkindale]

Hi Magnus,
I thought that it was recommended AGAINST talking to a co-located Riak on
the same host?
The reason being, the local Riak will take longer to start up than Riak CS,
once you have a sizeable amount of data. This means Riak CS starts up,
fails to connect to Riak, and exits.
You also end up in a situation where you must always restart Riak CS if you
restart the co-located Riak. (Otherwise the Riak CS PBC connections suffer
the same problem as I described in my earlier email, where Riak CS doesn't
realise it needs to reconnect them and returns errors).

Putting haproxy between Riak CS and Riak solved the problem of needing the
local Riak to be started first.
But it seems we just were putting the core problem off, rather than solving
it. ie. That Riak CS doesn't understand it needs to re-connect and retry.

I'm surprised more people with highly-available Riak CS installations
haven't hit the same issues.

Toby

On Wed, 4 Jan 2017 at 21:42 Magnus Kessler <mkess...@basho.com> wrote:

> Hi Toby,
>
> As far as I know Riak CS has none of the more advanced retry capabilities
> that Riak KV has. However, in the design of CS there seems to be an
> assumption that a CS instance will talk to a co-located KV node on the same
> host. To achieve high availability, in CS deployments HAProxy is often
> deployed in front of the CS nodes. Could you please let me know if this is
> an option for your setup?
>
> Kind Regards,
>
> Magnus
>
>
> On 4 January 2017 at 01:04, Toby Corkindale <t...@dryft.net> wrote:
>
> Hello all,
> Now that we're all back from the end-of-year holidays, I'd like to bump
> this question up.
> I feel like this has been a long-standing problem with Riak CS not
> handling dropped TCP connections.
> Last time the cause was haproxy dropping idle TCP connections after too
> long, but we solved that at the haproxy end.
>
> This time, it's harder -- we're failing over to a different Riak backend,
> so the TCP connections between Riak CS and Riak PBC *have* to go down, but
> Riak CS just doesn't handle it well at all.
>
> Is there a trick to configuring it better?
>
> Thanks
> Toby
>
>
> On Thu, 22 Dec 2016 at 16:48 Toby Corkindale <t...@dryft.net> wrote:
>
> Hi,
> We've been seeing some issues with Riak CS for a while in a specific
> situation. Maybe you can advise if we're doing something wrong?
>
> Our setup has redundant haproxy instances in front of a cluster of riak
> nodes, for both HTTP and PBC. The haproxy instances share a floating IP
> address.
> Only one node holds the IP, but if it goes down, another takes it up.
>
> Our Riak CS nodes are configured to talk to the haproxy on that floating
> IP.
>
> The problem occurs if the floating IP moves from one haproxy to another.
>
> Suddenly we see a flurry of errors in riak-cs log files.
>
> This is presumably because it was holding open TCP connections, and the
> new haproxy instance doesn't know anything about them, so they get TCP
> RESET and shutdown.
>
> The problem is that riak-cs doesn't try to reconnect and retry
> immediately, instead it just throws a 503 error back to the client. Who
> then retries, but Riak-CS has a pool of a couple of hundred connections to
> cycle through, all of which throw the error!
>
> Does this sound like it is a likely description of the fault?
> Do you have any ways to mitigate this issue in Riak CS when using TCP load
> balancing above Riak PBC?
>
> Toby
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
>
> --
> Magnus Kessler
> Client Services Engineer
> Basho Technologies Limited
>
> Registered Office - 8 Lincoln’s Inn Fields London WC2A 3BP Reg 07970431
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak-CS issues when Riak endpoint fails-over to new server

[Toby Corkindale]

Hello all,
Now that we're all back from the end-of-year holidays, I'd like to bump
this question up.
I feel like this has been a long-standing problem with Riak CS not handling
dropped TCP connections.
Last time the cause was haproxy dropping idle TCP connections after too
long, but we solved that at the haproxy end.

This time, it's harder -- we're failing over to a different Riak backend,
so the TCP connections between Riak CS and Riak PBC *have* to go down, but
Riak CS just doesn't handle it well at all.

Is there a trick to configuring it better?

Thanks
Toby


On Thu, 22 Dec 2016 at 16:48 Toby Corkindale <t...@dryft.net> wrote:

> Hi,
> We've been seeing some issues with Riak CS for a while in a specific
> situation. Maybe you can advise if we're doing something wrong?
>
> Our setup has redundant haproxy instances in front of a cluster of riak
> nodes, for both HTTP and PBC. The haproxy instances share a floating IP
> address.
> Only one node holds the IP, but if it goes down, another takes it up.
>
> Our Riak CS nodes are configured to talk to the haproxy on that floating
> IP.
>
> The problem occurs if the floating IP moves from one haproxy to another.
>
> Suddenly we see a flurry of errors in riak-cs log files.
>
> This is presumably because it was holding open TCP connections, and the
> new haproxy instance doesn't know anything about them, so they get TCP
> RESET and shutdown.
>
> The problem is that riak-cs doesn't try to reconnect and retry
> immediately, instead it just throws a 503 error back to the client. Who
> then retries, but Riak-CS has a pool of a couple of hundred connections to
> cycle through, all of which throw the error!
>
> Does this sound like it is a likely description of the fault?
> Do you have any ways to mitigate this issue in Riak CS when using TCP load
> balancing above Riak PBC?
>
> Toby
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Riak-CS issues when Riak endpoint fails-over to new server

[Toby Corkindale]

Hi,
We've been seeing some issues with Riak CS for a while in a specific
situation. Maybe you can advise if we're doing something wrong?

Our setup has redundant haproxy instances in front of a cluster of riak
nodes, for both HTTP and PBC. The haproxy instances share a floating IP
address.
Only one node holds the IP, but if it goes down, another takes it up.

Our Riak CS nodes are configured to talk to the haproxy on that floating IP.

The problem occurs if the floating IP moves from one haproxy to another.

Suddenly we see a flurry of errors in riak-cs log files.

This is presumably because it was holding open TCP connections, and the new
haproxy instance doesn't know anything about them, so they get TCP RESET
and shutdown.

The problem is that riak-cs doesn't try to reconnect and retry immediately,
instead it just throws a 503 error back to the client. Who then retries,
but Riak-CS has a pool of a couple of hundred connections to cycle through,
all of which throw the error!

Does this sound like it is a likely description of the fault?
Do you have any ways to mitigate this issue in Riak CS when using TCP load
balancing above Riak PBC?

Toby
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Java client -- slow to shutdown

[Toby Corkindale]

Hi,
I'm using the Java client via protocol buffers to Riak.
(Actually I'm using it via Scala 2.11.8 on OpenJDK 8)

After calling client.shutdown(), there is always a delay of 4 seconds
before the app actually exits. Why is this, and what can I do about it?

To demonstrate the issue, use these files:
https://gist.github.com/TJC/9a6a174cb1419a7c32e8018c5a495e3d

If you put both of them in a fresh directory and then run "sbt", it should
grab various dependencies and stuff, and then you can use "compile" and
"run" commands.
(You'll need to do "export RIAK_SERVER=my.riak.cluster.net" in the shell
before you run sbt)

If you do "run" a few times, you'll see it always takes four seconds to get
back to the sbt prompt. If you comment out the two riak statements in the
source code (the connection and shutdown), then "run" a few times, it takes
zero seconds.

I've tested this outside of sbt and the same issue exists.. it's just
easier to make a quick demo that works inside sbt.

Also reported as https://github.com/basho/riak-java-client/issues/689

Cheers
Toby
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Riak CS, logging and admin secrets

[Toby Corkindale]

Hi,
The other day I was concerned when I discovered the riak-cs console.log
contained the following:
```
2016-11-17 16:36:10.639 [warning]
<0.155.0>@riak_cs_app:check_admin_creds:76 The admin user's secret is
specified. Ignoring.
2016-11-17 16:36:10.680 [info]
<0.155.0>@riak_cs_app:fetch_and_cache_admin_creds:93 setting admin secret
as XX==
```

Except that the long string of XX was our real admin secret!

I saw the warning in the previous line, and wondered if perhaps something
had changed, and we shouldn't be specifying the admin details any longer?

But according to this, yes we should:
https://docs.basho.com/riak/cs/2.1.1/cookbooks/configuration/riak-cs/#specifying-the-admin-user

So.. what's the deal? What is this warning message about, and how can we
stop it logging out secret keys?

Cheers
Toby
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Java client via Scala - StringBinIndex.Name not accessible

[Toby Corkindale]

Replying to myself, but with the solution.
After some discussion on a Scala list, I was able to get the below Scala
code to compile. There were two important changes.

1) Using StringBinIndex.named(name) to build the index rather than
StringBinIndex.Name(name), due to the interop issues.

2) Parameterising the getIndex function explicitly. Someone on the scala
list suggested that this was because the type signature of the Java client
code is not strictly correct. I will present their comments here in case
someone can confirm?
Instead of:
public > V getIndex(T
name)
It should be:
public , T extends RiakIndex.Name> V getIndex(T
name)


The complete Scala example for writing to a key with 2i indexing is as
follows:
(assumes "riak" is an already connected riak instance)

  def storeByKeyRaw(bucketName: String, key: String, data: Array[Byte],
indexes: Map[String,String] = Map()) = {
val bucket = new Namespace("default", bucketName)
val loc = new Location(bucket, key)
val obj = new RiakObject
obj.setValue(BinaryValue.create(data))

indexes.foreach{ case (name, contents) =>
  val idx = StringBinIndex.named(name)

obj.getIndexes().getIndex[StringBinIndex,StringBinIndex.Name](idx).add(contents)
}

val response = riak.execute(new
StoreValue.Builder(obj).withLocation(loc).build)
   // ..
  }

On Wed, 16 Dec 2015 at 15:10 Toby Corkindale <t...@dryft.net> wrote:

> Hi,
> I'm having a lot of trouble getting a particular part of the Riak java
> client to work when called from Scala.
>
> There's a Java class which I'm not allowed to instantiate directly, but
> must instead call an embedded subclass's static constructor method.
>
> ie.
> import com.basho.riak.client.core.query.indexes.StringBinIndex
> val idx = new StringBinIndex.Name("colours")
>
> However, in Scala, I get this error reported:
> Constructor Name in class Name cannot be accessed in class [myclass]
>
> Docs for the class:
>
> http://basho.github.io/riak-java-client/2.0.1/index.html?com/basho/riak/client/core/query/indexes/StringBinIndex.html
>
> I don't suppose anyone can see what I'm doing wrong, or suggest how to
> make this work from Scala?
>
> Cheers,
> Toby
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Java client via Scala - StringBinIndex.Name not accessible

[Toby Corkindale]

Hi Luke,
Thanks for the quick response.. That link does sound like it's describing
the problem.
I'll see if I can figure out a way to get around it :(
The rest of the code works lovely with thin wrappers around the java
client, and it's only now in the project that I'm trying to get the indexes
added :(

Toby

On Wed, 16 Dec 2015 at 15:22 Luke Bakken <lbak...@basho.com> wrote:

> Hi Toby,
>
> I think you're running into something similar to this:
> http://www.scala-lang.org/old/node/1381.html
>
> which links to this: https://issues.scala-lang.org/browse/SI-1806
>
> Not being familar with Java / Scala interop I hope that the discussion
> and further links there can help you out.
>
> --
> Luke Bakken
> Engineer
> lbak...@basho.com
>
>
> On Tue, Dec 15, 2015 at 8:11 PM, Toby Corkindale <t...@dryft.net> wrote:
> > Hi,
> > I'm having a lot of trouble getting a particular part of the Riak java
> > client to work when called from Scala.
> >
> > There's a Java class which I'm not allowed to instantiate directly, but
> must
> > instead call an embedded subclass's static constructor method.
> >
> > ie.
> > import com.basho.riak.client.core.query.indexes.StringBinIndex
> > val idx = new StringBinIndex.Name("colours")
> >
> > However, in Scala, I get this error reported:
> > Constructor Name in class Name cannot be accessed in class [myclass]
> >
> > Docs for the class:
> >
> http://basho.github.io/riak-java-client/2.0.1/index.html?com/basho/riak/client/core/query/indexes/StringBinIndex.html
> >
> > I don't suppose anyone can see what I'm doing wrong, or suggest how to
> make
> > this work from Scala?
> >
> > Cheers,
> > Toby
> >
> >
> > ___
> > riak-users mailing list
> > riak-users@lists.basho.com
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Java client via Scala - StringBinIndex.Name not accessible

[Toby Corkindale]

Hi,
Reading further, it seems it's simply not possible to get Scala to work
with the static protected inner classes of the Index types in the Java
client :(

In the Java client, is there any other way to instantiate indexes? I
suspect not, but figure I could ask just in case.

Cheers
Toby

On Wed, 16 Dec 2015 at 15:34 Toby Corkindale <t...@dryft.net> wrote:

> Hi Luke,
> Thanks for the quick response.. That link does sound like it's describing
> the problem.
> I'll see if I can figure out a way to get around it :(
> The rest of the code works lovely with thin wrappers around the java
> client, and it's only now in the project that I'm trying to get the indexes
> added :(
>
> Toby
>
> On Wed, 16 Dec 2015 at 15:22 Luke Bakken <lbak...@basho.com> wrote:
>
>> Hi Toby,
>>
>> I think you're running into something similar to this:
>> http://www.scala-lang.org/old/node/1381.html
>>
>> which links to this: https://issues.scala-lang.org/browse/SI-1806
>>
>> Not being familar with Java / Scala interop I hope that the discussion
>> and further links there can help you out.
>>
>> --
>> Luke Bakken
>> Engineer
>> lbak...@basho.com
>>
>>
>> On Tue, Dec 15, 2015 at 8:11 PM, Toby Corkindale <t...@dryft.net> wrote:
>> > Hi,
>> > I'm having a lot of trouble getting a particular part of the Riak java
>> > client to work when called from Scala.
>> >
>> > There's a Java class which I'm not allowed to instantiate directly, but
>> must
>> > instead call an embedded subclass's static constructor method.
>> >
>> > ie.
>> > import com.basho.riak.client.core.query.indexes.StringBinIndex
>> > val idx = new StringBinIndex.Name("colours")
>> >
>> > However, in Scala, I get this error reported:
>> > Constructor Name in class Name cannot be accessed in class [myclass]
>> >
>> > Docs for the class:
>> >
>> http://basho.github.io/riak-java-client/2.0.1/index.html?com/basho/riak/client/core/query/indexes/StringBinIndex.html
>> >
>> > I don't suppose anyone can see what I'm doing wrong, or suggest how to
>> make
>> > this work from Scala?
>> >
>> > Cheers,
>> > Toby
>> >
>> >
>> > ___
>> > riak-users mailing list
>> > riak-users@lists.basho.com
>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> >
>>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Java client via Scala - StringBinIndex.Name not accessible

[Toby Corkindale]

Hi,
I'm having a lot of trouble getting a particular part of the Riak java
client to work when called from Scala.

There's a Java class which I'm not allowed to instantiate directly, but
must instead call an embedded subclass's static constructor method.

ie.
import com.basho.riak.client.core.query.indexes.StringBinIndex
val idx = new StringBinIndex.Name("colours")

However, in Scala, I get this error reported:
Constructor Name in class Name cannot be accessed in class [myclass]

Docs for the class:
http://basho.github.io/riak-java-client/2.0.1/index.html?com/basho/riak/client/core/query/indexes/StringBinIndex.html

I don't suppose anyone can see what I'm doing wrong, or suggest how to make
this work from Scala?

Cheers,
Toby
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak in Docker - Error folding keys - incomplete_hint

[Toby Corkindale]

Hi Hector,
You can see the Dockerfile here:
https://gist.github.com/TJC/cb3184705bc0eacde885

It's a work in progress, but also, not that involved.

Ubuntu 14.04 is used as both the docker host, and the docker container.
It's on the btrfs storage driver. (I've had too many issues with the other
two)
The Riak data directory is a volume, and is mounted to an external,
persistent location. (Which is also btrfs)

I suspect there's an issue around Riak shutting down uncleanly when the
docker container is stopped.
I have already had to add this to the start-up each time:
find /var/lib/riak -name "bitcask.*.lock" -delete

So it's clear that Riak is getting killed rather than shutting down
cleanly; but even so, I'd hope that Riak would cope with that, rather than
getting into a permanent state of throwing errors.

Toby


On Fri, 23 Oct 2015 at 00:01 Hector Castro <hectcas...@gmail.com> wrote:

> Can't say I've paid enough attention to the logs in my single-machine
> Riak within Docker setups to confirm.
>
> Do you have the container image definitions somewhere public? That may
> help someone reproduce the issue. Also, did you ensure that the Riak
> data directory is setup as a Docker volume?
>
> Other things that come to mind:
>
> - What OS is the Docker host running?
> - What storage driver are you using for Docker?
> - What file system is the Docker data directory using?
>
> --
> Hector
>
>
> On Thu, Oct 22, 2015 at 2:27 AM, Toby Corkindale <t...@dryft.net> wrote:
> > Anyone?
> >
> > I note that after 24 hours (on a very lightly loaded test cluster) I'm
> still
> > seeing these scroll by a lot - 600 an hour per node.
> > Really curious to know if this is expected behaviour or if this is
> resulting
> > from some kind of node corruption.
> >
> > Cheers
> > Toby
> >
> >
> >
> > On Wed, 21 Oct 2015 at 12:23 Toby Corkindale <t...@dryft.net> wrote:
> >>
> >> Hi,
> >> I've been working on getting Riak to run inside Docker containers - in a
> >> multi-machine cluster. (Previous work I've seen has only run Riak as a
> >> cluster all on the same machine.)
> >> I thought I had it cracked, although I tripped up on the existing issue
> >> with Riak and lockfiles[1]. But the nodes have been generating an awful
> lot
> >> of errors like the below, and I wondered if anyone here can give me an
> >> explanation? (And, is it a problem?)
> >>
> >> 2015-10-21 01:19:23.567 [error] <0.24495.0> Error folding keys for
> >> "/var/lib/riak/bitcask.1h/2283596
> >> 30832953580969325755111919221821239459840/2.bitcask.data":
> >> {incomplete_hint,4}
> >>
> >> 1: Related issues to the lockfiles --
> >> I note that many are closed, but the problem still exists, and is
> >> particularly triggered by using Docker and stopping/killing Riak more
> >> violently than it likes.
> >> https://github.com/basho/bitcask/issues/163 (closed)
> >> https://github.com/basho/riak/issues/535 (open)
> >> https://github.com/basho/bitcask/issues/167 (closed)
> >> https://github.com/basho/bitcask/issues/99 (closed)
> >
> >
> > ___
> > riak-users mailing list
> > riak-users@lists.basho.com
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak in Docker - Error folding keys - incomplete_hint

[Toby Corkindale]

Quick follow-up: As a bit of a hack, deleting all the .hint files prior to
each start-up does resolve the errors, and immediately results in a whole
lot of Bitcask merges happening.
But that doesn't strike me as a good long-term fix.

On Fri, 23 Oct 2015 at 10:52 Toby Corkindale <t...@dryft.net> wrote:

> Hi Hector,
> You can see the Dockerfile here:
> https://gist.github.com/TJC/cb3184705bc0eacde885
>
> It's a work in progress, but also, not that involved.
>
> Ubuntu 14.04 is used as both the docker host, and the docker container.
> It's on the btrfs storage driver. (I've had too many issues with the other
> two)
> The Riak data directory is a volume, and is mounted to an external,
> persistent location. (Which is also btrfs)
>
> I suspect there's an issue around Riak shutting down uncleanly when the
> docker container is stopped.
> I have already had to add this to the start-up each time:
> find /var/lib/riak -name "bitcask.*.lock" -delete
>
> So it's clear that Riak is getting killed rather than shutting down
> cleanly; but even so, I'd hope that Riak would cope with that, rather than
> getting into a permanent state of throwing errors.
>
> Toby
>
>
> On Fri, 23 Oct 2015 at 00:01 Hector Castro <hectcas...@gmail.com> wrote:
>
>> Can't say I've paid enough attention to the logs in my single-machine
>> Riak within Docker setups to confirm.
>>
>> Do you have the container image definitions somewhere public? That may
>> help someone reproduce the issue. Also, did you ensure that the Riak
>> data directory is setup as a Docker volume?
>>
>> Other things that come to mind:
>>
>> - What OS is the Docker host running?
>> - What storage driver are you using for Docker?
>> - What file system is the Docker data directory using?
>>
>> --
>> Hector
>>
>>
>> On Thu, Oct 22, 2015 at 2:27 AM, Toby Corkindale <t...@dryft.net> wrote:
>> > Anyone?
>> >
>> > I note that after 24 hours (on a very lightly loaded test cluster) I'm
>> still
>> > seeing these scroll by a lot - 600 an hour per node.
>> > Really curious to know if this is expected behaviour or if this is
>> resulting
>> > from some kind of node corruption.
>> >
>> > Cheers
>> > Toby
>> >
>> >
>> >
>> > On Wed, 21 Oct 2015 at 12:23 Toby Corkindale <t...@dryft.net> wrote:
>> >>
>> >> Hi,
>> >> I've been working on getting Riak to run inside Docker containers - in
>> a
>> >> multi-machine cluster. (Previous work I've seen has only run Riak as a
>> >> cluster all on the same machine.)
>> >> I thought I had it cracked, although I tripped up on the existing issue
>> >> with Riak and lockfiles[1]. But the nodes have been generating an
>> awful lot
>> >> of errors like the below, and I wondered if anyone here can give me an
>> >> explanation? (And, is it a problem?)
>> >>
>> >> 2015-10-21 01:19:23.567 [error] <0.24495.0> Error folding keys for
>> >> "/var/lib/riak/bitcask.1h/2283596
>> >> 30832953580969325755111919221821239459840/2.bitcask.data":
>> >> {incomplete_hint,4}
>> >>
>> >> 1: Related issues to the lockfiles --
>> >> I note that many are closed, but the problem still exists, and is
>> >> particularly triggered by using Docker and stopping/killing Riak more
>> >> violently than it likes.
>> >> https://github.com/basho/bitcask/issues/163 (closed)
>> >> https://github.com/basho/riak/issues/535 (open)
>> >> https://github.com/basho/bitcask/issues/167 (closed)
>> >> https://github.com/basho/bitcask/issues/99 (closed)
>> >
>> >
>> > ___
>> > riak-users mailing list
>> > riak-users@lists.basho.com
>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> >
>>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak in Docker - Error folding keys - incomplete_hint

[Toby Corkindale]

Anyone?

I note that after 24 hours (on a very lightly loaded test cluster) I'm
still seeing these scroll by a lot - 600 an hour per node.
Really curious to know if this is expected behaviour or if this is
resulting from some kind of node corruption.

Cheers
Toby



On Wed, 21 Oct 2015 at 12:23 Toby Corkindale <t...@dryft.net> wrote:

> Hi,
> I've been working on getting Riak to run inside Docker containers - in a
> multi-machine cluster. (Previous work I've seen has only run Riak as a
> cluster all on the same machine.)
> I thought I had it cracked, although I tripped up on the existing issue
> with Riak and lockfiles[1]. But the nodes have been generating an awful lot
> of errors like the below, and I wondered if anyone here can give me an
> explanation? (And, is it a problem?)
>
> 2015-10-21 01:19:23.567 [error] <0.24495.0> Error folding keys for
> "/var/lib/riak/bitcask.1h/2283596
> 30832953580969325755111919221821239459840/2.bitcask.data":
> {incomplete_hint,4}
>
> 1: Related issues to the lockfiles --
> I note that many are closed, but the problem still exists, and is
> particularly triggered by using Docker and stopping/killing Riak more
> violently than it likes.
> https://github.com/basho/bitcask/issues/163 (closed)
> https://github.com/basho/riak/issues/535 (open)
> https://github.com/basho/bitcask/issues/167 (closed)
> https://github.com/basho/bitcask/issues/99 (closed)
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Riak in Docker - Error folding keys - incomplete_hint

[Toby Corkindale]

 Hi,
I've been working on getting Riak to run inside Docker containers - in a
multi-machine cluster. (Previous work I've seen has only run Riak as a
cluster all on the same machine.)
I thought I had it cracked, although I tripped up on the existing issue
with Riak and lockfiles[1]. But the nodes have been generating an awful lot
of errors like the below, and I wondered if anyone here can give me an
explanation? (And, is it a problem?)

2015-10-21 01:19:23.567 [error] <0.24495.0> Error folding keys for
"/var/lib/riak/bitcask.1h/2283596
30832953580969325755111919221821239459840/2.bitcask.data":
{incomplete_hint,4}

1: Related issues to the lockfiles --
I note that many are closed, but the problem still exists, and is
particularly triggered by using Docker and stopping/killing Riak more
violently than it likes.
https://github.com/basho/bitcask/issues/163 (closed)
https://github.com/basho/riak/issues/535 (open)
https://github.com/basho/bitcask/issues/167 (closed)
https://github.com/basho/bitcask/issues/99 (closed)
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Problem with Riak not thinking hostname is fully-qualified

[Toby Corkindale]

I found the problem. It was quite annoying.
Essentially -- the systems were all valid, but Riak wouldn't recognise the
fully-qualified hostnames until I rm -rf /var/lib/riak/* and restarted riak.

On Thu, 13 Aug 2015 at 13:52 Toby Corkindale t...@dryft.net wrote:

 I'm trying to set up another cluster, and I'm hitting problems with Riak
 complaining that ** System running to use fully qualified hostnames **
 ** Hostname db04 is illegal **

 However, as far as I can see, the host does have a fully-qualified
 hostname:

 root@db04:/# hostname
 db04
 root@db04:/# hostname -f
 db04.dc.sdlocal.net

 /etc/hosts contains a line with both the short and fqdn names in it.

 Is there something else I should be checking?

 Cheers
 Toby

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Problem with Riak not thinking hostname is fully-qualified

[Toby Corkindale]

I should have mentioned, the actual Riak.conf has nodename set to the FQDN.
The FQDN resolves correctly in our DNS.

On Thu, 13 Aug 2015 at 13:52 Toby Corkindale t...@dryft.net wrote:

 I'm trying to set up another cluster, and I'm hitting problems with Riak
 complaining that ** System running to use fully qualified hostnames **
 ** Hostname db04 is illegal **

 However, as far as I can see, the host does have a fully-qualified
 hostname:

 root@db04:/# hostname
 db04
 root@db04:/# hostname -f
 db04.dc.sdlocal.net

 /etc/hosts contains a line with both the short and fqdn names in it.

 Is there something else I should be checking?

 Cheers
 Toby

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: is it possible to use FQDN instead of IP in riak.conf for listener.http.$name field?

[Toby Corkindale]

Hi Roman,
I just set the IP to 0.0.0.0 as then it'll bind to whatever IPs are
available, even if they change.

Obviously you want to make sure your firewall isn't allowing random
internet addresses into the Riak ports, though.

-Toby

On Tue, 21 Jul 2015 at 06:48 ROMAN SHESTAKOV romanshesta...@yahoo.co.uk
wrote:


 Hello,

 is there any way to specify in riak.conf in field listener.http.$name”
 FQDN instead of IP address?

 listener.http.$name  This is the IP address and TCP port to which the
 Riak HTTP interface will bind. {127.0.0.1,8098}

 why is it required to use IP address instead of a host name? In my setting
 RIAK_KV is deployed on dynamic VMs and IP could be changed if the VM is
 launched on a different hypervisor.


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: haproxy issues with protocol buffers on 2.x releases

[Toby Corkindale]

Ah hah!
Coming back to this problem after a month seems to have jogged my
thoughts. I've
figured it out.

The Basho guide to haproxy for Riak CS specifies:
timeout client5000
timeout server5000

In the example configuration. Those are values in milliseconds.
So if the client or server does not send any data over the connection for
5000 milliseconds, haproxy considers it dead, and closes the connection.

On a moderately well loaded production system, you probably have enough
data coming and going to keep those connections alive, but on a testing
instance, the connections will be getting closed down all the time.

My fix is to add these lines:
timeout tunnel 7d
timeout client-fin 30s

But I'd appreciate your thoughts too.

-Toby



On Mon, 6 Jul 2015 at 13:13 Toby Corkindale t...@dryft.net wrote:

 Hi Matt,
 I've tested against both haproxy 1.4 and haproxy 1.5,  both Riak 2.0.5 and
 2.1.1, and Riak CS 2.0.1 and Stanchion 2.0.0.
 I've test both KVM and LXC virtualisation.

 In all combinations I tested, the problem persists. If Riak CS connects to
 Riak via haproxy, then Riak CS frequently loses the connection and returns
 a failure to the s3 client.

 The failures show up very quickly, if you want to try and replicate this.
 I'll upload the configurations for you. They're for a three-node cluster,
 which is obviously too small for production, but should be the minimum to
 demonstrate this bug, right? I'm sure it'd manifest on a four or five node
 cluster too.

 I run this command:
 perl mkfiles.pl; s3cmd -c s3cfg sync *.txt s3://test/
 (mkfiles.pl just creates 100 small unique files by writing a bit of junk
 and an index value to them.)

 If Riak-CS is talking to the haproxy port instead of directly to Riak,
 it'll fail partway through the sync. Every time. The number of files it
 gets through varies between 1 and 40ish, I'd say, but I haven't actually
 kept track.

 Configuration files and test script at:
 https://www.dropbox.com/s/cnt9y0fegrb42fb/haproxy-riak-cs-issue.zip?dl=0

 Toby

 On Sat, 4 Jul 2015 at 01:30 Matthew Brender mbren...@basho.com wrote:

 Hey Toby,

 Did you find anything further during this testing? I'd love to make sure
 others on riak-user know how to configure local testing to prevent this
 situation.

 Cheers,
 Matt


 *Matt Brender | Developer Advocacy Lead*
 Basho Technologies
 t: @mjbrender https://twitter.com/mjbrender
 c: +1 617.817.3195

 On Fri, Jun 5, 2015 at 2:16 AM, Toby Corkindale t...@dryft.net wrote:

 Hi Kota,
 Our production nodes are Riak CS 1.5 and Riak 1.4.x -- they're running
 haproxy 1.4.x, and it's all been happy for some time now.

 Testing the new nodes, still same haproxy versions, but Riak CS 2.0.1
 and Riak 1.0.5.
 Very confused as to why the connections are being dropped when going
 through haproxy. The problem persists even after restarting CS.
 I tried staggering the restarts.. increasing the PB request pool.. etc..
  no change.

 But it works fine if CS connects directly to the localhost riak pb.
 (Which isn't a great idea.. big Riak instances sometimes take too long
 to start, and CS falls over because it started too fast and couldn't
 connect, if you're going to localhost)

 Confusing! I'm wondering if it's because the testing machines are in
 virtual machines, compared to production which is real hardware.
 But.. normally haproxy still works fine on VMs.

 I'll continue to play around.. Must be something that's botched on the
 testing setup... but don't want to replicate that into production!


 On Fri, 5 Jun 2015 at 13:59 Kota Uenishi k...@basho.com wrote:

 Toby,

 As PB connection management haven't been changed between CS 1.5 and
 2.0, I think it should work. What's the version the load balancing
 working stable? It depends of the reason why connection has been cut,
 but I would recommend you restart just the CS node and recreate the
 connection pool.

 On Thu, Jun 4, 2015 at 2:33 PM, Toby Corkindale t...@dryft.net wrote:
  Hi,
  I've been happily using haproxy in front of Riak and Riak CS 1.x in
  production for quite a while.
 
  I've been trying to bring up a new cluster based on riak/cs 2.0.x
 recently,
  as you've probably noticed from the flurry of emails to this list :)
 
  I'm discovering that if I have haproxy sitting between riak-cs and
 riak,
  then I get a lot of errors about disconnections. Initially I thought
 this
  must be related to pb backlogs or pool sizes or file handle limits --
 but
  I've played with all those things to no avail.
 
  I *have* noticed that if I get riak-cs to connect directly to a riak
  (bypassing haproxy) then everything is fine, including with the
 original
  default request pool and backlog sizes.
 
  I am essentially using the recommended haproxy.cfg, which has worked
 fine in
  production elsewhere.
 
  Any suggestions?
  Error message sample follows:
 
  2015-06-04 15:26:16.447 [warning]
  0.283.0@riak_cs_riak_client:get_user_with_pbc:293 Fetching user re
  cord with strong

Re: Riak-CS RAM usage

[Toby Corkindale]

Bitcask keeps all the *keys* in RAM. That's like an index in database terms.
So even if you have 9Gb of data, you should only have a tiny fraction of
that in keys, in memory.

So you might have some other problem going on there?

Riak uses quite a bit of memory for LevelDB buffers, by default, so maybe
you're seeing something like that?

-Toby

On Thu, 2 Jul 2015 at 21:55 Anita Wong anita.w...@talkboxapp.com wrote:

 HI,

 I’ve got a problem in using Riak-CS as file storage cluster.

 Now the bitcask folder has 9GB of data in it, and the RAM usage of Riak is
 8GB.

 Just checked that the bitcask backend will put all the keys into the RAM,
 but how come now I would get this large amount of RAM being used?


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Bug: Stanchion 1.5 in repo along riak cs 2.0

[Toby Corkindale]

Glad we got to the bottom of it.
You should probably look at updating your docs for Riak-CS and removing
stanchion 1.5 from the repo to avoid future confusion?

On Wed, 10 Jun 2015 at 00:46 Kota Uenishi k...@basho.com wrote:

 I got it! Good catch, Chris.

 Toby, you should follow these instructions to get the latest Stanchion:

 https://packagecloud.io/basho/stanchion/install

 as it looks like we have different repository since 2.0. I did follow
 these instructions prior to try my test earlier in this loop.

 For Riak CS package, this instructions might be more precise because
 it's automatically generated:

 https://packagecloud.io/basho/riak-cs/install

 On Tue, Jun 9, 2015 at 11:02 PM, Christopher Mancini cmanc...@basho.com
 wrote:
  Kota,
 
  I think Toby is right, if you look at packagecloud.io, there is no
 reference
  to stanchion 2, only 1.5.
 
  https://packagecloud.io/basho/riak-cs?filter=debs
 
  Not sure who can fix this, but its certainly not aligned with our
  recommendations.
 
 
  On Mon, Jun 8, 2015 at 9:04 PM, Toby Corkindale t...@dryft.net wrote:
 
  Hi Kota,
  I was installing these on fresh 14.04 VMs, so I don't know how I could
  have had stale apt-cache info re Stanchion, but still had up to date
 riak-cs
  package info.
  Oh well, if you couldn't reproduce it, never mind. Thanks for trying.
 
  Cheers
  Toby
 
  On Mon, 8 Jun 2015 at 11:54 Kota Uenishi k...@basho.com wrote:
 
  Toby,
 
  I tried to reproduce your problem in my clean ubuntu 14.04 which is a
  fresh docker image, but failed. No recommendation was made. Maybe
 state of
  your apt repository or something is stale?
 
  # apt-get install riak-cs=2.0.1-1
 
  Reading package lists... Done
 
  Building dependency tree
 
  Reading state information... Done
 
  The following NEW packages will be installed:
 
riak-cs
 
  0 upgraded, 1 newly installed, 0 to remove and 25 not upgraded.
 
  Need to get 20.1 MB of archives.
 
  After this operation, 36.8 MB of additional disk space will be used.
 
  Fetched 20.1 MB in 6s (3322 kB/s)
 
  Selecting previously unselected package riak-cs.
 
  (Reading database ... 12002 files and directories currently installed.)
 
  Preparing to unpack .../riak-cs_2.0.1-1_amd64.deb ...
 
  Unpacking riak-cs (2.0.1-1) ...
 
  Processing triggers for ureadahead (0.100.0-16) ...
 
  Setting up riak-cs (2.0.1-1) ...
 
  Adding group `riak' (GID 105) ...
 
  Done.
 
  Adding system user `riakcs' (UID 102) ...
 
  Adding new user `riakcs' (UID 102) with group `riak' ...
 
  Not creating home directory `/var/lib/riak-cs'.
 
  Processing triggers for ureadahead (0.100.0-16) ...
 
  # apt-get install stanchion=2.0.0-1
 
  Reading package lists... Done
 
  Building dependency tree
 
  Reading state information... Done
 
  The following NEW packages will be installed:
 
stanchion
 
  0 upgraded, 1 newly installed, 0 to remove and 25 not upgraded.
 
  Need to get 18.4 MB of archives.
 
  After this operation, 34.0 MB of additional disk space will be used.
 
  Fetched 18.4 MB in 5s (3163 kB/s)
 
  Selecting previously unselected package stanchion.
 
  (Reading database ... 13780 files and directories currently installed.)
 
  Preparing to unpack .../stanchion_2.0.0-1_amd64.deb ...
 
  Unpacking stanchion (2.0.0-1) ...
 
  Processing triggers for ureadahead (0.100.0-16) ...
 
  Setting up stanchion (2.0.0-1) ...
 
  Adding system user `stanchion' (UID 103) ...
 
  Adding new user `stanchion' (UID 103) with group `riak' ...
 
  Not creating home directory `/var/lib/stanchion'.
 
  Processing triggers for ureadahead (0.100.0-16) ...
 
  #
 
 
 
  On Fri, Jun 5, 2015 at 11:40 PM, Matthew Brender mbren...@basho.com
  wrote:
 
  For others not finding it, the instructions show up here:
 
 
 https://packagecloud.io/basho/riak-cs/packages/ubuntu/trusty/riak-cs_2.0.0-1_amd64.deb
 
 
  Matt Brender | Developer Advocacy Lead
  Basho Technologies
  t: @mjbrender
 
 
  On Fri, Jun 5, 2015 at 2:11 AM, Toby Corkindale t...@dryft.net
 wrote:
   Hi Kota,
   I'm installing Riak CS upon Ubuntu 14.04 (trusty), and was doing
 this
   by
   following the instructions to add the packagecloud.io Apt
 repository.
   However that repository contains stanchion 1.5 rather than 2.0. (It
   does,
   however, contain Riak CS 2.0.1)
  
 
 
  ___
  riak-users mailing list
  riak-users@lists.basho.com
  http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
 
 
 
 
  --
  Sincerely,
 
  Christopher Mancini
  -
 
  employee = {
  purpose: solve problems with code,
  phone:7164625591,
  email: cmanc...@basho.com,
  github:http://www.github.com/christophermancini
  }



 --
 Kota UENISHI / @kuenishi
 Basho Japan KK

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Bug: Stanchion 1.5 in repo along riak cs 2.0

[Toby Corkindale]

Hi Kota,
I was installing these on fresh 14.04 VMs, so I don't know how I could have
had stale apt-cache info re Stanchion, but still had up to date riak-cs
package info.
Oh well, if you couldn't reproduce it, never mind. Thanks for trying.

Cheers
Toby

On Mon, 8 Jun 2015 at 11:54 Kota Uenishi k...@basho.com wrote:

 Toby,

 I tried to reproduce your problem in my clean ubuntu 14.04 which is a
 fresh docker image, but failed. No recommendation was made. Maybe state of
 your apt repository or something is stale?

 # apt-get install riak-cs=2.0.1-1

 Reading package lists... Done

 Building dependency tree

 Reading state information... Done

 The following NEW packages will be installed:

   riak-cs

 0 upgraded, 1 newly installed, 0 to remove and 25 not upgraded.

 Need to get 20.1 MB of archives.

 After this operation, 36.8 MB of additional disk space will be used.

 Fetched 20.1 MB in 6s (3322 kB/s)

 Selecting previously unselected package riak-cs.

 (Reading database ... 12002 files and directories currently installed.)

 Preparing to unpack .../riak-cs_2.0.1-1_amd64.deb ...

 Unpacking riak-cs (2.0.1-1) ...

 Processing triggers for ureadahead (0.100.0-16) ...

 Setting up riak-cs (2.0.1-1) ...

 Adding group `riak' (GID 105) ...

 Done.

 Adding system user `riakcs' (UID 102) ...

 Adding new user `riakcs' (UID 102) with group `riak' ...

 Not creating home directory `/var/lib/riak-cs'.

 Processing triggers for ureadahead (0.100.0-16) ...

 # apt-get install stanchion=2.0.0-1

 Reading package lists... Done

 Building dependency tree

 Reading state information... Done

 The following NEW packages will be installed:

   stanchion

 0 upgraded, 1 newly installed, 0 to remove and 25 not upgraded.

 Need to get 18.4 MB of archives.

 After this operation, 34.0 MB of additional disk space will be used.

 Fetched 18.4 MB in 5s (3163 kB/s)

 Selecting previously unselected package stanchion.

 (Reading database ... 13780 files and directories currently installed.)

 Preparing to unpack .../stanchion_2.0.0-1_amd64.deb ...

 Unpacking stanchion (2.0.0-1) ...

 Processing triggers for ureadahead (0.100.0-16) ...

 Setting up stanchion (2.0.0-1) ...

 Adding system user `stanchion' (UID 103) ...

 Adding new user `stanchion' (UID 103) with group `riak' ...

 Not creating home directory `/var/lib/stanchion'.

 Processing triggers for ureadahead (0.100.0-16) ...

 #


 On Fri, Jun 5, 2015 at 11:40 PM, Matthew Brender mbren...@basho.com
 wrote:

 For others not finding it, the instructions show up here:

 https://packagecloud.io/basho/riak-cs/packages/ubuntu/trusty/riak-cs_2.0.0-1_amd64.deb


 Matt Brender | Developer Advocacy Lead
 Basho Technologies
 t: @mjbrender


 On Fri, Jun 5, 2015 at 2:11 AM, Toby Corkindale t...@dryft.net wrote:
  Hi Kota,
  I'm installing Riak CS upon Ubuntu 14.04 (trusty), and was doing this by
  following the instructions to add the packagecloud.io Apt repository.
  However that repository contains stanchion 1.5 rather than 2.0. (It
 does,
  however, contain Riak CS 2.0.1)
 


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak CS docs have incorrect packagecloud URL

[Toby Corkindale]

Hi Matt,
This page contains the incorrect URLs:
http://docs.basho.com/riakcs/latest/cookbooks/installing/Installing-Riak-CS/

If you search on that page for this text, you'll find it right
underneath. The resulting file should hold contents like the following

Cheers,
Toby

On Thu, 4 Jun 2015 at 06:31 Matthew Brender mbren...@basho.com wrote:

 For what it's worth, it looks good on the packagecloud directions as
 of the most recent update [1]:

   curl -s
 https://packagecloud.io/install/repositories/basho/riak-cs/script.deb.sh
 | sudo bash
   sudo apt-get install riak-cs=2.0.1-1

 [1]
 https://packagecloud.io/basho/riak-cs/packages/ubuntu/precise/riak-cs_2.0.1-1_amd64.deb


 Matt Brender | Developer Advocacy Lead
 Basho Technologies
 t: @mjbrender



 On Wed, Jun 3, 2015 at 4:02 PM, Matthew Brender mbren...@basho.com
 wrote:
  Hey Toby,
 
  Which documentation pointed you to the wrong URL? And did the dash
  address the issue? We can and should update that documentation.
 
  On a related note, our packaging is up for discussion. A few of us
  started discussing it here [1]. I'd love for you to weigh in.
 
  [1] https://github.com/basho-labs/the-riak-community/issues/49
 
  Thanks,
  Matt
 
  Matt Brender | Developer Advocacy Lead
  Basho Technologies
  t: @mjbrender
 
 
  On Tue, Jun 2, 2015 at 10:07 PM, Toby Corkindale t...@dryft.net wrote:
 
  Hi,
  Riak CS 2 moved to a new Ubuntu/Debian repository, at packagecloud.io.
  However attempts to use it as per the documentation result in the
 message Incorrect username or token and an HTTP status of 401.
 
  This is because the docs give the URL as similar to:
  deb https://packagecloud.io/basho/riak_cs/ubuntu/ precise main
 
  However they are supposed to be?:
 
  deb https://packagecloud.io/basho/riak-cs/ubuntu/ precise main
 
  Note the dash instead of an underscore.
 
  -Toby
 
  ___
  riak-users mailing list
  riak-users@lists.basho.com
  http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
 

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: haproxy issues with protocol buffers on 2.x releases

[Toby Corkindale]

Hi Kota,
Our production nodes are Riak CS 1.5 and Riak 1.4.x -- they're running
haproxy 1.4.x, and it's all been happy for some time now.

Testing the new nodes, still same haproxy versions, but Riak CS 2.0.1 and
Riak 1.0.5.
Very confused as to why the connections are being dropped when going
through haproxy. The problem persists even after restarting CS.
I tried staggering the restarts.. increasing the PB request pool.. etc..
 no change.

But it works fine if CS connects directly to the localhost riak pb.
(Which isn't a great idea.. big Riak instances sometimes take too long to
start, and CS falls over because it started too fast and couldn't connect,
if you're going to localhost)

Confusing! I'm wondering if it's because the testing machines are in
virtual machines, compared to production which is real hardware.
But.. normally haproxy still works fine on VMs.

I'll continue to play around.. Must be something that's botched on the
testing setup... but don't want to replicate that into production!


On Fri, 5 Jun 2015 at 13:59 Kota Uenishi k...@basho.com wrote:

 Toby,

 As PB connection management haven't been changed between CS 1.5 and
 2.0, I think it should work. What's the version the load balancing
 working stable? It depends of the reason why connection has been cut,
 but I would recommend you restart just the CS node and recreate the
 connection pool.

 On Thu, Jun 4, 2015 at 2:33 PM, Toby Corkindale t...@dryft.net wrote:
  Hi,
  I've been happily using haproxy in front of Riak and Riak CS 1.x in
  production for quite a while.
 
  I've been trying to bring up a new cluster based on riak/cs 2.0.x
 recently,
  as you've probably noticed from the flurry of emails to this list :)
 
  I'm discovering that if I have haproxy sitting between riak-cs and riak,
  then I get a lot of errors about disconnections. Initially I thought this
  must be related to pb backlogs or pool sizes or file handle limits -- but
  I've played with all those things to no avail.
 
  I *have* noticed that if I get riak-cs to connect directly to a riak
  (bypassing haproxy) then everything is fine, including with the original
  default request pool and backlog sizes.
 
  I am essentially using the recommended haproxy.cfg, which has worked
 fine in
  production elsewhere.
 
  Any suggestions?
  Error message sample follows:
 
  2015-06-04 15:26:16.447 [warning]
  0.283.0@riak_cs_riak_client:get_user_with_pbc:293 Fetching user re
  cord with strong option failed: disconnected
  2015-06-04 15:26:16.447 [warning]
  0.2095.0@riak_cs_pbc:check_connection_status:97 Connection status
  of 0.287.0 at maybe_create_user: {false,[]}
 
 
 
  Cheers
  Toby
 
  ___
  riak-users mailing list
  riak-users@lists.basho.com
  http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
 



 --
 Kota UENISHI / @kuenishi
 Basho Japan KK

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Bug: Stanchion 1.5 in repo along riak cs 2.0

[Toby Corkindale]

Hi Kota,
I'm installing Riak CS upon Ubuntu 14.04 (trusty), and was doing this by
following the instructions to add the packagecloud.io Apt repository.
However that repository contains stanchion 1.5 rather than 2.0. (It does,
however, contain Riak CS 2.0.1)

Cheers
Toby


On Fri, 5 Jun 2015 at 13:02 Kota Uenishi k...@basho.com wrote:

 Toby, thank you for reporting. However, I could not figure out which
 part of the doc is wrong and leading to Stanchion 1.5. Could you
 elaborate more details?

 Technically, it is correct to use Stanchion 2.0 with Riak CS 2.0.1.
 And more, difference between Stanchion 2.0 and 1.5 is very small, it's
 just about configuration file format, and several items.

 On Thu, Jun 4, 2015 at 1:45 PM, Toby Corkindale t...@dryft.net wrote:
  Hi,
  I think this is a bug?
 
  Riak CS 2.0.1 recommends Stanchion 2.0.0 to be installed.
 
  However if you follow the instructions to add the Riak CS repo, you get
 Riak
  CS 2.0.1, and Stanchion 1.5.0.
 
  Toby
 
  ___
  riak-users mailing list
  riak-users@lists.basho.com
  http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
 



 --
 Kota UENISHI / @kuenishi
 Basho Japan KK

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Bug: Stanchion 1.5 in repo along riak cs 2.0

[Toby Corkindale]

Hi,
I think this is a bug?

Riak CS 2.0.1 recommends Stanchion 2.0.0 to be installed.

However if you follow the instructions to add the Riak CS repo, you get
Riak CS 2.0.1, and Stanchion 1.5.0.

Toby
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

haproxy issues with protocol buffers on 2.x releases

[Toby Corkindale]

Hi,
I've been happily using haproxy in front of Riak and Riak CS 1.x in
production for quite a while.

I've been trying to bring up a new cluster based on riak/cs 2.0.x recently,
as you've probably noticed from the flurry of emails to this list :)

I'm discovering that if I have haproxy sitting between riak-cs and riak,
then I get a lot of errors about disconnections. Initially I thought this
must be related to pb backlogs or pool sizes or file handle limits -- but
I've played with all those things to no avail.

I *have* noticed that if I get riak-cs to connect directly to a riak
(bypassing haproxy) then everything is fine, including with the original
default request pool and backlog sizes.

I am essentially using the recommended haproxy.cfg, which has worked fine
in production elsewhere.

Any suggestions?
Error message sample follows:

 2015-06-04 15:26:16.447 [warning]
0.283.0@riak_cs_riak_client:get_user_with_pbc:293 Fetching user re
cord with strong option failed: disconnected
2015-06-04 15:26:16.447 [warning]
0.2095.0@riak_cs_pbc:check_connection_status:97 Connection status
of 0.287.0 at maybe_create_user: {false,[]}



Cheers
Toby
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Riak CS docs have incorrect packagecloud URL

[Toby Corkindale]

Hi,
Riak CS 2 moved to a new Ubuntu/Debian repository, at packagecloud.io.
However attempts to use it as per the documentation result in the message
Incorrect username or token and an HTTP status of 401.

This is because the docs give the URL as similar to:
deb https://packagecloud.io/basho/riak_cs/ubuntu/ precise main

However they are supposed to be?:

deb https://packagecloud.io/basho/riak-cs/ubuntu/ precise main

Note the dash instead of an underscore.

-Toby
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Riak CS 2 backend config confusion

[Toby Corkindale]

Hi
I'm in the process of moving from a Riak 1.4.x w/CS installation, to a Riak
CS 2 installation, and I'm confused by comments in the riak.conf compared
to the documentation online for riak cs, regarding the backend.

riak.conf contains the following lines:

 ## Specifies the storage engine used for Riak's key-value data
## and secondary indexes (if supported).
##
## Default: bitcask
##
## Acceptable values:
##   - one of: bitcask, leveldb, memory, multi, prefix_multi
storage_backend = bitcask

## Simplify prefix_multi configuration for Riak CS. Keep this
## commented out unless Riak is configured for Riak CS.
##
## Acceptable values:
##   - an integer
## cs_version = 2

What is that bit about cs_version and prefix_multi there for?
There's no mention of that here:
http://docs.basho.com/riakcs/latest/cookbooks/configuration/Configuring-Riak/


Cheers,
Toby
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Ansible galaxy roles only support three-year-old Ubuntu 12.04

[Toby Corkindale]

Hi guys,
I've sent three small pull requests through, addressing different issues.
I've only been testing my changes on Ubuntu 12.04 and 14.04 (the two most
recent LTS versions) but have left some notes referring to points that are
worth you checking on Red Hat.

There's still some big issues with the role failing if certain variables
haven't been defined -- but I'm not sure if that's because you're in the
middle of refactoring the way that stuff works, or some other
in-development plan you haven't finished yet.

There are also issues where buckets and security settings will fail on
everything after the first run, because they already exist. ie. The
commands aren't idempotent.

Cheers
Toby

On Thu, 28 May 2015 at 15:41 Toby Corkindale t...@dryft.net wrote:

 Hi guys,
 I'm afraid I'm already wearing too many hats to pick up the role of
 maintainer of the Ansible-riak repository. However, definitely happy to
 help out a little.
 I'll take a look at the dev-2.x branch and see what I have to change to
 make it run on Ubuntu 14.04.

 Cheers,
 Toby


  On 27 May 2015 at 18:34, Matthew Brender mbren...@basho.com wrote:
  Hey Toby,
 
  While I'm not thrilled our role is out of date, I'm excited to hear
  you care. We're actively looking for a maintainer of the Ansible Riak
  repo [1]. A quick review of the branching shows that it's a little
  wild west in there.
 
  If you're game to step up as the maintainer, I'd love to sync up with
  you. We have some great resources (people and existing code) and we're
  in need of someone who wants to curate them all.
 
  [1] https://github.com/basho-labs/ansible-riak
 
  Thanks,
 
  Matt Brender | Developer Advocacy Lead
  Basho Technologies
  t: @mjbrender
 
 
 
  On Wed, May 27, 2015 at 5:24 AM, Christopher Mancini 
 cmanc...@basho.com wrote:
  Hi Toby,
 
  I am actually working on a rewrite of the Galaxy role that we used to
 have
  up there to be simpler and support all of the 2.x features of Riak.
 You can
  access it here:
 https://github.com/basho-labs/ansible-riak/tree/dev-2.x
 
  I would greatly appreciate any feedback, suggestions as well as any
  contributions you may have from your testing on Ubuntu / Debian,
 since I am
  using it with CentOS7 and may not have time to test it with Ubuntu
 for a
  little while still.
 
  Chris
 


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Ansible galaxy roles only support three-year-old Ubuntu 12.04

[Toby Corkindale]

Hi,
Searching for riak roles on Ansible Galaxy brings up this one:
https://galaxy.ansible.com/list#/roles/777

However this is listed as only supporting RHEL 6 and Ubuntu 12.04.

Ubuntu 14.04 is the current long-term-support version, and what we're
deploying stuff to; it'd be great if the Ansible roles available could
support it too.

Cheers,
Toby
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Ansible galaxy roles only support three-year-old Ubuntu 12.04

[Toby Corkindale]

Hi guys,
I'm afraid I'm already wearing too many hats to pick up the role of
maintainer of the Ansible-riak repository. However, definitely happy to
help out a little.
I'll take a look at the dev-2.x branch and see what I have to change to
make it run on Ubuntu 14.04.

Cheers,
Toby


  On 27 May 2015 at 18:34, Matthew Brender mbren...@basho.com wrote:
  Hey Toby,
 
  While I'm not thrilled our role is out of date, I'm excited to hear
  you care. We're actively looking for a maintainer of the Ansible Riak
  repo [1]. A quick review of the branching shows that it's a little
  wild west in there.
 
  If you're game to step up as the maintainer, I'd love to sync up with
  you. We have some great resources (people and existing code) and we're
  in need of someone who wants to curate them all.
 
  [1] https://github.com/basho-labs/ansible-riak
 
  Thanks,
 
  Matt Brender | Developer Advocacy Lead
  Basho Technologies
  t: @mjbrender
 
 
 
  On Wed, May 27, 2015 at 5:24 AM, Christopher Mancini 
 cmanc...@basho.com wrote:
  Hi Toby,
 
  I am actually working on a rewrite of the Galaxy role that we used to
 have
  up there to be simpler and support all of the 2.x features of Riak.
 You can
  access it here:
 https://github.com/basho-labs/ansible-riak/tree/dev-2.x
 
  I would greatly appreciate any feedback, suggestions as well as any
  contributions you may have from your testing on Ubuntu / Debian, since
 I am
  using it with CentOS7 and may not have time to test it with Ubuntu for
 a
  little while still.
 
  Chris
 


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Prometheus stats gathering for Riak? Riak CS?

[Toby Corkindale]

Hi,
I wondered if anyone has written a stats gathering plugin for Prometheus?
It doesn't look like it'd be too hard to do; but I'm still lazy enough to
hope that someone else has done it first :)

http://prometheus.io/

Toby
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak CS expiration

[Toby Corkindale]

On 7 February 2015 at 06:21, Kota Uenishi k...@basho.com wrote:
 Bitcask doesn't have such interface like hooking anything. As of
 internal data structure of Riak CS built on top of Riak, bitcask keys
 don't correspond simply to leveldb key. Say, things like this cannot
 happen: all tombstones of blocks of this object were collected and
 space were reclaimed by merged, so I'm gonna officially hit S3
 interface to delete that object. This is because blocks are
 distributed all across the cluster.

 I'd rather recommend to make up a simple cron job that walks over
 objects and see timestamp and hit CS API to delete them when they're
 old, or wait for lifecycle API gets implemented.

Definitely. And it's seriously only a few lines of code -- mine is
implemented like this, in Scala:

val allFiles = s3.listObjects(sys.env(S3_BUCKET))

allFiles filter(_.getLastModifiedDate.before(expiryDate)) foreach { f =
  logger.info(sExpiring file=${f.getKey})
  s3.deleteObject(bucket, f.getKey)
}


 On Fri, Feb 6, 2015 at 12:24 AM, Damien Krotkine dkrotk...@gmail.com wrote:

 Hi,

 As far as I understand, Riak CS uses bitcask for data storage and
 leveldb for metadata.

 Trying to implement expiration of data in riak cs, bitcask expiration
 works fine, but of course metadata are still in the leveldb backend. Is
 there any way to expire metadata from riak cs automatically ?

 Ideally, I'd like to be able to install a hook or callback on the
 bitcask compaction. Is that possible ?

 Thanks,
 dams

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



 --
 Kota UENISHI / @kuenishi
 Basho Japan KK

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



-- 
Turning and turning in the widening gyre
The falcon cannot hear the falconer
Things fall apart; the center cannot hold
Mere anarchy is loosed upon the world

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: RiakCS poor s3 upload speeds 2MB/s

[Toby Corkindale]

I turned the triggers and thresholds down, to:

 {max_file_size,805306368}, %% 768 MB
 {dead_bytes_merge_trigger, 134217728}, %% dead bytes  128 MB
 {dead_bytes_threshold,  33554432}  %% dead bytes  32 MB

And restarted nodes; however after 24 hours, disk utilisation remains the same.
ie. For about 6GB of files in CS, 600GB is on disk.
(Previously, when we had 100 GB in CS, we had terabytes on disk)

I do wonder if we hit some kind of issue with Riak CS earlier in the
cluster's life, and have somehow ended up with a lot of dead bytes
in there.
This is just data stored by Riak CS, so it should take responsibility
for siblings and their resolution, yes?

If I write something to download files and then re-upload them again,
to the same path, would that cause Riak CS to fix up any issues around
siblings or duplicately-stored data?

Cheers,
Toby

On 22 January 2015 at 03:20, Luke Bakken lbak...@basho.com wrote:
 You should be able to help with disk usage by turning down the
 trigger and threshold values described here:

 http://docs.basho.com/riak/latest/ops/advanced/backends/bitcask/

 Your cluster will merge more data which should help with disk usage.
 If your typical use is to create and delete objects frequently, this
 will help.

 --
 Luke Bakken
 Engineer
 lbak...@basho.com

 On Wed, Jan 21, 2015 at 4:40 AM, Toby Corkindale t...@dryft.net wrote:
 On 21 January 2015 at 15:22, Luke Bakken lbak...@basho.com wrote:
 Hi Toby -

 Are you using the stock bitcask configuration for merging?

 Hi Luke,
 Yes, pretty much.

 On Tue, Jan 20, 2015 at 5:07 PM, Toby Corkindale t...@dryft.net wrote:
 Hi Kota,
 I had a bit of an off-list chat about this a while ago, plus continued
 to investigate locally, and eventually achieved some faster speeds,
 around 15MByte/sec writes.
 Things that were changed:
  * Adjusted Riak CS GC to be spread out over the cluster much more.
  * Tweaked up the put buffers and concurrency further
  * Moved most of the files out of CS and into Amazon S3+Glacier
  * Switched from nginx to haproxy
  * simplified firewalling for internal clients

 Each one of those changes made a small to modest improvement, but
 overall combined to make a quite noticeable improvement.

 I did notice something odd though -- despite moving most of the data
 out of the cluster, the disk-space-in-use by Riak is still very large
 compared to the amount stored. I mean, we moved more than 90% of the
 data out of the cluster, yet the actual disk space used only halved.
 For every gigabyte of file stored in CS, dozens of gigabytes are
 actually on disk!

 Either the garbage collection algorithm is very, very lazy, or somehow
 something has gone a bit wrong in the past, which might have explained
 part of the performance problems.

 We're going to look at redeploying a new, fresh cluster based on Riak
 2 in the not too distant future, once Riak CS looks like it's approved
 for use there, and maybe that'll clear all of this up.

 Toby



-- 
Turning and turning in the widening gyre
The falcon cannot hear the falconer
Things fall apart; the center cannot hold
Mere anarchy is loosed upon the world

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: RiakCS poor s3 upload speeds 2MB/s

[Toby Corkindale]

On 21 January 2015 at 15:22, Luke Bakken lbak...@basho.com wrote:
 Hi Toby -

 Are you using the stock bitcask configuration for merging?

Hi Luke,
Yes, pretty much.

 On Tue, Jan 20, 2015 at 5:07 PM, Toby Corkindale t...@dryft.net wrote:
 Hi Kota,
 I had a bit of an off-list chat about this a while ago, plus continued
 to investigate locally, and eventually achieved some faster speeds,
 around 15MByte/sec writes.
 Things that were changed:
  * Adjusted Riak CS GC to be spread out over the cluster much more.
  * Tweaked up the put buffers and concurrency further
  * Moved most of the files out of CS and into Amazon S3+Glacier
  * Switched from nginx to haproxy
  * simplified firewalling for internal clients

 Each one of those changes made a small to modest improvement, but
 overall combined to make a quite noticeable improvement.

 I did notice something odd though -- despite moving most of the data
 out of the cluster, the disk-space-in-use by Riak is still very large
 compared to the amount stored. I mean, we moved more than 90% of the
 data out of the cluster, yet the actual disk space used only halved.
 For every gigabyte of file stored in CS, dozens of gigabytes are
 actually on disk!

 Either the garbage collection algorithm is very, very lazy, or somehow
 something has gone a bit wrong in the past, which might have explained
 part of the performance problems.

 We're going to look at redeploying a new, fresh cluster based on Riak
 2 in the not too distant future, once Riak CS looks like it's approved
 for use there, and maybe that'll clear all of this up.

 Toby



-- 
Turning and turning in the widening gyre
The falcon cannot hear the falconer
Things fall apart; the center cannot hold
Mere anarchy is loosed upon the world

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: RiakCS poor s3 upload speeds 2MB/s

[Toby Corkindale]

Hi Kota,
I had a bit of an off-list chat about this a while ago, plus continued
to investigate locally, and eventually achieved some faster speeds,
around 15MByte/sec writes.
Things that were changed:
 * Adjusted Riak CS GC to be spread out over the cluster much more.
 * Tweaked up the put buffers and concurrency further
 * Moved most of the files out of CS and into Amazon S3+Glacier
 * Switched from nginx to haproxy
 * simplified firewalling for internal clients

Each one of those changes made a small to modest improvement, but
overall combined to make a quite noticeable improvement.

I did notice something odd though -- despite moving most of the data
out of the cluster, the disk-space-in-use by Riak is still very large
compared to the amount stored. I mean, we moved more than 90% of the
data out of the cluster, yet the actual disk space used only halved.
For every gigabyte of file stored in CS, dozens of gigabytes are
actually on disk!

Either the garbage collection algorithm is very, very lazy, or somehow
something has gone a bit wrong in the past, which might have explained
part of the performance problems.

We're going to look at redeploying a new, fresh cluster based on Riak
2 in the not too distant future, once Riak CS looks like it's approved
for use there, and maybe that'll clear all of this up.

Toby

On 21 January 2015 at 11:07, Kota Uenishi k...@basho.com wrote:
 Toby and David,

 Thank you for working on Riak CS and I apologize for being late responder.

 I believe the reason of being slow down is different between Toby's
 and David's cluster.

 Toby's reason looks like that's just because of the data increasing.
 How much data per vnode do you have in your cluster, Toby? Do you have
 deletion in your workload?
 Riak CS's garbage collection, deleting block keys in Riak and merging
 Bitcask files make some load more than the exact amount of data
 visible via CS (even taking the replication factor into account).
 Also, if you turn on AAE, building AAE trees scans all the data stored
 in Riak to fix unexpected bit rot or partial replication. I'd like you
 to check the background load to underlying storage. If the performance
 decrease is *not* due to such background load, maybe there's the same
 dragon lurking under the water as David's cluster.

 One thing I can suggest from David's app.config, SSL is turned on -
 Riak CS uses Erlang's built-in SSL library for https scheme which is
 said to have not so good performance. I wonder those benchmarks were
 done over https or just http.

 As far as I test Riak CS, local or cluster-wide, I haven't met such
 bad performance less than 10MB/s in such a fresh state cluster. There
 should be something wrong, either the setup or the software. Would you
 mind sending us the result riak-debug and riak-cs-debug commands, if
 you still can reproduce such situation. Those packs up the environment
 info as much as it can.

 Thanks,
 Kota

 On Thu, Nov 27, 2014 at 10:36 AM, Toby Corkindale t...@dryft.net wrote:
 Thanks, that's interesting to hear.
 How have you been finding the stability and reliability to be with
 leofs, over time?


 I still wish I could just get our Riak CS cluster performing better;
 it just seems so unreasonably slow at the moment, that I suspect
 there's *something* holding it back. I can build a test cluster on my
 desktop, and even with five virtual riak nodes on the one machine, I
 still see 20-40x the performance, so it seems bizarre that dedicated
 bare-metal servers would be so slow. (Although obviously there's much
 more network latency between real machines, than a virtual cluster on
 one desktop; and they have a lot more data in their bitcask databases)


 However I've tried fiddling with all the Riak and Riak CS options..
 ethtool offloads.. mount options.. sysctls.. MTU sizes.. even dropping
 single nodes out of the cluster one at a time in case they were
 somehow at fault..  seems like the only performance changes I can make
 are negative.

 We're still double the speed of the original poster in this thread,
 but.. that isn't saying much.

 Toby


 On 26 November 2014 at 06:44, Heinz Nikolaus Gies he...@licenser.net wrote:
 If you’re evaluating RiakCS vs. Ceph you might want to toss LeoFS[1] in the
 mix and give it a run. Just as RiakCS it is a dynamo inspired system build
 in Erlang and comes with the same advantages and disadvantages. But unlike
 RiakCS it is pretty much exclusive a Object Store so can take a few
 different optimizations for this kind of work that might not be possible in
 a general purpose database as Riak (this is my personal guess not a research
 founded conclusion).  The team is (much) smaller then bash (obviously) but
 they’re a very nice and responsive bunch. I ended up using it as a s3
 backend for Project-FiFo due to it’s performance characteristics. With
 current releases I manage to get a sigle file upload speed of ~1.2GB/s using
 gof3r[2] (this might be a client limitation but I haven’t had time

Re: Using apache as a reverse proxy

[Toby Corkindale]

Just a thought, but have you tried to replicate the problem when using
haproxy (with the example configs provided on basho's site)?
That might let you discover if the problem really IS with Apache, or
if it's something to do with your Riak config instead.

On 15 January 2015 at 11:05, Fred Grim fg...@vmware.com wrote:
 Hello List,

 So I am using apache on ubuntu as a reverse proxy to a riak cluster. 
 Everything seems to work fine except for largish streaming map reduce 
 queries.  The config is:
 VirtualHost *:8098
 ServerAdmin aper...@acompany.com
 Proxy balancer://riak_cluster
 BalancerMember http://ip1:8098
 BalancerMember http://ip2:8098
 BalancerMember http://ip3:8098
 BalancerMember http://ip4:8098
 BalancerMember http://ip5:8098
 BalancerMember http://ip6:8098
 BalancerMember http://ip7:8098
 /Proxy
 Location /
 ProxyPass balancer://riak_cluster/
 ProxyPassReverse balancer://riak_cluster/
 /Location
 /VirtualHost

 What happens is that my streamed queries seem to partially come in which 
 makes me think that the load balancer is switching them or something. I know 
 I am missing some config value in here but I cannot, for the life of me, 
 figure out what it is. Does anyone have an example config they could through 
 up?

 F
 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



-- 
Turning and turning in the widening gyre
The falcon cannot hear the falconer
Things fall apart; the center cannot hold
Mere anarchy is loosed upon the world

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Rick CS MethodNotAllowed

[Toby Corkindale]

I'm going to take a punt that you're using the wrong domain name.
Eg. you've told riak to use, say, s3.internal as the base, but you're
actually sending queries to s3.example.com (or if you are using
default settings on some tools, s3.amazonaws.com)

If you're using a reverse proxy in front, check that you're passing
the Host: header back.

On 15 January 2015 at 06:42, Lee Sylvester lee.sylves...@gmail.com wrote:
 Hi guys,

 I’m pulling out hair, at the moment.  I have Riak, Riak CS and Stanchion 
 configured and running, but when I try to create a bucket with erlcloud, I 
 get an error with:

 {{aws_error,{http_error,405,[],
?xml version=\1.0\ 
 encoding=\UTF-8\?ErrorCodeMethodNotAllowed/CodeMessageThe 
 specified method is not allowed against this 
 resource./MessageResource/buckets/ResourceRequestId12345/RequestId/Error”}}

 Does anyone have any idea’s why this might be so?

 Thanks,
 Lee



 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



-- 
Turning and turning in the widening gyre
The falcon cannot hear the falconer
Things fall apart; the center cannot hold
Mere anarchy is loosed upon the world

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: RiakCS poor s3 upload speeds 2MB/s

[Toby Corkindale]

Thanks, that's interesting to hear.
How have you been finding the stability and reliability to be with
leofs, over time?


I still wish I could just get our Riak CS cluster performing better;
it just seems so unreasonably slow at the moment, that I suspect
there's *something* holding it back. I can build a test cluster on my
desktop, and even with five virtual riak nodes on the one machine, I
still see 20-40x the performance, so it seems bizarre that dedicated
bare-metal servers would be so slow. (Although obviously there's much
more network latency between real machines, than a virtual cluster on
one desktop; and they have a lot more data in their bitcask databases)


However I've tried fiddling with all the Riak and Riak CS options..
ethtool offloads.. mount options.. sysctls.. MTU sizes.. even dropping
single nodes out of the cluster one at a time in case they were
somehow at fault..  seems like the only performance changes I can make
are negative.

We're still double the speed of the original poster in this thread,
but.. that isn't saying much.

Toby


On 26 November 2014 at 06:44, Heinz Nikolaus Gies he...@licenser.net wrote:
 If you’re evaluating RiakCS vs. Ceph you might want to toss LeoFS[1] in the
 mix and give it a run. Just as RiakCS it is a dynamo inspired system build
 in Erlang and comes with the same advantages and disadvantages. But unlike
 RiakCS it is pretty much exclusive a Object Store so can take a few
 different optimizations for this kind of work that might not be possible in
 a general purpose database as Riak (this is my personal guess not a research
 founded conclusion).  The team is (much) smaller then bash (obviously) but
 they’re a very nice and responsive bunch. I ended up using it as a s3
 backend for Project-FiFo due to it’s performance characteristics. With
 current releases I manage to get a sigle file upload speed of ~1.2GB/s using
 gof3r[2] (this might be a client limitation but I haven’t had time to
 investigate the details).

 [1] http://leo-project.net/leofs/
 [2] https://github.com/rlmcpherson/s3gof3r/tree/master/gof3r
 ---
 Cheers,
 Heinz Nikolaus Gies
 he...@licenser.net



 On Nov 25, 2014, at 6:08, Toby Corkindale t...@dryft.net wrote:

 Hi,
 I wondered if you managed to significantly improve your Riak CS
 performance, or not?

 I just ask as we've been getting not-dissimilar performance out of
 Riak CS too (4-5 mbyte/sec max per client, on bare metal hardware),
 for quite a long time. (I swear it was faster originally, when there
 was a lot less data in the whole system.)
 This is after applying all the tweaks available -- networking stack,
 filesystem mount options, assorted Erlang vm.args, and increased put
 concurrency/buffer options.

 We put up with it because it's been just-about sufficient enough for
 our needs and Riak CS has been reliable and easy to administer -- but
 it's becoming more of an issue, and so I'm curious to know if other
 people *do* manage to achieve *good* per-client speeds out of Riak CS
 or if this is just how things always are?
 And we're way off the mark, maybe we can find out why..

 Details of our setup:
 6 node cluster. RIng size of 64.
 Riak 1.4.10
 Riak CS 1.5.2
 (installed from official Basho repos)

 Tests conducted using both multi-part and non-multi-part upload mode;
 performance is similar with both. Tested against cluster when very
 lightly loaded.
 For the sake of testing, a 100M file is being used, that contains
 random (hard to compress) data.

 Cheers,
 Toby

 On 8 November 2014 at 01:41, David Meekin david.mee...@autotrader.co.uk
 wrote:

 Hi,
 I’ve setup a test 4 node RiakCS cluster on HP BL460c hardware and I can’t
 seem to get S3 upload speeds above 2MB/s
 I’m connecting direct to RiackCS on one of the nodes so there is no load
 balancing software in place.
 I have also installed s3cmd locally onto one of the nodes and the speeds
 locally are the same.
 These 4 nodes also run a test CEPH cluster with RadosGW and s3 uploads to
 CEPH achieve 125MB/s
 Any help would be appreciated as I’m currently evaluating both CEPH and
 RiakCS.


 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





-- 
Turning and turning in the widening gyre
The falcon cannot hear the falconer
Things fall apart; the center cannot hold
Mere anarchy is loosed upon the world

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: RiakCS poor s3 upload speeds 2MB/s

[Toby Corkindale]

Hi,
I wondered if you managed to significantly improve your Riak CS
performance, or not?

I just ask as we've been getting not-dissimilar performance out of
Riak CS too (4-5 mbyte/sec max per client, on bare metal hardware),
for quite a long time. (I swear it was faster originally, when there
was a lot less data in the whole system.)
This is after applying all the tweaks available -- networking stack,
filesystem mount options, assorted Erlang vm.args, and increased put
concurrency/buffer options.

We put up with it because it's been just-about sufficient enough for
our needs and Riak CS has been reliable and easy to administer -- but
it's becoming more of an issue, and so I'm curious to know if other
people *do* manage to achieve *good* per-client speeds out of Riak CS
or if this is just how things always are?
And we're way off the mark, maybe we can find out why..

Details of our setup:
6 node cluster. RIng size of 64.
Riak 1.4.10
Riak CS 1.5.2
(installed from official Basho repos)

Tests conducted using both multi-part and non-multi-part upload mode;
performance is similar with both. Tested against cluster when very
lightly loaded.
For the sake of testing, a 100M file is being used, that contains
random (hard to compress) data.

Cheers,
Toby

On 8 November 2014 at 01:41, David Meekin david.mee...@autotrader.co.uk wrote:
 Hi,
 I’ve setup a test 4 node RiakCS cluster on HP BL460c hardware and I can’t 
 seem to get S3 upload speeds above 2MB/s
 I’m connecting direct to RiackCS on one of the nodes so there is no load 
 balancing software in place.
 I have also installed s3cmd locally onto one of the nodes and the speeds 
 locally are the same.
 These 4 nodes also run a test CEPH cluster with RadosGW and s3 uploads to 
 CEPH achieve 125MB/s
 Any help would be appreciated as I’m currently evaluating both CEPH and 
 RiakCS.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: RiakCS poor s3 upload speeds 2MB/s

[Toby Corkindale]

I found that adjusting these RiakCS settings improved write speeds a little
for us:

  {put_concurrency, 4},
  {put_buffer_factor, 8},

 As well as making sure the Riak vm.args included a high +zdbbl setting.

But even so, Riak CS has been really quite slow compared to other things
I've tried. However it has tended to work reasonably reliably and is easy
to maintain, so I went with it and it's been successful for us. I'd be
happier if write speeds were better though.


On 8 November 2014 01:41, David Meekin david.mee...@autotrader.co.uk
wrote:

  Hi,



 I’ve setup a test 4 node RiakCS cluster on HP BL460c hardware and I can’t
 seem to get S3 upload speeds above 2MB/s

 I’m connecting direct to RiackCS on one of the nodes so there is no load
 balancing software in place.

 I have also installed s3cmd locally onto one of the nodes and the speeds
 locally are the same.

 These 4 nodes also run a test CEPH cluster with RadosGW and s3 uploads to
 CEPH achieve 125MB/s

 Any help would be appreciated as I’m currently evaluating both CEPH and
 RiakCS.



 Configs below:



 %%%RIACK – app.config

 %% -*- mode: erlang;erlang-indent-level: 4;indent-tabs-mode: nil -*-

 %% ex: ft=erlang ts=4 sw=4 et

 [

 %% Riak Client APIs config

 {riak_api, [

 {pb_backlog, 256},

 {pb, [ {172.19.153.180, 8087 } ]}

 ]},



 %% Riak Core config

 {riak_core, [



   {default_bucket_props, [{allow_mult, true}]},



   {ring_state_dir, /var/lib/riak/ring},



   {http, [ {172.19.153.180, 8098 } ]},

   {https, [{ 172.19.153.180, 8096 }]},

   {ssl, [

  {certfile, /etc/riak/cert.pem},

  {keyfile, /etc/riak/key.pem}

 ]},



   {handoff_port, 8099 },



   {dtrace_support, false},



   {platform_bin_dir, /usr/sbin},

   {platform_data_dir, /var/lib/riak},

   {platform_etc_dir, /etc/riak},

   {platform_lib_dir, /usr/lib64/riak/lib},

   {platform_log_dir, /var/log/riak}

  ]},



 %% Riak KV config

 {riak_kv, [

 {add_paths, [/usr/lib64/riak-cs/lib/riak_cs-1.5.2/ebin]},

 {storage_backend, riak_cs_kv_multi_backend},

 {multi_backend_prefix_list, [{0b:, be_blocks}]},

 {multi_backend_default, be_default},

 {multi_backend, [

   {be_default, riak_kv_eleveldb_backend, [

   {max_open_files, 50},

   {data_root, /var/lib/riak/leveldb}

 ]},

 {be_blocks, riak_kv_bitcask_backend, [

   {data_root, /var/lib/riak/bitcask}

 ]}

 ]},





 {anti_entropy, {on, []}},



 {anti_entropy_build_limit, {1, 360}},

 {anti_entropy_expire, 60480},



 {anti_entropy_concurrency, 2},

 {anti_entropy_tick, 15000},



 {anti_entropy_data_dir, /var/lib/riak/anti_entropy},



 {anti_entropy_leveldb_opts, [{write_buffer_size, 4194304},

  {max_open_files, 20}]},



 {mapred_name, mapred},



 {mapred_2i_pipe, true},



 {map_js_vm_count, 8 },

 {reduce_js_vm_count, 6 },

 {hook_js_vm_count, 2 },



 {js_max_vm_mem, 8},



 {js_thread_stack, 16},



 {http_url_encoding, on},



 {vnode_vclocks, true},



 {listkeys_backpressure, true},



 {fsm_limit, 5},



 {object_format, v1}

]},



 %% Riak Search Config

 {riak_search, [

 {enabled, false}

]},



 %% Merge Index Config

 {merge_index, [

 {data_root, /var/lib/riak/merge_index},



 {buffer_rollover_size, 1048576},



 {max_compact_segments, 20}

]},



 %% Bitcask Config

 {bitcask, [

  {io_mode, erlang},



 {data_root, /var/lib/riak/bitcask}

]},



 %% eLevelDB Config

 {eleveldb, [

  {data_root, /var/lib/riak/leveldb}

 ]},



 %% Lager Config

 {lager, [

 {handlers, [

{lager_file_backend, [

{/var/log/riak/error.log, error,
 10485760, $D0, 5},

{/var/log/riak/console.log, info,
 10485760, $D0, 5}

]}

] },



 {crash_log, /var/log/riak/crash.log},



 {crash_log_msg_size, 65536},



 {crash_log_size, 10485760},



 {crash_log_date, $D0},



 {crash_log_count, 5},



 {error_logger_redirect, true},



 {error_logger_hwm, 100}

 ]},



 %% riak_sysmon config

 {riak_sysmon, [

 

Re: RiakCS poor s3 upload speeds 2MB/s

[Toby Corkindale]

I found that adjusting these RiakCS settings improved write speeds a
little for us:

  {put_concurrency, 4},
  {put_buffer_factor, 8},

 As well as making sure the Riak vm.args included a high +zdbbl setting.

But even so, Riak CS has been really quite slow compared to other
things I've tried. However it has tended to work reasonably reliably
and is easy to maintain, so I went with it and it's been successful
for us. I'd be happier if write speeds were better though.


On 8 November 2014 01:41, David Meekin david.mee...@autotrader.co.uk wrote:

 Hi,



 I’ve setup a test 4 node RiakCS cluster on HP BL460c hardware and I can’t 
 seem to get S3 upload speeds above 2MB/s

 I’m connecting direct to RiackCS on one of the nodes so there is no load 
 balancing software in place.

 I have also installed s3cmd locally onto one of the nodes and the speeds 
 locally are the same.

 These 4 nodes also run a test CEPH cluster with RadosGW and s3 uploads to 
 CEPH achieve 125MB/s

 Any help would be appreciated as I’m currently evaluating both CEPH and 
 RiakCS.



 Configs below:

 [snip]

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: [ANN] Riak CS 1.5.1

[Toby Corkindale]

My understanding was that Riak CS was still recommending that it is
run against Riak 1.4.10, not Riak 2.0?

On 21 October 2014 17:10, Seth Thomas stho...@opscode.com wrote:
 The repository builds are now being hosted by packagecloud.io. You can find
 install instructions here: https://packagecloud.io/basho/riak-cs/install.
 Currently only Riak 2.0+ and Riak CS 1.5.1 are hosted there. I've opened an
 issue[1] on the docs to get this updated for Riak CS

 [1] https://github.com/basho/basho_docs/issues/1394

 On Mon, Oct 20, 2014 at 10:40 PM, Toby Corkindale t...@dryft.net wrote:

 This release hasn't made it through to the Basho Ubuntu repositories
 yet; is that an oversight or by intent?

 for REL in precise trusty; do
   curl http://apt.basho.com/dists/$REL/main/binary-amd64/Packages |grep
 1.5.1
 done


 On 27 September 2014 14:56, Kota Uenishi k...@basho.com wrote:
  Hi all,
 
  Riak CS 1.5.1 is out. It's mostly a bugfix release, mainly for a GC
  worker stall bug. Another change newly introduced is the bucket number
  limitation. Users can create up to 100 buckets. See release notes [1]
  for detailed description, and get them from download page [2].
 
  [1]
  https://github.com/basho/riak_cs/blob/release/1.5/RELEASE-NOTES.md#riak-cs-151-release-notes
  [2] http://docs.basho.com/riakcs/latest/riakcs-downloads/
  --
  Kota UENISHI / @kuenishi
  Basho Japan KK
 
  ___
  riak-users mailing list
  riak-users@lists.basho.com
  http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



 --
 Turning and turning in the widening gyre
 The falcon cannot hear the falconer
 Things fall apart; the center cannot hold
 Mere anarchy is loosed upon the world

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





-- 
Turning and turning in the widening gyre
The falcon cannot hear the falconer
Things fall apart; the center cannot hold
Mere anarchy is loosed upon the world

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: [ANN] Riak CS 1.5.1

[Toby Corkindale]

Soo wouldn't it make sense for the repository containing Riak
1.4.x to be the same one that contains Riak CS 1.5.x, rather than Riak
2.x?
Otherwise I suspect it'll be a bit too easy to accidentally upgrade
riak from 1.4 to 2.0. (There are ways to prevent that with package
pinning, but maybe some users will forget)

Just a thought, anyway.

-Toby

On 22 October 2014 00:43, Shunichi Shinohara sh...@basho.com wrote:
 Yes. Riak CS 1.5.1 is tested with Riak 1.4.10.

 Shino

 On Tue, Oct 21, 2014 at 10:34 PM, Toby Corkindale t...@dryft.net wrote:
 My understanding was that Riak CS was still recommending that it is
 run against Riak 1.4.10, not Riak 2.0?

 On 21 October 2014 17:10, Seth Thomas stho...@opscode.com wrote:
 The repository builds are now being hosted by packagecloud.io. You can find
 install instructions here: https://packagecloud.io/basho/riak-cs/install.
 Currently only Riak 2.0+ and Riak CS 1.5.1 are hosted there. I've opened an
 issue[1] on the docs to get this updated for Riak CS

 [1] https://github.com/basho/basho_docs/issues/1394

 On Mon, Oct 20, 2014 at 10:40 PM, Toby Corkindale t...@dryft.net wrote:

 This release hasn't made it through to the Basho Ubuntu repositories
 yet; is that an oversight or by intent?

 for REL in precise trusty; do
   curl http://apt.basho.com/dists/$REL/main/binary-amd64/Packages |grep
 1.5.1
 done


 On 27 September 2014 14:56, Kota Uenishi k...@basho.com wrote:
  Hi all,
 
  Riak CS 1.5.1 is out. It's mostly a bugfix release, mainly for a GC
  worker stall bug. Another change newly introduced is the bucket number
  limitation. Users can create up to 100 buckets. See release notes [1]
  for detailed description, and get them from download page [2].
 
  [1]
  https://github.com/basho/riak_cs/blob/release/1.5/RELEASE-NOTES.md#riak-cs-151-release-notes
  [2] http://docs.basho.com/riakcs/latest/riakcs-downloads/
  --
  Kota UENISHI / @kuenishi
  Basho Japan KK
 
  ___
  riak-users mailing list
  riak-users@lists.basho.com
  http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



 --
 Turning and turning in the widening gyre
 The falcon cannot hear the falconer
 Things fall apart; the center cannot hold
 Mere anarchy is loosed upon the world

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





 --
 Turning and turning in the widening gyre
 The falcon cannot hear the falconer
 Things fall apart; the center cannot hold
 Mere anarchy is loosed upon the world

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



-- 
Turning and turning in the widening gyre
The falcon cannot hear the falconer
Things fall apart; the center cannot hold
Mere anarchy is loosed upon the world

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Brokenness specific to nginx and jets3t with Riak CS

[Toby Corkindale]

Hi,
I've hit an issue that only seems to occur when using the Jets3t
client library and Nginx as a load balancer in front of Riak CS.
The issue is NOT present when using other S3 libraries, nor is it
present if I switch out Nginx for haproxy.

Unfortunately, in this instance it is desirable to use both, unless it
turns out to be *really* troublesome.
I wondered if anyone here can offer advice?

The problem is that reads and writes to objects results in Riak CS,
apparently, disconnecting the client prematurely.
Nginx reports connection reset by peer and retries upstream Riak CS
servers several times before giving up.
The Riak CS access logs indicate what look like valid URLs, with a 403
status indicated.


Accessing the same buckets with the same auth and transferring the
same files is fine when done with s3cmd, and we have other apps that
have been running through nginx for a while using other S3 client
libraries.

A simple check if bucket exists command through jets3t seems to work OK.

Any thoughts on what could be going on, or is this all just a bit too vague?

Toby

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Brokenness specific to nginx and jets3t with Riak CS

[Toby Corkindale]

Quite update to mention that by removing some extra (unneeded) custom
headers from the proxy configuration and fiddling with some other
nginx options, I'm at a point where Riak CS throws an actual error
back rather than disconnecting prematurely.
The error now returned is below. I'm wondering if something else nginx
is doing is causing the request to get mangled enough that it doesn't
work.. but hard to tell what, or why :/

ResponseCode: 403, ResponseStatus: Forbidden, XML Error Message: ?xml
version=\1.0\
encoding=\UTF-8\?ErrorCodeAccessDenied/CodeMessageAccess
Denied/MessageResource/upload-service/da417126-8651-4be5-b552-6d125bb7b27c/coreos_production_ami_hvm.txt/ResourceRequestId/RequestId/Error


On 25 September 2014 15:59, Toby Corkindale t...@dryft.net wrote:
 Hi,
 I've hit an issue that only seems to occur when using the Jets3t
 client library and Nginx as a load balancer in front of Riak CS.
 The issue is NOT present when using other S3 libraries, nor is it
 present if I switch out Nginx for haproxy.

 Unfortunately, in this instance it is desirable to use both, unless it
 turns out to be *really* troublesome.
 I wondered if anyone here can offer advice?

 The problem is that reads and writes to objects results in Riak CS,
 apparently, disconnecting the client prematurely.
 Nginx reports connection reset by peer and retries upstream Riak CS
 servers several times before giving up.
 The Riak CS access logs indicate what look like valid URLs, with a 403
 status indicated.


 Accessing the same buckets with the same auth and transferring the
 same files is fine when done with s3cmd, and we have other apps that
 have been running through nginx for a while using other S3 client
 libraries.

 A simple check if bucket exists command through jets3t seems to work OK.

 Any thoughts on what could be going on, or is this all just a bit too vague?

 Toby



-- 
Turning and turning in the widening gyre
The falcon cannot hear the falconer
Things fall apart; the center cannot hold
Mere anarchy is loosed upon the world

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Brokenness specific to nginx and jets3t with Riak CS

[Toby Corkindale]

Following up further again, with a solution.

Turns out that Nginx canonicalises the URLs when proxying them, if and
only if, the proxy_pass parameter includes a path component, including
a plain slash.
Meanwhile, Jets3t URL-encodes the object path so that slashes become
%2F, but Nginx converts them to actual slashes before forwarding to
Riak CS.

Fix was to change
proxy_pass http://upstreamservers/;
to
proxy_pass http://upstreamservers;

-Toby

On 25 September 2014 16:24, Toby Corkindale t...@dryft.net wrote:
 Quite update to mention that by removing some extra (unneeded) custom
 headers from the proxy configuration and fiddling with some other
 nginx options, I'm at a point where Riak CS throws an actual error
 back rather than disconnecting prematurely.
 The error now returned is below. I'm wondering if something else nginx
 is doing is causing the request to get mangled enough that it doesn't
 work.. but hard to tell what, or why :/

 ResponseCode: 403, ResponseStatus: Forbidden, XML Error Message: ?xml
 version=\1.0\
 encoding=\UTF-8\?ErrorCodeAccessDenied/CodeMessageAccess
 Denied/MessageResource/upload-service/da417126-8651-4be5-b552-6d125bb7b27c/coreos_production_ami_hvm.txt/ResourceRequestId/RequestId/Error


 On 25 September 2014 15:59, Toby Corkindale t...@dryft.net wrote:
 Hi,
 I've hit an issue that only seems to occur when using the Jets3t
 client library and Nginx as a load balancer in front of Riak CS.
 The issue is NOT present when using other S3 libraries, nor is it
 present if I switch out Nginx for haproxy.

 Unfortunately, in this instance it is desirable to use both, unless it
 turns out to be *really* troublesome.
 I wondered if anyone here can offer advice?

 The problem is that reads and writes to objects results in Riak CS,
 apparently, disconnecting the client prematurely.
 Nginx reports connection reset by peer and retries upstream Riak CS
 servers several times before giving up.
 The Riak CS access logs indicate what look like valid URLs, with a 403
 status indicated.


 Accessing the same buckets with the same auth and transferring the
 same files is fine when done with s3cmd, and we have other apps that
 have been running through nginx for a while using other S3 client
 libraries.

 A simple check if bucket exists command through jets3t seems to work OK.

 Any thoughts on what could be going on, or is this all just a bit too vague?

 Toby



 --
 Turning and turning in the widening gyre
 The falcon cannot hear the falconer
 Things fall apart; the center cannot hold
 Mere anarchy is loosed upon the world



-- 
Turning and turning in the widening gyre
The falcon cannot hear the falconer
Things fall apart; the center cannot hold
Mere anarchy is loosed upon the world

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak CS: Undeletable broken buckets

[Toby Corkindale]

Hi Shunichi,
Enabling that temporarily then enabled me to remove the buckets by
first creating (s3cmd mb) and then removing (s3cmd rb) the problematic
buckets.

Thanks,
Toby

On 7 July 2014 19:38, Shunichi Shinohara sh...@basho.com wrote:
 Hi Toby,

 There is a rarely used option disable_local_bucket_check in Riak CS.
 I don't know it solves your case, please let me mention it.

 To use it, first set it in app.config of riak-cs (or
 application:set_env/3 in shell),
 {riak_cs, [...
{disable_local_bucket_check, true},
...]},
 Then create bucket as usual (e.g. s3cmd mb ...).

 These steps solves one of partial update patterns that Andrew mentioned.

 Thanks,
 Shino

 On Mon, Jul 7, 2014 at 2:12 PM, Toby Corkindale t...@dryft.net wrote:
 Hi Andrew,
 Thanks for the details.
 The Puppet config should never have let it be setup with
 allow_mult=false, but as this is a test cluster, it's possible
 something went awry there at some point.

 If it's not really a bug that needs reporting then I can let it go.
 Thanks,,
 Toby

 On 7 July 2014 12:21, Andrew Stone ast...@basho.com wrote:
 Hi Toby,

 We've seen this scenario before. It occurs because riak-cs stores bucket
 information in 2 places on disk:
   1) Inside the user record (for bucket permissions)
   2) Inside a global list of buckets, since each bucket must be unique

 What has happened most likely is that the bucket is no longer stored for the
 given user, but still in the global list of bucket. It shows up in bucket
 lists, but the current user doesn't have permission to actually do anything
 with it. Essentially you have partially written (or partially deleted) data.
 I believe the only time we saw this was when Riak was configured with
 {allow_mult, false} which is an invalid setting when used with riak-cs.
 Riak-cs uses siblings intelligently to merge conflicting data, and without
 that it's possible to end up in these types of scenarios. Later versions of
 riak-cs should refuse to run with {allow_mult, false}. I'd check your riak
 config to see if that is the case here.

 We actually have scripts to detect and remove the bad buckets that we've
 used in support. We can probably get you a copy if you want. Just let me
 know. And make sure when running in production that allow_mult = true.

 -Andrew



 On Sun, Jul 6, 2014 at 9:59 PM, Toby Corkindale t...@dryft.net wrote:

 Hi,
 At some point we've managed to create a couple of buckets that don't
 work and can't be deleted (in a development/testing cluster, not
 production).
 They show up with both 's3cmd ls' or by querying the HTTP API for a
 user's buckets.
 However attempting to list files in the bucket, or removing the
 bucket, or recreating the bucket, fails.

 It's not in a production cluster so it's not a huge concern to me, but
 thought I'd report the bug here in case it's of interest to you.
 Riak 1.4.9-1 and Riak-CS 1.4.5-1 on Ubuntu 12.04 LTS.

 $ s3cmd ls
 2014-02-07 00:07  s3://test5403
 2013-12-13 07:25  s3://test9857

 $ s3cmd ls s3://test5403
 ERROR: Bucket 'test5403' does not exist
 tobyc@adonai:~$ s3cmd ls s3://test9857
 ERROR: Bucket 'test9857' does not exist

 $ s3cmd rb s3://test5403
 ERROR: Bucket 'test5403' does not exist
 Bucket 's3://test5403/' removed

 $ s3cmd ls
 2014-02-07 00:07  s3://test5403
 2013-12-13 07:25  s3://test9857

 $ s3cmd mb s3://test5403
 Bucket 's3://test5403/' created

 $ s3cmd ls s3://test5403
 ERROR: Bucket 'test5403' does not exist

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





 --
 Turning and turning in the widening gyre
 The falcon cannot hear the falconer
 Things fall apart; the center cannot hold
 Mere anarchy is loosed upon the world

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



-- 
Turning and turning in the widening gyre
The falcon cannot hear the falconer
Things fall apart; the center cannot hold
Mere anarchy is loosed upon the world

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Riak CS: Undeletable broken buckets

[Toby Corkindale]

Hi,
At some point we've managed to create a couple of buckets that don't
work and can't be deleted (in a development/testing cluster, not
production).
They show up with both 's3cmd ls' or by querying the HTTP API for a
user's buckets.
However attempting to list files in the bucket, or removing the
bucket, or recreating the bucket, fails.

It's not in a production cluster so it's not a huge concern to me, but
thought I'd report the bug here in case it's of interest to you.
Riak 1.4.9-1 and Riak-CS 1.4.5-1 on Ubuntu 12.04 LTS.

$ s3cmd ls
2014-02-07 00:07  s3://test5403
2013-12-13 07:25  s3://test9857

$ s3cmd ls s3://test5403
ERROR: Bucket 'test5403' does not exist
tobyc@adonai:~$ s3cmd ls s3://test9857
ERROR: Bucket 'test9857' does not exist

$ s3cmd rb s3://test5403
ERROR: Bucket 'test5403' does not exist
Bucket 's3://test5403/' removed

$ s3cmd ls
2014-02-07 00:07  s3://test5403
2013-12-13 07:25  s3://test9857

$ s3cmd mb s3://test5403
Bucket 's3://test5403/' created

$ s3cmd ls s3://test5403
ERROR: Bucket 'test5403' does not exist

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak CS: Undeletable broken buckets

[Toby Corkindale]

Hi Andrew,
Thanks for the details.
The Puppet config should never have let it be setup with
allow_mult=false, but as this is a test cluster, it's possible
something went awry there at some point.

If it's not really a bug that needs reporting then I can let it go.
Thanks,,
Toby

On 7 July 2014 12:21, Andrew Stone ast...@basho.com wrote:
 Hi Toby,

 We've seen this scenario before. It occurs because riak-cs stores bucket
 information in 2 places on disk:
   1) Inside the user record (for bucket permissions)
   2) Inside a global list of buckets, since each bucket must be unique

 What has happened most likely is that the bucket is no longer stored for the
 given user, but still in the global list of bucket. It shows up in bucket
 lists, but the current user doesn't have permission to actually do anything
 with it. Essentially you have partially written (or partially deleted) data.
 I believe the only time we saw this was when Riak was configured with
 {allow_mult, false} which is an invalid setting when used with riak-cs.
 Riak-cs uses siblings intelligently to merge conflicting data, and without
 that it's possible to end up in these types of scenarios. Later versions of
 riak-cs should refuse to run with {allow_mult, false}. I'd check your riak
 config to see if that is the case here.

 We actually have scripts to detect and remove the bad buckets that we've
 used in support. We can probably get you a copy if you want. Just let me
 know. And make sure when running in production that allow_mult = true.

 -Andrew



 On Sun, Jul 6, 2014 at 9:59 PM, Toby Corkindale t...@dryft.net wrote:

 Hi,
 At some point we've managed to create a couple of buckets that don't
 work and can't be deleted (in a development/testing cluster, not
 production).
 They show up with both 's3cmd ls' or by querying the HTTP API for a
 user's buckets.
 However attempting to list files in the bucket, or removing the
 bucket, or recreating the bucket, fails.

 It's not in a production cluster so it's not a huge concern to me, but
 thought I'd report the bug here in case it's of interest to you.
 Riak 1.4.9-1 and Riak-CS 1.4.5-1 on Ubuntu 12.04 LTS.

 $ s3cmd ls
 2014-02-07 00:07  s3://test5403
 2013-12-13 07:25  s3://test9857

 $ s3cmd ls s3://test5403
 ERROR: Bucket 'test5403' does not exist
 tobyc@adonai:~$ s3cmd ls s3://test9857
 ERROR: Bucket 'test9857' does not exist

 $ s3cmd rb s3://test5403
 ERROR: Bucket 'test5403' does not exist
 Bucket 's3://test5403/' removed

 $ s3cmd ls
 2014-02-07 00:07  s3://test5403
 2013-12-13 07:25  s3://test9857

 $ s3cmd mb s3://test5403
 Bucket 's3://test5403/' created

 $ s3cmd ls s3://test5403
 ERROR: Bucket 'test5403' does not exist

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





-- 
Turning and turning in the widening gyre
The falcon cannot hear the falconer
Things fall apart; the center cannot hold
Mere anarchy is loosed upon the world

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Slow write performance for Riak CS

[Toby Corkindale]

On 4 July 2014 10:20, Matthew MacClary maccl...@gmail.com wrote:
 Hi all, a Riak CS user named Toby started this discussion about write
 performance. I am seeing the exact same behavior in terms of idle CPUs,
 network, and disks, but low throughput. Toby do you happen to have any
 follow up about settings to improve the raw Riak throughput and/or Riak CS
 throughput?

 http://lists.basho.com/pipermail/riak-users_lists.basho.com/2013-May/012177.html

Hi Matt,
I'm still running the settings I mentioned a year ago; I also setup
RiakCS to talk to a per-machine haproxy that proxies out to the Riak
CS port on all machines, rather than just talking to 127.0.0.01

Performance isn't great per client for putting data into the store,
but it's fast enough for our purposes, and does parallelize OK. (We're
a read-heavy workload and also writes are naturally split over a bunch
of clients writing smaller files, rather than fewer with large files)

Toby

 On 29/05/13 06:12, Reid Draper wrote:
 Hey Toby,

 Another option is to explore some of the Riak CS PUT configuration
 parameters, like the internal write concurrency and the buffer size. These
 can either be changed by editing the app.config, or changing them at
 run-time in an Erlang shell (via riak-cs attach). The two parameters are
 `put_concurrency` and `put_buffer_factor`.

 They both default to 1. The `put_concurrency` controls the number of
 threads inside of Riak CS that are used to write blocks to Riak. The
 `put_buffer_factor` controls the number of blocks that will be buffered
 in-memory in Riak CS before it starts to slow down reading from the HTTP
 client. I suggesting trying to raise these values to get higher
 single-client throughput. If you wish to edit the app.config, add lines like
 in the `riak_cs` section:

 {put_concurrency, 8},
 {put_buffer_factor, 16},


 Ah, thanks -- that's interesting. Bumping up those up a bit did improve
 things a bit -- I went for a conservative concurrency of 4,
 buffer_factor of 8, and that took speeds on 500M files from 8-9mb/s to
 12-13mb/sec.

 Might play with other values when I next get time, but for now that's a
 good cheap win.

 In increasing these values, you might also find it useful to run a load
 balancer between the Riak CS and Riak nodes, instead of having Riak CS just
 communicate with the local Riak node. We intend to have this behavior built
 into Riak CS in the future, but for now a load balancer will suffice.

 Ah, ta. Can do.

 Furthermore, please make sure you've looked through the Linux Tuning page
 for Riak [1]. And as for +zdbbl, you can try even higher, like 128MB: +zdbbl
 131072.

 Thanks.
 I'd been through there already, and applied things like filesystem and
 scheduler options.
 I haven't touched the networking sysctls since they came with a warning
 that they should only be messed with if networking *was* the bottleneck.

 Given what I've mentioned so far, do you think it's likely?
 I guess it can't hurt to adjust them and see..

 [1]
 http://docs.basho.com/riak/1.3.1/cookbooks/Linux-Performance-Tuning/#Linux-Tuning


 Thanks for your advice,
 Toby




 Reid

 On May 27, 2013, at 11:50 PM, Toby Corkindale toby.corkindale at
 strategicdata.com.au wrote:

 On 28/05/13 13:11, Jared Morrow wrote:
 Toby,

 Can you trying putting data with a second client simultaneously?  When
 people have slow benchmarking, lots of times just using multiple
 worker/clients helps.  Also, what client library are you using?

 Running up three S3 clients (on separate machines) simultaneously saw
 them return 8, 8, 6 MB/sec. Interesting to note that the performance hasn't
 dropped threefold, but still, it'd be really nice if an individual transfer
 would run faster, given the performance of the underlying hardware.

 The nodes hardly get utilised during a write operation. I've uploading a
 total of 1500M from the three nodes, and yet CPUs are 90% idle, and there's
 no real disk activity going on, apart from a couple of times when several
 hundred get flushed out over the course of a second.

 I feel like something isn't quite right. What is the system *waiting*
 for? There's plenty of CPU and IO to go around.


 In the case of the current benchmarking, I'm using either s3cmd or curl.



 Also I meant to mention in my first reply, but Boundary
 http://boundary.com/ worked wonders for us being able to see how much
 data was really moving around.  They have a free trial as far as I know.
   It might be worth it to see if there are any obvious bottlenecks.

 Thanks, I'll have a look and see if the effort of setting it all up looks
 worthwhile.

 Cheers,
 Toby


 On Mon, May 27, 2013 at 8:46 PM, Toby Corkindale
 toby.corkindale at strategicdata.com.au
 mailto:toby.corkindale at strategicdata.com.au wrote:

 On 28/05/13 01:41, Jared Morrow wrote:

 Toby,

 If you write with multiple clients does it still stick to 9mb/s
 or
 does it increase?  What is the network link between your client

Re: CS: Query path for storage/access statistics appears wrong?

[Toby Corkindale]

Hi Kelly,
I've submitted this as issue:
https://github.com/basho/riak_cs/issues/909
Thanks,
Toby

On 3 July 2014 01:40, Kelly McLaughlin ke...@basho.com wrote:
 Toby,

 I checked on this and you are correct that it is a bug. The query parameters
 for the usage API requests are omitted during the URL processing step. This
 leads to the not_requested return you are seeing because when the usage
 request is processed after the URL processing step the indicators that any
 data was requested are no longer there.

 Would you mind filing an issue on the riak_cs github repo for this?

 Thanks,

 Kelly

 On July 1, 2014 at 11:25:00 PM, Toby Corkindale (t...@dryft.net) wrote:

 e=20140602T00Zs=20140528T00Z



-- 
Turning and turning in the widening gyre
The falcon cannot hear the falconer
Things fall apart; the center cannot hold
Mere anarchy is loosed upon the world

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: CS: Query path for storage/access statistics appears wrong?

[Toby Corkindale]

On 3 July 2014 01:57, Hector Castro hec...@basho.com wrote:
 Hey Toby,

 Are you authenticating the request that uses query string parameters?

Hi Hector,
In both cases I am authenticating the request, using the AWS header method.
I noted that I needed to only do the auth hash based on the path, not
the query parameters, in order for it to succeed though. I just
assumed that was intentional as the AWS spec does indicate that too.
Toby


 On Wed, Jul 2, 2014 at 1:23 AM, Toby Corkindale t...@dryft.net wrote:
 Hi,
 Maybe I'm doing something wrong, but I find I can only access the Riak
 CS storage/access information per-user by using the s3-compatible
 path, not the REST API path -- even though I am connecting to the REST
 API port.

 ie. To get the statistics, I am querying
 http://SERVERNAME:8080/riak-cs/usage/USER-ID/b/STIME/ETIME
 An actual URI generated this way would be:
 /riak-cs/usage/ZAZ6_VYSIE7QMYQRRHY3/b/20140528T00Z/20140602T00Z


 If I try to use the proper API method, I consistently receive
 not_requested as the results.
 The API URL is:
 http://SERVERNAME:8080/riak-cs/usage/USER-ID?be=ETIMEs=STIME
 An actual URI generated this way would be:
 /riak-cs/usage/JEJ-XPZBSIKSU_Z4EGPC?be=20140602T00Zs=20140528T00Z

 Is this a known bug or should this be working (and I'm just doing it wrong?).
 Current latest stable Riak - 1.4.9-1 from Basho Ubuntu repo.


 Cheers,
 Toby

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



-- 
Turning and turning in the widening gyre
The falcon cannot hear the falconer
Things fall apart; the center cannot hold
Mere anarchy is loosed upon the world

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

CS: Query path for storage/access statistics appears wrong?

[Toby Corkindale]

Hi,
Maybe I'm doing something wrong, but I find I can only access the Riak
CS storage/access information per-user by using the s3-compatible
path, not the REST API path -- even though I am connecting to the REST
API port.

ie. To get the statistics, I am querying
http://SERVERNAME:8080/riak-cs/usage/USER-ID/b/STIME/ETIME
An actual URI generated this way would be:
/riak-cs/usage/ZAZ6_VYSIE7QMYQRRHY3/b/20140528T00Z/20140602T00Z


If I try to use the proper API method, I consistently receive
not_requested as the results.
The API URL is:
http://SERVERNAME:8080/riak-cs/usage/USER-ID?be=ETIMEs=STIME
An actual URI generated this way would be:
/riak-cs/usage/JEJ-XPZBSIKSU_Z4EGPC?be=20140602T00Zs=20140528T00Z

Is this a known bug or should this be working (and I'm just doing it wrong?).
Current latest stable Riak - 1.4.9-1 from Basho Ubuntu repo.


Cheers,
Toby

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Understanding an intermittent mapreduce error

[Toby Corkindale]

Hi,
I'm trying to understand why I'm getting an intermittent error on a
particular map-reduce call to Riak 1.4.9.
The query isn't particularly complicated, and filters mean it should
only be running over ~100 keys, and returning 10 each time, so I
don't think I should be hitting any stack or memory limits in the JS
VMs.

It fails frequently, but not every time. When pointing it at just a
single Riak server for debugging purposes, it fails around 30% of the
time.
The error returned is as follows:

{
   input : null,
   stack : null,
   phase : 1,
   error : 
{undef,[{riak_kv_mapred_json,jsonify_not_found,[{struct,[{\email\,\u...@example.com\},{\attributes\,[\someattributes\,\somemoreattributes\,\w...\,...]},...]}],...},...]},
   type : null
}


There are no errors in the Riak server logs.


Thanks,
Toby

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Understanding an intermittent mapreduce error

[Toby Corkindale]

Hi Sean,
Thanks for your response -- sorry about the delay. I'm in a far away timezone.
The response I get is:
Eshell V5.9.1  (abort with ^G)
(riak@riak01.internal)1 rr(rpc:multicall(riak_kv_mapred_json,
module_info, [exports])).
{error,invalid_filename}
(riak@riak01.internal)2

-Toby

On 16 June 2014 23:22, Sean Cribbs s...@basho.com wrote:
 Toby,

 Could you try this?

 1. Run `riak attach`
 2. Type into the console: rr(rpc:multicall(riak_kv_mapred_json, module_info,
 [exports])).
 3. Paste back to me what the result is

 Cheers,


 On Mon, Jun 16, 2014 at 3:31 AM, Toby Corkindale t...@dryft.net wrote:

 Hi,
 I'm trying to understand why I'm getting an intermittent error on a
 particular map-reduce call to Riak 1.4.9.
 The query isn't particularly complicated, and filters mean it should
 only be running over ~100 keys, and returning 10 each time, so I
 don't think I should be hitting any stack or memory limits in the JS
 VMs.

 It fails frequently, but not every time. When pointing it at just a
 single Riak server for debugging purposes, it fails around 30% of the
 time.
 The error returned is as follows:

 {
input : null,
stack : null,
phase : 1,
error :
 {undef,[{riak_kv_mapred_json,jsonify_not_found,[{struct,[{\email\,\u...@example.com\},{\attributes\,[\someattributes\,\somemoreattributes\,\w...\,...]},...]}],...},...]},
type : null
 }


 There are no errors in the Riak server logs.


 Thanks,
 Toby

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




 --
 Sean Cribbs s...@basho.com
 Software Engineer
 Basho Technologies, Inc.
 http://basho.com/



-- 
Turning and turning in the widening gyre
The falcon cannot hear the falconer
Things fall apart; the center cannot hold
Mere anarchy is loosed upon the world

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

riak-admin backup - Backup documentation out of date?

[Toby Corkindale]

Hi,
The documentation here, apparently the latest, doesn't appear to
mention the riak-admin backup command at all, nor is it mentioned on
the command-line tools page for riak-admin.
http://docs.basho.com/riak/latest/ops/running/backups/
http://docs.basho.com/riak/1.4.9/ops/running/tools/riak/

Is the command suitable for use yet? It'll simplify backups greatly if so.

Cheers,
Toby

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Understanding an intermittent mapreduce error

[Toby Corkindale]

 rp(rpc:multicall(riak_kv_mapred_json, module_info, [exports])).
{[[{parse_inputs,1},
   {parse_query,1},
   {dejsonify_not_found,1},
   {module_info,0},
   {module_info,1},
   {jsonify_not_found,1},
   {jsonify_bkeys,2},
   {parse_request,1},
   {jsonify_pipe_error,2}],
  {badrpc,{'EXIT',{undef,[{riak_kv_mapred_json,module_info,
   [exports],
   []},
  {rpc,'-handle_call_call/6-fun-0-',5,
   [{file,rpc.erl},{line,203}]}]}}},
  {badrpc,{'EXIT',{undef,[{riak_kv_mapred_json,module_info,
   [exports],
   []},
  {rpc,'-handle_call_call/6-fun-0-',5,
   [{file,rpc.erl},{line,203}]}]}}},
  [{parse_inputs,1},
   {parse_query,1},
   {dejsonify_not_found,1},
   {module_info,0},
   {module_info,1},
   {jsonify_not_found,1},
   {jsonify_bkeys,2},
   {parse_request,1},
   {jsonify_pipe_error,2}],
  [{parse_inputs,1},
   {parse_query,1},
   {dejsonify_not_found,1},
   {module_info,0},
   {module_info,1},
   {jsonify_not_found,1},
   {jsonify_bkeys,2},
   {parse_request,1},
   {jsonify_pipe_error,2}],
  [{parse_inputs,1},
   {parse_query,1},
   {module_info,0},
   {module_info,1},
   {jsonify_not_found,1},
   {dejsonify_not_found,1},
   {jsonify_bkeys,2},
   {parse_request,1},
   {jsonify_pipe_error,2}]],
 []}
ok


On 17 June 2014 11:19, Sean Cribbs s...@basho.com wrote:
 Sorry for the typo, replace 'rr' with 'rp' on the front of that.

 Sean Cribbs

 On Jun 16, 2014, at 7:59 PM, Toby Corkindale t...@dryft.net wrote:

 Hi Sean,
 Thanks for your response -- sorry about the delay. I'm in a far away 
 timezone.
 The response I get is:
 Eshell V5.9.1  (abort with ^G)
 (riak@riak01.internal)1 rr(rpc:multicall(riak_kv_mapred_json,
 module_info, [exports])).
 {error,invalid_filename}
 (riak@riak01.internal)2

 -Toby

 On 16 June 2014 23:22, Sean Cribbs s...@basho.com wrote:
 Toby,

 Could you try this?

 1. Run `riak attach`
 2. Type into the console: rr(rpc:multicall(riak_kv_mapred_json, module_info,
 [exports])).
 3. Paste back to me what the result is

 Cheers,


 On Mon, Jun 16, 2014 at 3:31 AM, Toby Corkindale t...@dryft.net wrote:

 Hi,
 I'm trying to understand why I'm getting an intermittent error on a
 particular map-reduce call to Riak 1.4.9.
 The query isn't particularly complicated, and filters mean it should
 only be running over ~100 keys, and returning 10 each time, so I
 don't think I should be hitting any stack or memory limits in the JS
 VMs.

 It fails frequently, but not every time. When pointing it at just a
 single Riak server for debugging purposes, it fails around 30% of the
 time.
 The error returned is as follows:

 {
   input : null,
   stack : null,
   phase : 1,
   error :
 {undef,[{riak_kv_mapred_json,jsonify_not_found,[{struct,[{\email\,\u...@example.com\},{\attributes\,[\someattributes\,\somemoreattributes\,\w...\,...]},...]}],...},...]},
   type : null
 }


 There are no errors in the Riak server logs.


 Thanks,
 Toby

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




 --
 Sean Cribbs s...@basho.com
 Software Engineer
 Basho Technologies, Inc.
 http://basho.com/



 --
 Turning and turning in the widening gyre
 The falcon cannot hear the falconer
 Things fall apart; the center cannot hold
 Mere anarchy is loosed upon the world



-- 
Turning and turning in the widening gyre
The falcon cannot hear the falconer
Things fall apart; the center cannot hold
Mere anarchy is loosed upon the world

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Cost of many PBC connections held open?

[Toby Corkindale]

Hi,
Just looking for a bit of guidance on good practice when using Riak
via the PBC interface.

To keep request latency low, it's good to keep the connection open
between requests, rather than making a new connection for just a few
requests and then dropping it again, right?

In the JVM client, pools of connections are shared between large
numbers of threads. However it's harder to do that in Perl, which some
of our codebase is written in.
It'd be a lot easier to have one connection per process, but that's
potentially quite a lot of connections, albeit ones that are idle most
of the time.

I'd like to get a feeling for how expensive is it for the Erlang VM to
hold those connections open. Does it consume a lot of resources or is
it negligible?

Thanks,
Toby

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Riak CS - POST upload support

[Toby Corkindale]

Hi,
I wondered if Basho is intending to add support for user POST uploads
in Riak CS at any time soon?
(ie. As per 
http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-UsingHTTPPOST.html
)

Cheers,
Toby

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Cleaning up bucket after basho_bench run

[Toby Corkindale]

Hi Istvan,
I posted about this same problem to the list quite a while ago.
The problem is that the Erlang API allows arbitrary binary sequences,
whereas the PBC (via Java) and HTTP APIs only handle valid Unicode
strings.
So basho_bench creates keys that can't be manipulated by anything
except erlang :/
You can delete them if you write some erlang though..
-Toby

On 15 March 2014 04:00, István lecc...@gmail.com wrote:
 Hi,

 I am trying to clean up some of the test data that was inserted by
 basho_bench. The first approach to use curl and streaming the keys
 fails like this:

 # curl -XGET -i http://127.0.0.1:8098/buckets/test/keys?keys=stream
 HTTP/1.1 200 OK
 Vary: Accept-Encoding
 Transfer-Encoding: chunked
 Server: MochiWeb/1.1 WebMachine/1.10.0 (never breaks eye contact)
 Date: Fri, 14 Mar 2014 16:59:08 GMT
 Content-Type: application/json

 curl: (18) transfer closed with outstanding read data remaining

 When I am trying to the same thing with MapReduce it fails like this:

 curl -X POST http://localhost:8098/mapred; -H Content-Type:
 application/json -d '{
 inputs: test,
 query: [
 {
 map: {
 language: javascript,
 source: function(riakObject) { return [riakObject.key]; }
 }
 }
 ]
 }'

 Error:

 {phase:0,error:bad_utf8_character_code,input:{ok,{r_object,\test\,0,116,71,0,[{r_content,{dict,3,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[[\X-Riak-VTag\,71,81,80,81,87,76,105,54,113,120,97,116,114,106,51,86,72,53,67,50,82]],[[\index\]],[],[[\X-Riak-Last-Modified\|{1391,27501,255280}]],[],[]}}},75,191,51,171,193,113,206,163,24,68,247,188,84,72,5,72,179,195,99,44,202,122,136,31,250,94,166,5,160,199,182,137,40,6,253,115,100,4,34,67,64,10,25,210,58,23,104,97,228,...}],...},...}}

 I am wondering how else could I just get a list of keys in that
 bucket. The ultimate goal is to be able to delete them all.

 Thank you in advance,
 Istvan

 --
 the sun shines for all

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



-- 
Turning and turning in the widening gyre
The falcon cannot hear the falconer
Things fall apart; the center cannot hold
Mere anarchy is loosed upon the world

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: [ANN] Riak 1.4.8

[Toby Corkindale]

On 27 February 2014 02:04, John Caprice jcapr...@basho.com wrote:
 Toby,

 Were you running with AAE disabled until 1.4.8 was released?  If so, AAE is
 likely repairing objects that needed to be read repaired during that time.

It was still running for a fair while; the instructions to disable AAE
only came through fairly late in the game.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: [ANN] Riak 1.4.8

[Toby Corkindale]

Hi Engel,
Ah, thanks, I'll do that now.

Cheers,
Toby

On 27 February 2014 04:59, Engel Sanchez en...@basho.com wrote:
 Toby,

 Since data inserted into the AAE trees while versions 1.4.4 - 1.4.7 of Riak
 were running was incorrect, they should be discarded before the upgrade.
 You are seeing them valiantly trying to repair themselves based on the
 incoming writes, but that activity is wasted since they require a full scan
 of the data to be useful again.  We are going to add a paragraph to the
 release notes to warn others about this and make sure the anti_entropy
 directories are cleared before Riak is updated to 1.4.8 and AAE is enabled
 again.  We recommend you disable AAE, remove all AAE trees (the contents of
 the anti_entropy directory) and enable it again during a period of low
 cluster activity.  They will all be built again and you should be back in
 business once that the new trees are ready.

 Engel


 On Tue, Feb 25, 2014 at 11:02 PM, Toby Corkindale t...@dryft.net wrote:

 Hi,
 After upgrading to 1.4.8 in staging, last week, we've been seeing a
 fairly constant extra amount of riak hits in the stats. Consistently
 300-400 a minute. Can get lost in the much higher, varying levels
 during busy hours, but outside of that it's quite visible as a
 baseline.
 The logs show a lot of AAE work going on.

 I wondered if this was normal behaviour of Riak cleaning up the
 AAE-related bug from earlier versions, and will go away eventually? Or
 that unlikely?
 I emphasize that there's no significant server load resulting from
 this -- I'm just curious if it's something I need to look into or not.

 Cheers,
 Toby


 On 21 February 2014 04:00, Tom Santero tsant...@basho.com wrote:
  Hi,
 
  Today, Basho released a minor update of Riak and Riak Enterprise
  Edition,
  version 1.4.8. This bug-fix release addresses a regression[0] in Riak's
  active anti-entropy (AAE) system. The regression existed in versions
  1.4.3
  through 1.4.7.
 
  The complete release notes[1] and package downloads[2] are now
  available.
 
  Thanks for being the best community ever.
 
  Regards,
  The Basho Team
 
  [0]
 
  http://lists.basho.com/pipermail/riak-users_lists.basho.com/2014-January/014551.html
  [1] https://github.com/basho/riak/blob/1.4/RELEASE-NOTES.md
  [2] http://docs.basho.com/riak/latest/downloads/
 
 
  ___
  riak-users mailing list
  riak-users@lists.basho.com
  http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
 



 --
 Turning and turning in the widening gyre
 The falcon cannot hear the falconer
 Things fall apart; the center cannot hold
 Mere anarchy is loosed upon the world

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





-- 
Turning and turning in the widening gyre
The falcon cannot hear the falconer
Things fall apart; the center cannot hold
Mere anarchy is loosed upon the world

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: [ANN] Riak 1.4.8

[Toby Corkindale]

Hi,
After upgrading to 1.4.8 in staging, last week, we've been seeing a
fairly constant extra amount of riak hits in the stats. Consistently
300-400 a minute. Can get lost in the much higher, varying levels
during busy hours, but outside of that it's quite visible as a
baseline.
The logs show a lot of AAE work going on.

I wondered if this was normal behaviour of Riak cleaning up the
AAE-related bug from earlier versions, and will go away eventually? Or
that unlikely?
I emphasize that there's no significant server load resulting from
this -- I'm just curious if it's something I need to look into or not.

Cheers,
Toby


On 21 February 2014 04:00, Tom Santero tsant...@basho.com wrote:
 Hi,

 Today, Basho released a minor update of Riak and Riak Enterprise Edition,
 version 1.4.8. This bug-fix release addresses a regression[0] in Riak's
 active anti-entropy (AAE) system. The regression existed in versions 1.4.3
 through 1.4.7.

 The complete release notes[1] and package downloads[2] are now available.

 Thanks for being the best community ever.

 Regards,
 The Basho Team

 [0]
 http://lists.basho.com/pipermail/riak-users_lists.basho.com/2014-January/014551.html
 [1] https://github.com/basho/riak/blob/1.4/RELEASE-NOTES.md
 [2] http://docs.basho.com/riak/latest/downloads/


 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




-- 
Turning and turning in the widening gyre
The falcon cannot hear the falconer
Things fall apart; the center cannot hold
Mere anarchy is loosed upon the world

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: RIAK 1.4.6 - Mass key deletion

[Toby Corkindale]

On 19 February 2014 03:18, Matthew Von-Maszewski matth...@basho.com wrote:
 Riak 2.0 is coming.  Hold your mass delete until then.  The bug is within
 Google's original leveldb architecture.  Riak 2.0 sneaks around to get the
 disk space freed.

I'm interested to know what happens if someone deletes a lot of data
on Riak 1.4.x, and then later upgrades to Riak 2.0.
Will the missing space be recovered at that point?

Toby

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Riak crashing with {error,{badmatch,{error,eexist}}} in bitcask:do_put/5 line 1232

[Toby Corkindale]

Our test environment Riak cluster is looking pretty unhealthy at the
moment, with quite a lot of crashes being reported. Is anyone able to
advise what the cause is?

From error.log:

[error] 0.13413.31 CRASH REPORT Process 0.13413.31 with 0
neighbours exited with reason: no match of right hand value
{error,{badmatch,{error,eexist}}} in bitcask:do_put/5 line 1232 in
gen_server:terminate/6 line 747


From crash.log:

=ERROR REPORT
** Generic server 0.5600.32 terminating
** Last message in was
{'EXIT',0.5599.32,{{badmatch,{error,{badmatch,{error,eexist,[{bitcask,do_put,5,[{file,src/bitcask.erl},{line,1232}]},{bitcask,put,3,[{file,src/bitcask.erl},{line,244}]},{riak_kv_bitcask_backend,put,5,[{file,src/riak_kv_bitcask_backend.erl},{line,168}]},{riak_cs_kv_multi_backend,put,5,[{file,src/riak_cs_kv_multi_backend.erl},{line,255}]},{riak_kv_vnode,encode_and_put,6,[{file,src/riak_kv_vnode.erl},{line,1776}]},{riak_kv_vnode,perform_put,3,[{file,src/riak_kv_vnode.erl},{line,1162}]},{riak_kv_vnode,do_put,7,[{file,src/riak_kv_vnode.erl},{line,1009}]},{riak_kv_vnode,handle_command,3,[{file,src/riak_kv_vnode.erl},{line,419}]}]}}
** When Server state ==
{state,{0.5600.32,poolboy_sup},simple_one_for_one,[{child,undefined,riak_core_vnode_worker,{riak_core_vnode_worker,start_link,[[{worker_module,riak_core_vnode_worker},{worker_args,[91343852333181432387730302044767688728495783936,[],worker_props,0.5598.32]},{worker_callback_mod,riak_kv_worker},{size,10},{max_overflow,0}]]},temporary,5000,worker,[riak_core_vnode_worker]}],{set,10,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[0.5603.32],[0.5604.32],[0.5605.32],[0.5606.32],[0.5607.32],[0.5608.32],[0.5609.32],[0.5610.32],[],[],[],[],[],[],[0.5601.32],[0.5602.32]}}},0,1,[],poolboy_sup,{riak_core_vnode_worker,[{worker_module,riak_core_vnode_worker},{worker_args,[91343852333181432387730302044767688728495783936,[],worker_props,0.5598.32]},{worker_callback_mod,riak_kv_worker},{size,10},{max_overflow,0}]}}
** Reason for termination ==
** 
{{badmatch,{error,{badmatch,{error,eexist,[{bitcask,do_put,5,[{file,src/bitcask.erl},{line,1232}]},{bitcask,put,3,[{file,src/bitcask.erl},{line,244}]},{riak_kv_bitcask_backend,put,5,[{file,src/riak_kv_bitcask_backend.erl},{line,168}]},{riak_cs_kv_multi_backend,put,5,[{file,src/riak_cs_kv_multi_backend.erl},{line,255}]},{riak_kv_vnode,encode_and_put,6,[{file,src/riak_kv_vnode.erl},{line,1776}]},{riak_kv_vnode,perform_put,3,[{file,src/riak_kv_vnode.erl},{line,1162}]},{riak_kv_vnode,do_put,7,[{file,src/riak_kv_vnode.erl},{line,1009}]},{riak_kv_vnode,handle_command,3,[{file,src/riak_kv_vnode.erl},{line,419}]}]}
=CRASH REPORT
  crasher:
initial call: supervisor:poolboy_sup/1
pid: 0.5600.32
registered_name: []
exception exit:
{{{badmatch,{error,{badmatch,{error,eexist,[{bitcask,do_put,5,[{file,src/bitcask.erl},{line,1232}]},{bitcask,put,3,[{file,src/bitcask.erl},{line,244}]},{riak_kv_bitcask_backend,put,5,[{file,src/riak_kv_bitcask_backend.erl},{line,168}]},{riak_cs_kv_multi_backend,put,5,[{file,src/riak_cs_kv_multi_backend.erl},{line,255}]},{riak_kv_vnode,encode_and_put,6,[{file,src/riak_kv_vnode.erl},{line,1776}]},{riak_kv_vnode,perform_put,3,[{file,src/riak_kv_vnode.erl},{line,1162}]},{riak_kv_vnode,do_put,7,[{file,src/riak_kv_vnode.erl},{line,1009}]},{riak_kv_vnode,handle_command,3,[{file,src/riak_kv_vnode.erl},{line,419}]}]},[{gen_server,terminate,6,[{file,gen_server.erl},{line,747}]},{proc_lib,init_p_do_apply,3,[{file,proc_lib.erl},{line,227}]}]}
ancestors: 
[0.5599.32,0.5598.32,0.5479.32,riak_core_vnode_sup,riak_core_sup,0.154.0]
messages: []
links: []
dictionary: []
trap_exit: true
status: running
heap_size: 6765
stack_size: 24
reductions: 1945
  neighbours:
=ERROR REPORT
** Generic server 0.11652.32 terminating
** Last message in was
{'DOWN',#Ref0.0.36.156577,process,0.5479.32,{{badmatch,{error,{badmatch,{error,eexist,[{bitcask,do_put,5,[{file,src/bitcask.erl},{line,1232}]},{bitcask,put,3,[{file,src/bitcask.erl},{line,244}]},{riak_kv_bitcask_backend,put,5,[{file,src/riak_kv_bitcask_backend.erl},{line,168}]},{riak_cs_kv_multi_backend,put,5,[{file,src/riak_cs_kv_multi_backend.erl},{line,255}]},{riak_kv_vnode,encode_and_put,6,[{file,src/riak_kv_vnode.erl},{line,1776}]},{riak_kv_vnode,perform_put,3,[{file,src/riak_kv_vnode.erl},{line,1162}]},{riak_kv_vnode,do_put,7,[{file,src/riak_kv_vnode.erl},{line,1009}]},{riak_kv_vnode,handle_command,3,[{file,src/riak_kv_vnode.erl},{line,419}]}]}}
** When Server state == {state,undefined,undefined}
** Reason for termination ==
** 
{{badmatch,{error,badarg}},[{bitcask_file,handle_info,2,[{file,src/bitcask_file.erl},{line,170}]},{gen_server,handle_msg,5,[{file,gen_server.erl},{line,607}]},{proc_lib,init_p_do_apply,3,[{file,proc_lib.erl},{line,227}]}]}
2014-02-07 11:12:41 =CRASH REPORT
  crasher:
initial call: bitcask_file:init/1
pid: 0.11652.32
registered_name: []
exception exit:

Re: Changing app.config of Live Riak Cluster

[Toby Corkindale]

On 06/01/14 05:19, Praveen Baratam wrote:

Currently its configured with *multi_backend* and two buckets as follows-


[snip]


Now we want to delete the *cache* bucket and add another bucket
*temp* for another purpose.

[snip]

I am aware that I can do it using the HTTP Buckets API but can I shut
down all nodes, modify the app.config on all nodes and restart the
cluster safely with out losing any data?



Praveen, you are confusing the concepts of backend and bucket. They 
are not the same at all. Have a quick read thru the documentation and 
you should find things make sense quickly.


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Keys that won't disappear from indexes

[Toby Corkindale]

Hi Evan,
Thanks -- that resulted in a list that looks like:
{ok,[128,0,0,0,
 128,1,0,0,
 128,2,0,0,
 128,3,0,0,
 128,4,0,0,
 128,5,0,0,
... etc ...

That confirms my suspicions that the Java client is mangling the keys 
while processing them as unicode strings, because if I print out the 
values of the characters of the string there, I get:


65533,0,0,0
65533,1,0,0
65533,2,0,0
etc


On 06/11/13 15:57, Evan Vigil-McClanahan wrote:

Since we're talking about indices:

you can attach to the console with `riak attach` (then detach with
control-d (1.3 and below) or control-c (1.4+) note that this is can
kill your node on older versions, so be careful to use the correct
one).  Then edit the below to match up with your query, and then paste
it in (note that all the period below are intentional and required):

{ok, C} = riak:local_client().
{ok, Query} = riak_index:to_index_query(fieldname_bin,
[begin_point, end_point]).
C:get_index(bucket_name, Query).

If you need to adjust the Query because of a typo or something, just
do f(Query). to unbind it.

fieldname_bin is the binary index or whatever you're searching for. If
you were using $key, your Query might look something like this:

(perfdev@127.0.0.1)17 {ok, Query} =
riak_index:to_index_query($key, [,
255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255]).
{ok,{range,$key,,}}
(perfdev@127.0.0.1)20 C:put(riak_object:new(test, b, c)).
ok
(perfdev@127.0.0.1)21 C:get_index(test, Query).
{ok,[b]}

This should get you a list of keys in erlang binary format, which will
look something like asdasdasd or 44,0,32,130 if there are
unprintable characters in the string.


On Tue, Nov 5, 2013 at 4:40 PM, Toby Corkindale
toby.corkind...@strategicdata.com.au wrote:

On 06/11/13 11:30, Evan Vigil-McClanahan wrote:


You can replace int_to_bin with int_to_str to make it easier to debug
in the future, I suppose.  I am not sure how to get them to be fetched
as bytes, without may altering the client.

You could just attach to the console and run whatever listing command
you're running there, which would give you the answer as unfiltered
erlang binaries, which are easy to understand.



Ah, I'm really not familiar enough with Erlang and Riak to be doing that.
Which API applies to console commands? I'll take a look. (Is it just the
same as the Erlang client?)




Is this easily replicable on a new cluster?



I think it should be -- the only difference over default configuration is
that LevelDB is configured as the default backend.
Run basho_bench with the pbc-client test to generate the initial keys and
you should be set.


T


On Tue, Nov 5, 2013 at 4:17 PM, Toby Corkindale
toby.corkind...@strategicdata.com.au wrote:


Hi Evan,
These keys were originally created by basho-bench, using:
{key_generator, {int_to_bin, {uniform_int, 1}}}.

Of the 10k keys, it seems half could be removed, but not the other half.

Now I've tried storing keys with the same key as the un-deleteable ones,
waiting a minute, and then deleting them again.. this isn't seeming to
help!

I don't know if it's significant, but I'm working with the Java client
here
(protocol buffers). I note that the bad keys are basically just bytes,
not
actual ascii strings, and they do contain nulls.

Actually, here's something I just noticed -- the keys I'm getting from
the
index are repeating! It's the same 39 keys, repeated 128 times.

O.o

Are there any known bugs in the PBC interface when it comes to binary
keys?
I know the HTTP interface just crashes out completely.

I'm fetching the keys in a manner that returns strings; is there a way to
fetch them as bytes? Maybe that would work better; I'm wondering if the
client is attempting to convert the bytes into unicode strings and
dropping
invalid characters?


On 05/11/13 03:44, Evan Vigil-McClanahan wrote:



Hi Toby.

It's possible, since they're stored separately, that the objects were
deleted but the indices were left in place because of some error (e.g.
the operation failed for some reason between the object removal and
the index removal).  One of the things on the feature list for the
next release is AAE of index values, which should take care of this
case.  This is really rare, but not unknown.  It'd be interesting to
know how you ended up with so many.

In the mean time, the only way I can think of to get rid of them
(other than deleting them from the console, which would require taking
nodes down and a lot of manual effort), would be to write another
value that would have the same index, then delete it, which should
normally succeed.

I'll ask around to see if there is anything that might work better.






___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Keys that won't disappear from indexes

[Toby Corkindale]

Hi Evan,
These keys were originally created by basho-bench, using:
{key_generator, {int_to_bin, {uniform_int, 1}}}.

Of the 10k keys, it seems half could be removed, but not the other half.

Now I've tried storing keys with the same key as the un-deleteable ones, 
waiting a minute, and then deleting them again.. this isn't seeming to help!


I don't know if it's significant, but I'm working with the Java client 
here (protocol buffers). I note that the bad keys are basically just 
bytes, not actual ascii strings, and they do contain nulls.


Actually, here's something I just noticed -- the keys I'm getting from 
the index are repeating! It's the same 39 keys, repeated 128 times.


O.o

Are there any known bugs in the PBC interface when it comes to binary 
keys? I know the HTTP interface just crashes out completely.


I'm fetching the keys in a manner that returns strings; is there a way 
to fetch them as bytes? Maybe that would work better; I'm wondering if 
the client is attempting to convert the bytes into unicode strings and 
dropping invalid characters?


On 05/11/13 03:44, Evan Vigil-McClanahan wrote:

Hi Toby.

It's possible, since they're stored separately, that the objects were
deleted but the indices were left in place because of some error (e.g.
the operation failed for some reason between the object removal and
the index removal).  One of the things on the feature list for the
next release is AAE of index values, which should take care of this
case.  This is really rare, but not unknown.  It'd be interesting to
know how you ended up with so many.

In the mean time, the only way I can think of to get rid of them
(other than deleting them from the console, which would require taking
nodes down and a lot of manual effort), would be to write another
value that would have the same index, then delete it, which should
normally succeed.

I'll ask around to see if there is anything that might work better.

On Sun, Nov 3, 2013 at 7:40 PM, Toby Corkindale
toby.corkind...@strategicdata.com.au wrote:

On 01/11/13 14:04, Toby Corkindale wrote:


Hi,
I have around 5000 keys which just won't die.
No matter how many times I delete them, they still show up in the 2i
$bucket=_ index.

Actually attempting to retrieve the keys results in a not-found - even

if I've requested that tombstones be returned.

I'm interested to know what is going on here?



Anyone?

Should I report this as a bug against 1.4.2?


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Keys that won't disappear from indexes

[Toby Corkindale]

On 06/11/13 11:30, Evan Vigil-McClanahan wrote:

You can replace int_to_bin with int_to_str to make it easier to debug
in the future, I suppose.  I am not sure how to get them to be fetched
as bytes, without may altering the client.

You could just attach to the console and run whatever listing command
you're running there, which would give you the answer as unfiltered
erlang binaries, which are easy to understand.


Ah, I'm really not familiar enough with Erlang and Riak to be doing 
that. Which API applies to console commands? I'll take a look. (Is it 
just the same as the Erlang client?)




Is this easily replicable on a new cluster?


I think it should be -- the only difference over default configuration 
is that LevelDB is configured as the default backend.
Run basho_bench with the pbc-client test to generate the initial keys 
and you should be set.


T


On Tue, Nov 5, 2013 at 4:17 PM, Toby Corkindale
toby.corkind...@strategicdata.com.au wrote:

Hi Evan,
These keys were originally created by basho-bench, using:
{key_generator, {int_to_bin, {uniform_int, 1}}}.

Of the 10k keys, it seems half could be removed, but not the other half.

Now I've tried storing keys with the same key as the un-deleteable ones,
waiting a minute, and then deleting them again.. this isn't seeming to help!

I don't know if it's significant, but I'm working with the Java client here
(protocol buffers). I note that the bad keys are basically just bytes, not
actual ascii strings, and they do contain nulls.

Actually, here's something I just noticed -- the keys I'm getting from the
index are repeating! It's the same 39 keys, repeated 128 times.

O.o

Are there any known bugs in the PBC interface when it comes to binary keys?
I know the HTTP interface just crashes out completely.

I'm fetching the keys in a manner that returns strings; is there a way to
fetch them as bytes? Maybe that would work better; I'm wondering if the
client is attempting to convert the bytes into unicode strings and dropping
invalid characters?


On 05/11/13 03:44, Evan Vigil-McClanahan wrote:


Hi Toby.

It's possible, since they're stored separately, that the objects were
deleted but the indices were left in place because of some error (e.g.
the operation failed for some reason between the object removal and
the index removal).  One of the things on the feature list for the
next release is AAE of index values, which should take care of this
case.  This is really rare, but not unknown.  It'd be interesting to
know how you ended up with so many.

In the mean time, the only way I can think of to get rid of them
(other than deleting them from the console, which would require taking
nodes down and a lot of manual effort), would be to write another
value that would have the same index, then delete it, which should
normally succeed.

I'll ask around to see if there is anything that might work better.



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Keys that won't disappear from indexes

[Toby Corkindale]

On 01/11/13 14:04, Toby Corkindale wrote:

Hi,
I have around 5000 keys which just won't die.
No matter how many times I delete them, they still show up in the 2i
$bucket=_ index.

Actually attempting to retrieve the keys results in a not-found - even
if I've requested that tombstones be returned.

I'm interested to know what is going on here?


Anyone?

Should I report this as a bug against 1.4.2?

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Import big data to Riak

[Toby Corkindale]

On 30/10/13 02:59, Georgi Ivanov wrote:

On Tuesday 29 October 2013 16:50:25 Rune Skou Larsen wrote:

Den 29-10-2013 16:32, Georgi Ivanov skrev:

Hi and thank you for the reply. My comment follow:

Your tests are not close to what you are going to have in production


My tests are exactly what we will have in production. 1 node or in best
case 2 nodes.
We don't care about durability here.


If you don't care about durability, why do you want multiple copies of
each object? I.e. why not just keep n=1?

The only reason is that if the HW dies we will have to import the data again
.. which takes like one week.

That's why i would prefer 2 nodes setup



   Our request per second will be extremely

low too(300 per day ?)
Also we use 2i indexes.



Given your parameters, I have to say.. are you sure Riak is the right 
tool for your job?


A classical database, like Postgresql, could be a better fit.
It can import data much faster than Riak for a single client, and 
provides indexes, searching, etc.
And of course you can have streaming replication so that you have a hot 
spare in case of hardware failure.




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Keys that won't disappear from indexes

[Toby Corkindale]

Hi,
I have around 5000 keys which just won't die.
No matter how many times I delete them, they still show up in the 2i 
$bucket=_ index.


Actually attempting to retreive the keys results in a not-found - even 
if I've requested that tombstones be returned.


I'm interested to know what is going on here?

Cheers,
Toby

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: benchmark results on my $75/month 5 node cluster :)

[Toby Corkindale]

Thanks,
it's interesting to hear of what performance other people are achieving 
with Riak.


With a not-too-dissimilarly-configured[1] basho_bench test, we see 
throughput between 6000-7000 ops/sec. get/put latencies are under 2/4ms 
respectively for mean values, and under 6/12ms for 99th percentile.


This is on real (non-virtual) hardware, but not high-end gear.
Hopefully we're not too too far off what can reasonably be achieved.

-T

[1: Test parameters:
  {concurrent, 16}.
  {driver, basho_bench_driver_riakc_pb}.
  {key_generator, {int_to_bin, {uniform_int, 1}}}.
  {value_generator, {fixed_bin, 1}}.
  {riakc_pb_replies, 1}.
  {operations, [{get, 3}, {put, 1} ]}.
]

On 28/10/13 06:35, Alex Rice wrote:

Here is an updated 2 hour graph of the same X-Small 5-node cluster,
after having performed all the Linux tuning steps I had missed first
time around. Also added ntpd service to all nodes. Looks like slightly
better throughput  latency, but a much cleaner graph overall, and it
should burst much better:
https://dl.dropboxusercontent.com/u/22076075/summary-xsmall-bitcask-5nodes-2hr-tuned-vm.png

Here is the same cluster rebooted as Medium Azure VMs, same
benchmark settings except: 8 workers, shorter duration. Not sure why
it has errors, yet:
https://dl.dropboxusercontent.com/u/22076075/summary-medium-bitcask-5nodes-30-min.png

Anyways- back to work for me!!!
:)

On Wed, Oct 2, 2013 at 12:14 PM, Alex Rice a...@mindlube.com wrote:

Hey, something that is impressive about Riak is how well it performs
even on  super low end cloud virtualization (unlike for example,
couchbase, which hardly runs at all unless you have quad cores per
each node)

Having so much fun benchmarking, it's really distracting me from coding!

the cluster : 5 nodes on Windows Azure cloud
Ubuntu 12.04, Riak 1.2.4
node are xsmall vm (1 shared cpu core, 768 mb ram)

The xmall vms are only $15/month and they are not recommended for
production. but in a 5 node riak cluster, it's really nothing to
scoff at!

I suspect the main bottleneck is not the cpu or disk, rather I think
it's the each xsmall node is allowed only 5Mbit/sec on the ethernet.

Here is a basho_bench graph, if anyone feels like viewing it
https://dl.dropboxusercontent.com/u/22076075/summary-xsmall-bitcask-5nodes-20min.png

note the basho bench was running on a quad core machine on the same
vlan with 5 workers.


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Date implementation in Riak's java vm

[Toby Corkindale]

Hi,
I've been having some trouble getting the javascript vm in Riak to 
operate correctly on Date objects.


According to this, I should be able to create dates using an ISO8601 
date string, or a simplified -MM-DD format.


http://www.ecma-international.org/ecma-262/5.1/#sec-15.9.1.15

However in practice, this just results in Invalid Date errors.

Is this a known issue, or am I doing something wrong?

thanks,
Toby

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

List buckets results in error: bad_utf8_character_code

[Toby Corkindale]

If I run curl -i http://bld-riak/buckets?buckets=true; I receive this 
error about a bad utf8 character code, and nothing else.


This is Riak version 1.4.2-1 (from the official apt repo)


HTTP/1.1 500 Internal Server Error
Vary: Accept-Encoding
Server: MochiWeb/1.1 WebMachine/1.10.0 (never breaks eye contact)
Date: Thu, 17 Oct 2013 00:51:03 GMT
Content-Type: text/html
Content-Length: 1103

htmlheadtitle500 Internal Server 
Error/title/headbodyh1Internal Server Error/h1The server 
encountered an error while processing this 
request:brpre{error,{exit,{ucs,{bad_utf8_character_code}},

 [{xmerl_ucs,from_utf8,1,[{file,xmerl_ucs.erl},{line,185}]},
  {mochijson2,json_encode_string,2,
  [{file,src/mochijson2.erl},{line,186}]},
  {mochijson2,'-json_encode_array/2-fun-0-',3,
  [{file,src/mochijson2.erl},{line,157}]},
  {lists,foldl,3,[{file,lists.erl},{line,1197}]},
  {mochijson2,json_encode_array,2,
  [{file,src/mochijson2.erl},{line,159}]},
  {mochijson2,'-json_encode_proplist/2-fun-0-',3,
  [{file,src/mochijson2.erl},{line,167}]},
  {lists,foldl,3,[{file,lists.erl},{line,1197}]},
  {mochijson2,json_encode_proplist,2,

[{file,src/mochijson2.erl},{line,170}]}]}}/prePHRADDRESSmochiweb+webmachine 
web server/ADDRESS/body/html


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: List buckets results in error: bad_utf8_character_code

[Toby Corkindale]

On 17/10/13 11:54, Toby Corkindale wrote:

If I run curl -i http://bld-riak/buckets?buckets=true; I receive this
error about a bad utf8 character code, and nothing else.

This is Riak version 1.4.2-1 (from the official apt repo)


HTTP/1.1 500 Internal Server Error
Vary: Accept-Encoding
Server: MochiWeb/1.1 WebMachine/1.10.0 (never breaks eye contact)
Date: Thu, 17 Oct 2013 00:51:03 GMT
Content-Type: text/html
Content-Length: 1103

htmlheadtitle500 Internal Server
Error/title/headbodyh1Internal Server Error/h1The server
encountered an error while processing this
request:brpre{error,{exit,{ucs,{bad_utf8_character_code}},
  [{xmerl_ucs,from_utf8,1,[{file,xmerl_ucs.erl},{line,185}]},
   {mochijson2,json_encode_string,2,
   [{file,src/mochijson2.erl},{line,186}]},
   {mochijson2,'-json_encode_array/2-fun-0-',3,
   [{file,src/mochijson2.erl},{line,157}]},
   {lists,foldl,3,[{file,lists.erl},{line,1197}]},
   {mochijson2,json_encode_array,2,
   [{file,src/mochijson2.erl},{line,159}]},
   {mochijson2,'-json_encode_proplist/2-fun-0-',3,
   [{file,src/mochijson2.erl},{line,167}]},
   {lists,foldl,3,[{file,lists.erl},{line,1197}]},
   {mochijson2,json_encode_proplist,2,

[{file,src/mochijson2.erl},{line,170}]}]}}/prePHRADDRESSmochiweb+webmachine
web server/ADDRESS/body/html



If I use the Java client over PBC to get the list, then there is no 
crash, but I can see that many of the bucket names look like garbage.

I'd guess that those were created by Riak CS?
Is that normal? It's a pity it breaks the HTTP interface.


buckets: java.util.Set[String] =  [0o:uT�?�p�B?�, 0b: ��J}��Ԭ, 
0o:?i�!��?�?�/�.�, 0b:�^�?�e��?gl�'�|_, moss.buckets, test-bucket, 
0o:�?TV�:X?�1�Y`�D�, 0o:H�?p�`��l�K�h?��, auth_attr, 0o:s??���#�O�  �, 
0o:���?�m�?�$w飋?, 0b:�?��I�7s C�d[_4�, 0o:=�'=}?N�?A f ??, 
0o:?bg���`c9� ??/I , 0b:�Ug���ǻ?7'?��3P, 0o:���TT��WP?��?֦, 1d:WebVal, 
0o:??�L?X�A���.ɝ�, 0b:�ݜ��l4�+؈�?�V�, 0o: H�v�R�K^?wu�H��, 1h:test, 
riak-cs-gc, 0b:� U�?!�ף?��R=?+, 0o:��K�a d���w�_�zF, 0b:��@��8ۃ` ?w :��, 
0o:b��~��[fy��3?��, 0b:?��,�zS�`|5J�?, 0b:?�i�� ?_B�?§��,, 0o:��l�Q�鳾 
�~�%�, 0b:���q�u��k��1, 0b:���b���#?�hC�Ě%, 0o:��@��8ۃ` ?w :��, 
0b:--�?q�, 0o:�?=u^��b�m�J���, 0o:���?��E?��?j9�`^, 
0o:?c�ZhRi��Z��`��?, 0o:V�U��*:5VaG��?X�, 0o:��$��V���0?$�f, 
0o:|I��'�v�I]?MI-, 0o:v?vL� �d��M�M{, 0o:--�?q���...



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Unit testing, Riak buckets

[Toby Corkindale]

Hi,
I'd like to hear how other people are approaching the problem of 
cleaning Riak buckets up at the end of unit tests for their apps.


The problem I have is that multiple tests may be run at once (by 
different developers or different Jenkins' jobs or even just a 
parallelised test suite) so I can't really run a blanket delete-all at 
the end of the test suite, unless I use a randomly-named bucket each 
time. Yet if I do that, I'm concerned the test suite may crash out prior 
to the end sometimes, and then never delete that randomly-named bucket.



If secondary indexes aren't required, then the easy solution is to use a 
randomly-named Bitcask bucket which has a backend configured for a 
fairly short TTL.



Otherwise, I have wondered about creating buckets with a certain format, 
perhaps test-XX--MM-DD, (x=random) and then a nightly cron 
script can run to find all buckets timestamped from the previous day or 
earlier, and remove them. I gather listing all buckets is an expensive 
operation though, although it'll only be running on a testing Riak cluster.



So I wondered how other developers are approaching this issue?


Cheers,
Toby

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Invalid hintfiles

[Toby Corkindale]

On 19/09/13 12:02, Seth Thomas wrote:

Toby,

Although I haven't traced a root cause just yet I've had several reports
in IRC of odd behavior when using nginx as a load balancer for Riak CS.
In almost every case, removing nginx from the equation and switching to
haproxy or pointing directly at the nodes in the application has
resolved issues. It's anecdotal until I can actually determine a root
cause but it's worth testing out if you're experiencing trouble.


Thanks for mentioning that Seth.
I posted some stuff to the list some months ago about config tweaks I 
had to make to get nginx to play nicely with Riak. Out of the box it 
definitely didn't work quite right.


In this instance, I had been able to reproduce the problems when going 
directly to the nodes from applications.. but it's interesting to hear 
that nginx potentially has issues beyond those we've experienced, so 
I'll look at getting haproxy used at some point. I've used haproxy in 
smaller test clusters and it's certainly felt like a better fit.


Cheers,
Toby


On September 18, 2013 at 8:48:27 PM, Toby Corkindale
(toby.corkind...@strategicdata.com.au) wrote:


On 19/09/13 11:17, Luke Bakken wrote:
 Hi Toby,

 Invalid hint files won't cause Riak to fail requests - there must have
 been something else happening. Hint files are used by Riak to speed
 start time when loading a large key set.

 You mentioned a load-balancer pool - are you using something like
 HAProxy to load-balance requests to your Riak CS cluster?

 The error: disconnected message is a good clue. If you can provide
 log files that may point to the cause.

Hi Luke,
I'm still seeing quite a few failed requests. I've been chasing the
hintfiles but I guess that was a red herring.

We're using nginx to load balance requests to Riak CS.
I tried going directly to each node in turn, and it didn't show that any
one node was reliably failing every request.

Hitting one server just now came up with OK/403/403/OK/OK.
Trying another was OK/OK/OK/OK/403 though.

Here's some logs from riak-cs:




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Debugging mapreduce

[Toby Corkindale]

On 25/09/13 11:20, Charl Matthee wrote:

Hi,

I am trying to run the following mapreduce query across my cluster:

# curl -XPOST http://10.179.229.209:8098/mapred -H Content-Type:
application/json -d '{inputs:tweets,
query:[{map:{language:javascript, source:function(value,
keyData, arg) {t = JSON.parse(value.values[0].data)[0]; if ((new Date
- new Date(t.created_at)) / 1000  2592000) return [t.id]; else return
[]}, keep:true}}]}'
{lineno:466,message:SyntaxError: syntax error,source:()}


Have you tried executing your javascript outside of Riak?
ie. paste the function into the Chrome debugger, then call it with a 
Riak-like data structure.


Also, consider wrapping the code in your function with an eval so you 
can catch errors that occur. (Then either ejslog them or return them as 
results of the map phase)




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Riak CS 1.4.1 bug? InvalidAccessKeyId reported for deletion calls

[Toby Corkindale]

I've just upgraded from Riak CS 1.3.1 to 1.4.1

Using s3cmd to test a few things, I've found some odd behaviour.
Creating a bucket and putting a file works just fine, eg:

s3cmd mb s3://test
s3cmd put README s3://test
s3cmd get s3://test/README

However if I try to delete a file or bucket, it throws an error:

s3cmd del s3://test/README
ERROR: S3 error: 403 (InvalidAccessKeyId): The AWS Access Key Id you 
provided does not exist in our records.


s3cmd rb s3://test
ERROR: S3 error: 403 (InvalidAccessKeyId): The AWS Access Key Id you 
provided does not exist in our records.



Have I messed something up during the upgrade, or is this a bug in 1.4.1?

ta
Toby

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak CS 1.4.1 bug? InvalidAccessKeyId reported for deletion calls

[Toby Corkindale]

Ah, hold on.. have just discovered that rather than it being deletion 
calls, it seems to just be every X calls of any sort.. sounds like one 
of the servers in the load-balancer pool must be misconfigured somehow, 
but the rest are OK.


On 19/09/13 08:34, Toby Corkindale wrote:

I've just upgraded from Riak CS 1.3.1 to 1.4.1

Using s3cmd to test a few things, I've found some odd behaviour.
Creating a bucket and putting a file works just fine, eg:

s3cmd mb s3://test
s3cmd put README s3://test
s3cmd get s3://test/README

However if I try to delete a file or bucket, it throws an error:

s3cmd del s3://test/README
ERROR: S3 error: 403 (InvalidAccessKeyId): The AWS Access Key Id you
provided does not exist in our records.

s3cmd rb s3://test
ERROR: S3 error: 403 (InvalidAccessKeyId): The AWS Access Key Id you
provided does not exist in our records.


Have I messed something up during the upgrade, or is this a bug in 1.4.1?

ta
Toby

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Invalid hintfiles (was: Riak CS 1.4.1 bug? InvalidAccessKeyId reported for deletion calls)

[Toby Corkindale]

I found one Riak server was reporting a lot of errors like
[error] 0.808.0 Hintfile 
'/var/lib/riak/bitcask/68507889249886074290797726533575766546371837952/3.bitcask.hint' 
invalid


And the Riak CS logs contained a lot of messages about being unable to 
retrieve s3 user details because error: disconnected


I think I've blown away the bad hintfiles and have had them repaired 
from other replicas now, and I haven't seen any more errors for a little 
while.


I'm not sure what caused those to become invalid.
Just a thought, but would be good if Riak could automatically repair 
them rather than failing requests.


Cheer,s
Toby

On 19/09/13 08:42, Toby Corkindale wrote:

Ah, hold on.. have just discovered that rather than it being deletion
calls, it seems to just be every X calls of any sort.. sounds like one
of the servers in the load-balancer pool must be misconfigured somehow,
but the rest are OK.

On 19/09/13 08:34, Toby Corkindale wrote:

I've just upgraded from Riak CS 1.3.1 to 1.4.1

Using s3cmd to test a few things, I've found some odd behaviour.
Creating a bucket and putting a file works just fine, eg:

s3cmd mb s3://test
s3cmd put README s3://test
s3cmd get s3://test/README

However if I try to delete a file or bucket, it throws an error:

s3cmd del s3://test/README
ERROR: S3 error: 403 (InvalidAccessKeyId): The AWS Access Key Id you
provided does not exist in our records.

s3cmd rb s3://test
ERROR: S3 error: 403 (InvalidAccessKeyId): The AWS Access Key Id you
provided does not exist in our records.


Have I messed something up during the upgrade, or is this a bug in 1.4.1?

ta
Toby

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Invalid hintfiles

[Toby Corkindale]

On 19/09/13 11:17, Luke Bakken wrote:

Hi Toby,

Invalid hint files won't cause Riak to fail requests - there must have
been something else happening. Hint files are used by Riak to speed
start time when loading a large key set.

You mentioned a load-balancer pool - are you using something like
HAProxy to load-balance requests to your Riak CS cluster?

The error: disconnected message is a good clue. If you can provide
log files that may point to the cause.


Hi Luke,
I'm still seeing quite a few failed requests. I've been chasing the 
hintfiles but I guess that was a red herring.


We're using nginx to load balance requests to Riak CS.
I tried going directly to each node in turn, and it didn't show that any 
one node was reliably failing every request.


Hitting one server just now came up with OK/403/403/OK/OK.
Trying another was OK/OK/OK/OK/403 though.

Here's some logs from riak-cs:

error.log:2013-09-19 11:37:02.242 [error] 
0.5105.0@riak_cs_wm_common:maybe_create_user:223 Retrieval of user 
record for s3 failed. Reason: disconnected


There wasn't anything immediately either side of that. The riak logs for 
the same minute on that server likewise do not have anything.


There's quite a lot of free memory on the servers; they have 32000 file 
handles available.


Toby



On Wed, Sep 18, 2013 at 4:29 PM, Toby Corkindale
toby.corkind...@strategicdata.com.au wrote:

I found one Riak server was reporting a lot of errors like
[error] 0.808.0 Hintfile
'/var/lib/riak/bitcask/68507889249886074290797726533575766546371837952/3.bitcask.hint'
invalid

And the Riak CS logs contained a lot of messages about being unable to
retrieve s3 user details because error: disconnected

I think I've blown away the bad hintfiles and have had them repaired from
other replicas now, and I haven't seen any more errors for a little while.

I'm not sure what caused those to become invalid.
Just a thought, but would be good if Riak could automatically repair them
rather than failing requests.

Cheer,s
Toby

On 19/09/13 08:42, Toby Corkindale wrote:


Ah, hold on.. have just discovered that rather than it being deletion
calls, it seems to just be every X calls of any sort.. sounds like one
of the servers in the load-balancer pool must be misconfigured somehow,
but the rest are OK.

On 19/09/13 08:34, Toby Corkindale wrote:


I've just upgraded from Riak CS 1.3.1 to 1.4.1

Using s3cmd to test a few things, I've found some odd behaviour.
Creating a bucket and putting a file works just fine, eg:

s3cmd mb s3://test
s3cmd put README s3://test
s3cmd get s3://test/README

However if I try to delete a file or bucket, it throws an error:

s3cmd del s3://test/README
ERROR: S3 error: 403 (InvalidAccessKeyId): The AWS Access Key Id you
provided does not exist in our records.

s3cmd rb s3://test
ERROR: S3 error: 403 (InvalidAccessKeyId): The AWS Access Key Id you
provided does not exist in our records.


Have I messed something up during the upgrade, or is this a bug in 1.4.1?



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Invalid hintfiles

[Toby Corkindale]

Hi Andrew,
Thanks for that -- so far things are looking stable after making that 
change.


-T

On 19/09/13 13:27, Andrew Stone wrote:

Hi Toby,

Can you try raising the pb_backlog to 128 in your riak app.config on
each node. It's likely those disconnect errors are left over from the
stampede of connections from the CS connection pool on startup. For one
reason or another the resets don't come through and the hanging
disconnected socket isn't discovered until the next send attempt.

-Andrew


On Wed, Sep 18, 2013 at 10:31 PM, Toby Corkindale
toby.corkind...@strategicdata.com.au
mailto:toby.corkind...@strategicdata.com.au wrote:

On 19/09/13 11:17, Luke Bakken wrote:

The error: disconnected message is a good clue. If you can provide
log files that may point to the cause.


See below for logs from riak and riak cs, for a roughly five minute
window during which I'd fiddled with the load balancer and DNS to
try and isolate this server from other requests, and only send mine
to it.

Unfortunately I really don't see much in there apart from the
disconnected messages. I did note that there were warnings about
system hitting high watermarks for memory at one point though.
That's a bit odd, as the machine wasn't near max capacity at any
point - it's only 8GB, but 'free' was reporting that most was being
used just for buffers and cache at the time; as follows:

ie.
  total   used   free sharedbuffers
cached
Mem:   817641238034044373008  0  34112
  3052340
-/+ buffers/cache: 7169527459460
Swap:  3903484  03903484





___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: riak control post upgrade

[Toby Corkindale]

On 06/08/13 07:51, Paul Ingalls wrote:

Hope you all don't get pissed for me spamming the list….;)

I upgraded my cluster to 1.4.1 in hopes the levelDB changes may help.
  The cluster is now up and running, however Riak Control is complaining:

The following nodes are currently incompatible with Riak Control:
all my nodes


I've hit exactly the same problem, after doing a rolling upgrade to 
1.4.1 on Ubuntu.

I'm using the official Riak apt repo.

Any hints on how to fix this?

ta,
Toby

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: SSL certificates and Riak CS

[Toby Corkindale]

On 31/07/13 15:14, Toby Corkindale wrote:

Hi,
Apologies if this has been covered before, but I didn't spot anything.

What sort of SSL certificate do you need to get to cover all the
subdomains that may exist for a Riak CS instance?

As I understand it, each bucket gets turned into a subdomain, so you'd
need an SSL cert covering *.s3.example.com, *.*.s3.example.com,
*.*.*.s3.example.com and so forth... The problem is that AFAICT you
can't buy SSL certs with any more than one wildcard in them.

What are other people doing for this in the wild? I presume there's some
way to do this, or else all the other S3-like systems wouldn't work over
HTTPS!



So I looked into this a bit more, and I realised Amazon's S3 system has 
the same problem -- buckets with dots in the name throw SSL cert errors too.


It's just an SSL cert issue, not much anyone can do about it I think?

-Toby

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

SSL certificates and Riak CS

[Toby Corkindale]

Hi,
Apologies if this has been covered before, but I didn't spot anything.

What sort of SSL certificate do you need to get to cover all the 
subdomains that may exist for a Riak CS instance?


As I understand it, each bucket gets turned into a subdomain, so you'd 
need an SSL cert covering *.s3.example.com, *.*.s3.example.com, 
*.*.*.s3.example.com and so forth... The problem is that AFAICT you 
can't buy SSL certs with any more than one wildcard in them.


What are other people doing for this in the wild? I presume there's some 
way to do this, or else all the other S3-like systems wouldn't work over 
HTTPS!


Cheers,
Toby

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com