Re: ANN: Riak KV 2.9.0 P4 - New patch update available

[Martin Sumner]

Note the link to packages was incorrect in the previous announcement.
Corrected link below.


The latest patch for the 2.9.0 is now available.

On github the release can be found at -
https://github.com/basho/riak/tree/riak-2.9.0p4. Updated packages are also
available (thanks to Nick Adams at TI Tokyo) -
https://files.tiot.jp/riak/kv/2.9/2.9.0p4/

Details on the fixes are available via the release notes -
https://github.com/basho/riak/blob/riak-2.9.0p4/RELEASE-NOTES.md.

Previously I had announced concerns over the potential severity of an issue
1707 (https://github.com/basho/riak_kv/issues/1707) in the 2.9.0 release -
http://lists.basho.com/pipermail/riak-users_lists.basho.com/2019-August/039368.html.
Although this issue remains unexplained, it is no longer believed to be as
a result of recent changes, or present a direct risk of data loss.

For those running earlier patches of the Riak KV 2.9.0 release, upgrading
is recommended, but  the update is not considered urgent unless running
with Tictac AAE enabled.

Regards

On Mon, 9 Sep 2019 at 09:53, Martin Sumner 
wrote:

> The latest patch for the 2.9.0 is now available.
>
> On github the release can be found at -
> https://github.com/basho/riak/tree/riak-2.9.0p4. Updated packages are
> also available (thanks to Nick Adams at TI Tokyo) -
> https://files.tiot.jp/riak/kv/2.9/2.9.0p4/
> .
>
> Details on the fixes are available via the release notes -
> https://github.com/basho/riak/blob/riak-2.9.0p4/RELEASE-NOTES.md.
>
> Previously I had announced concerns over the potential severity of an
> issue 1707 (https://github.com/basho/riak_kv/issues/1707) in the 2.9.0
> release -
> http://lists.basho.com/pipermail/riak-users_lists.basho.com/2019-August/039368.html.
> Although this issue remains unexplained, it is no longer believed to be as
> a result of recent changes, or present a direct risk of data loss.
>
> For those running earlier patches of the Riak KV 2.9.0 release, upgrading
> is recommended, but  the update is not considered urgent unless running
> with Tictac AAE enabled.
>
> Regards
>
> Martin
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: File corruption after power cut

[Fred Dushin]

Just remember if you are not running AAE, you will want to do a manual repair 
on that partition, in order to ensure you have adequate replication:

https://docs.riak.com/riak/kv/2.2.3/using/repair-recovery/repairs.1.html#repairing-partitions

> On Sep 4, 2019, at 5:19 AM, Bryan Hunt  
> wrote:
> 
> Easiest solution is to just delete the vnode - and let it recover from 
> replicas - vnode directory will be 
> 844930634081928249586505293914101120738586001408
> 
> 
> 
>> On 4 Sep 2019, at 10:01, Guido Medina > > wrote:
>> 
>> Hi all,
>> 
>> We had a power cut which caused one of the nodes to corrupt one of the 
>> LevelDB files, after this that node doesn't even want to start, here is the 
>> error we are seeing:
>>> 2019-09-04 08:46:41.584 [error] <0.2329.0>@riak_kv_vnode:init:527 Failed to 
>>> start riak_kv_eleveldb_backend backend for index 
>>> 844930634081928249586505293914101120738586001408 error: 
>>> {db_open,"Corruption: CURRENT file does not end with newline"}
>> 
>> Thanks in advance for your help ;-)
>> Guido.
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com 
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> 
> 
> Code Sync & Erlang Solutions Conferences
> Code BEAM Lite  
> - Budapest: 20 September 2019
> Code BEAM Lite  
> - New York City: 01 October 2019
> Code BEAM Lite  
> - Berlin: 11 October 2019
> RabbitMQ Summit  
> - London: 4 November 2019
> Code Mesh LDN  - 
> London: 7-8 November 2019
> Code BEAM Lite  
> - Bangalore: 14 November 2019
> Code BEAM Lite  
> - Amsterdam: 29 November 2019
> Lambda Days  - 
> Kraków: 13-14 February 2020
> Code BEAM SF  - 
> San Francisco: 5-6 March 2020
> Code BEAM STO - Stockholm: 28-29 May 2020
> 
> Erlang Solutions cares about your data and privacy; please find all details 
> about the basis for communicating with you and the way we process your data 
> in our Privacy Policy 
> .You can update your 
> email preferences or opt-out from receiving Marketing emails here 
> .
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: File corruption after power cut

[Guido Medina]

Bryan* (misspelled your name), need a morning coffee :(

On 04/09/2019 10:29, Guido Medina wrote:
Thanks a lot Brian, I have deleted such directory and now the node 
started, let's see how it behaves.


Guido.

On 04/09/2019 10:19, Bryan Hunt wrote:
Easiest solution is to just delete the vnode - and let it recover 
from replicas - vnode directory will be 
844930634081928249586505293914101120738586001408


On 4 Sep 2019, at 10:01, Guido Medina > wrote:


Hi all,

We had a power cut which caused one of the nodes to corrupt one of 
the LevelDB files, after this that node doesn't even want to start, 
here is the error we are seeing:
2019-09-04 08:46:41.584 [error] <0.2329.0>@riak_kv_vnode:init:527 
Failed to start riak_kv_eleveldb_backend backend for index 
844930634081928249586505293914101120738586001408 error: 
{db_open,"Corruption: CURRENT file does not end with newline"}


Thanks in advance for your help ;-)
Guido.
___
riak-users mailing list
riak-users@lists.basho.com 
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



Code Sync & Erlang Solutions Conferences

Code BEAM Lite 
- 
Budapest: 20 September 2019


Code BEAM Lite 
- New 
York City: 01 October 2019


Code BEAM Lite 
- 
Berlin: 11 October 2019


RabbitMQ Summit 
- 
London: 4 November 2019


Code Mesh LDN 
- 
London: 7-8 November 2019


Code BEAM Lite 
- 
Bangalore: 14 November 2019


Code BEAM Lite 
- 
Amsterdam: 29 November 2019


Lambda Days 
- 
Kraków: 13-14 February 2020


Code BEAM SF 
- San 
Francisco: 5-6 March 2020


Code BEAM STO - Stockholm: 28-29 May 2020

/Erlang Solutions cares about your data and privacy; please find all 
details about the basis for communicating with you and the way we 
process your data in our //Privacy Policy/ 
/.You can 
update your email preferences or opt-out from receiving Marketing 
emails here ./






___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: File corruption after power cut

[Guido Medina]

Thanks a lot Brian, I have deleted such directory and now the node 
started, let's see how it behaves.


Guido.

On 04/09/2019 10:19, Bryan Hunt wrote:
Easiest solution is to just delete the vnode - and let it recover from 
replicas - vnode directory will be 
844930634081928249586505293914101120738586001408


On 4 Sep 2019, at 10:01, Guido Medina > wrote:


Hi all,

We had a power cut which caused one of the nodes to corrupt one of 
the LevelDB files, after this that node doesn't even want to start, 
here is the error we are seeing:
2019-09-04 08:46:41.584 [error] <0.2329.0>@riak_kv_vnode:init:527 
Failed to start riak_kv_eleveldb_backend backend for index 
844930634081928249586505293914101120738586001408 error: 
{db_open,"Corruption: CURRENT file does not end with newline"}


Thanks in advance for your help ;-)
Guido.
___
riak-users mailing list
riak-users@lists.basho.com 
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



Code Sync & Erlang Solutions Conferences

Code BEAM Lite 
- 
Budapest: 20 September 2019


Code BEAM Lite 
- New 
York City: 01 October 2019


Code BEAM Lite 
- Berlin: 
11 October 2019


RabbitMQ Summit 
- London: 
4 November 2019


Code Mesh LDN 
- London: 
7-8 November 2019


Code BEAM Lite 
- 
Bangalore: 14 November 2019


Code BEAM Lite 
- 
Amsterdam: 29 November 2019


Lambda Days 
- Kraków: 
13-14 February 2020


Code BEAM SF 
- San 
Francisco: 5-6 March 2020


Code BEAM STO - Stockholm: 28-29 May 2020

/Erlang Solutions cares about your data and privacy; please find all 
details about the basis for communicating with you and the way we 
process your data in our //Privacy Policy/ 
/.You can update 
your email preferences or opt-out from receiving Marketing emails here 
./




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: File corruption after power cut

[Bryan Hunt]

Easiest solution is to just delete the vnode - and let it recover from replicas 
- vnode directory will be 844930634081928249586505293914101120738586001408



> On 4 Sep 2019, at 10:01, Guido Medina  wrote:
> 
> Hi all,
> 
> We had a power cut which caused one of the nodes to corrupt one of the 
> LevelDB files, after this that node doesn't even want to start, here is the 
> error we are seeing:
>> 2019-09-04 08:46:41.584 [error] <0.2329.0>@riak_kv_vnode:init:527 Failed to 
>> start riak_kv_eleveldb_backend backend for index 
>> 844930634081928249586505293914101120738586001408 error: 
>> {db_open,"Corruption: CURRENT file does not end with newline"}
> 
> Thanks in advance for your help ;-)
> Guido.
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


-- 


Code Sync & Erlang Solutions Conferences

Code BEAM Lite 
 - Budapest: 
20 September 2019

Code BEAM Lite 
 - New York 
City: 01 October 2019

Code BEAM Lite 
 - Berlin: 11 
October 2019

RabbitMQ Summit 
 - London: 4 
November 2019

Code Mesh LDN 
 - London: 7-8 
November 2019

Code BEAM Lite 
 - Bangalore: 
14 November 2019

Code BEAM Lite 
 - Amsterdam: 
29 November 2019

Lambda Days 
 - Kraków: 
13-14 February 2020

Code BEAM SF 
 - San 
Francisco: 5-6 March 2020



Code BEAM STO - Stockholm: 28-29 May 2020





*Erlang Solutions cares about your data and privacy; please find all 
details about the basis for communicating with you and the way we process 
your data in our **Privacy Policy* 
*.You can update your 
email preferences or opt-out from receiving Marketing emails here 
.*
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: ANN: Riak 2.9.0 Urgent Patches Pending

[DeadZen]

We all appreciate it, and erring on the side of caution.  Thank you.

On Thu, Aug 22, 2019 at 12:22 PM Martin Sumner 
wrote:

> Follow-up on this announcement.  After further work today, it looks like
> the initial diagnosis of the Issue 1707 was incorrect, and that the defect
> does not obviously present a significant risk of data loss.  Work will
> continue on providing an updated patch to Riak 2.9.0 - but there should be
> no immediate concern over use of the unpatched version.
>
> Sorry for causing false alarm.
>
> Regards
>
> Martin
>
> On Thu, 22 Aug 2019 at 01:36, Martin Sumner 
> wrote:
>
>> There are 2 defects in the 2.9.0 release which should be patched within
>> the next 7 days, with a new version of 2.9.0 to be published before the end
>> of the month.  The defects are:
>>
>> https://github.com/basho/riak_kv/issues/1706
>> https://github.com/basho/riak_kv/issues/1707
>>
>> Note that the latter defect 1707, the scope of this bug is currently
>> unclear, but there exists significant risk that there is potential within
>> this bug for data loss, due to incorrect pruning of siblings.
>>
>> Previous releases of Riak also have this bug, if the metadata cache
>> (which is disabled by default) is enabled.  Enabling this cache is
>> controlled via
>> https://docs.riak.com/riak/kv/latest/configuring/reference/index.html 
>> (metadata_cache_size).
>> However, unlike previous releases the 2.9.0 release, when combined with use
>> of the leveled backend, Riak 2.9.0 will enable the path to this bug by
>> default.  This is because the logic of the metadata_cache is reused in
>> order to enable leveled HEAD requests within the PUT path.  Riak 2.9.0 with
>> other backends, and the metadata cache left disabled, will not have this
>> bug.
>>
>> I apologise for the ongoing churn of issues uncovered in release 2.9.0,
>> and for the potential risks associated with this latest bug in particular.
>> Clearly, the high rate of discovered problems has exposed that the test
>> process which surrounded this release was not good enough.  For release
>> 2.9.1 and release 3.0 the move has already been made to involve independent
>> testing organisations as a fundamental part of the development process,
>> from the start of the process.  Regardless of this, once the immediate
>> issues are patched, there is a need for some further reflection on what is
>> required to assure the safety of a release of Riak in the future.
>>
>> Regards
>>
>> Martin
>>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: ANN: Riak 2.9.0 Urgent Patches Pending

[Martin Sumner]

Follow-up on this announcement.  After further work today, it looks like
the initial diagnosis of the Issue 1707 was incorrect, and that the defect
does not obviously present a significant risk of data loss.  Work will
continue on providing an updated patch to Riak 2.9.0 - but there should be
no immediate concern over use of the unpatched version.

Sorry for causing false alarm.

Regards

Martin

On Thu, 22 Aug 2019 at 01:36, Martin Sumner 
wrote:

> There are 2 defects in the 2.9.0 release which should be patched within
> the next 7 days, with a new version of 2.9.0 to be published before the end
> of the month.  The defects are:
>
> https://github.com/basho/riak_kv/issues/1706
> https://github.com/basho/riak_kv/issues/1707
>
> Note that the latter defect 1707, the scope of this bug is currently
> unclear, but there exists significant risk that there is potential within
> this bug for data loss, due to incorrect pruning of siblings.
>
> Previous releases of Riak also have this bug, if the metadata cache (which
> is disabled by default) is enabled.  Enabling this cache is controlled via
> https://docs.riak.com/riak/kv/latest/configuring/reference/index.html 
> (metadata_cache_size).
> However, unlike previous releases the 2.9.0 release, when combined with use
> of the leveled backend, Riak 2.9.0 will enable the path to this bug by
> default.  This is because the logic of the metadata_cache is reused in
> order to enable leveled HEAD requests within the PUT path.  Riak 2.9.0 with
> other backends, and the metadata cache left disabled, will not have this
> bug.
>
> I apologise for the ongoing churn of issues uncovered in release 2.9.0,
> and for the potential risks associated with this latest bug in particular.
> Clearly, the high rate of discovered problems has exposed that the test
> process which surrounded this release was not good enough.  For release
> 2.9.1 and release 3.0 the move has already been made to involve independent
> testing organisations as a fundamental part of the development process,
> from the start of the process.  Regardless of this, once the immediate
> issues are patched, there is a need for some further reflection on what is
> required to assure the safety of a release of Riak in the future.
>
> Regards
>
> Martin
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak 2.9.0 - Update Available

[Martin Sumner]

There is now a third update available for 2.9.0:
https://github.com/basho/riak/tree/riak-2.9.0p3.

Again, the fixes are related to memory management in leveled, and
specifically related to references to sub-binaries.  This main issue was
related to a lazy-load of file metadata which occurs following a riak
restart, plus also an issue with managing memory use during journal
compaction following many days of repeated compactions.  Release notes (
https://github.com/basho/riak/blob/riak-2.9.0p3/RELEASE-NOTES.md) contain
some more details and links.

I would recommend updating from any previous release of 2.9.0 if you have
enabled either the leveled backend, or Tictac AAE.

Updated packages are available (thanks to Nick Adams at TI Tokyo) -
https://files.tiot.jp/riak/kv/2.9/2.9.0p3/.

Thanks again to the testing team at the NHS Spine project, Aaron Gibbon
(BJSS) and Ramen Sen, for their continued efforts to stress Riak 2.9.0 in
different environments and scenarios and uncover these problems.

On a more general note, there are ongoing tests of a pre-release of 2.9.1
that have been happening over the past month, so we continue to make
progress towards that release.  No major issues have been highlighted so
far.  Work on Riak 3.0 has slowed over the summer, but I hope we can pick
up the pace again and make further progress in September.

Regards

Martin

On Fri, 28 Jun 2019 at 09:34, Martin Sumner 
wrote:

> There is now a second update available for 2.9.0:
> https://github.com/basho/riak/tree/riak-2.9.0p2.
>
> This patch, like the patch before, resolves a memory management issue in
> leveled, which this time could be triggered by sending many large objects
> in a short period of time.  The underlying problem is described a bit
> further here https://github.com/martinsumner/leveled/issues/285, and is
> resolved by leveled working more sympathetically with the beam binary
> memory management.
>
> Switching to the patched version is not urgent unless you are using the
> leveled backend, and may send a large number of large objects in a burst.
>
> Updated packages are available (thanks to Nick Adams at TI Tokyo) -
> https://files.tiot.jp/riak/kv/2.9/2.9.0p2/
>
> Thanks again to the testing team at the NHS Spine project, Aaron Gibbon
> (BJSS) and Ramen Sen, who discovered the problem.  The issue was discovered
> in a handoff scenario where there were a tens of thousands of 2MB objects
> stored in a portion of the keyspace at the end of the handoff - which led
> to memory issues until either more PUTs were received (to force a persist
> to disk) or a restart occurred..
>
> Regards
>
>
> On Sat, 25 May 2019 at 09:35, Martin Sumner 
> wrote:
>
>> Unfortunately, Riak 2.9.0 was released with an issue whereby a race
>> condition in heavy-PUT scenarios (e.g. handoffs), could cause a leak of
>> file descriptors.
>>
>> The issue is described here -
>> https://github.com/basho/riak_kv/issues/1699, and the underlying issue
>> here - https://github.com/martinsumner/leveled/issues/278.
>>
>> There is a new patched version of the release available (2.9.0p1) at
>> https://github.com/basho/riak/tree/riak-2.9.0p1.  This should be used in
>> preference to the original release of 2.9.0.
>>
>> Updated packages are available (thanks to Nick Adams at TI Tokyo) -
>> https://files.tiot.jp/riak/kv/2.9/2.9.0p1/
>>
>> Thanks also to the testing team at the NHS Spine project, Aaron Gibbon
>> (BJSS) and Ramen Sen, who discovered the problem.
>>
>> Regards
>>
>> Martin
>>
>>
>>
>>
>>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak 2.9.0 - Update Available

[Martin Sumner]

Russell,

WRT naming.  As we'd already announced at CodeBEAM that 2.9.1 was pending
in September and would be adding some extra functionality (the automated
repl replacement), I didn't want to call the patched versions of 2.9.0 by
that name, as that might cause confusion.  The whole choosing 2.9 thing has
unnecessarily cramped naming up, which was my bad.  I've turned Riak
release numbering into a confusing mess.  So apologies for that.

Hopefully we can return to a more sane numbering system from 3.0.  Perhaps
someone else should choose!

Regards

On Fri, 28 Jun 2019 at 09:34, Martin Sumner 
wrote:

> There is now a second update available for 2.9.0:
> https://github.com/basho/riak/tree/riak-2.9.0p2.
>
> This patch, like the patch before, resolves a memory management issue in
> leveled, which this time could be triggered by sending many large objects
> in a short period of time.  The underlying problem is described a bit
> further here https://github.com/martinsumner/leveled/issues/285, and is
> resolved by leveled working more sympathetically with the beam binary
> memory management.
>
> Switching to the patched version is not urgent unless you are using the
> leveled backend, and may send a large number of large objects in a burst.
>
> Updated packages are available (thanks to Nick Adams at TI Tokyo) -
> https://files.tiot.jp/riak/kv/2.9/2.9.0p2/
>
> Thanks again to the testing team at the NHS Spine project, Aaron Gibbon
> (BJSS) and Ramen Sen, who discovered the problem.  The issue was discovered
> in a handoff scenario where there were a tens of thousands of 2MB objects
> stored in a portion of the keyspace at the end of the handoff - which led
> to memory issues until either more PUTs were received (to force a persist
> to disk) or a restart occurred..
>
> Regards
>
>
> On Sat, 25 May 2019 at 09:35, Martin Sumner 
> wrote:
>
>> Unfortunately, Riak 2.9.0 was released with an issue whereby a race
>> condition in heavy-PUT scenarios (e.g. handoffs), could cause a leak of
>> file descriptors.
>>
>> The issue is described here -
>> https://github.com/basho/riak_kv/issues/1699, and the underlying issue
>> here - https://github.com/martinsumner/leveled/issues/278.
>>
>> There is a new patched version of the release available (2.9.0p1) at
>> https://github.com/basho/riak/tree/riak-2.9.0p1.  This should be used in
>> preference to the original release of 2.9.0.
>>
>> Updated packages are available (thanks to Nick Adams at TI Tokyo) -
>> https://files.tiot.jp/riak/kv/2.9/2.9.0p1/
>>
>> Thanks also to the testing team at the NHS Spine project, Aaron Gibbon
>> (BJSS) and Ramen Sen, who discovered the problem.
>>
>> Regards
>>
>> Martin
>>
>>
>>
>>
>>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: riak-users Digest, Vol 118, Issue 1

[Tom Santero]

You want to do that here, Ben:
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
bottom of the page.

Have a nice weekend.

On Fri, Jun 28, 2019 at 12:30 PM Ben Stineman  wrote:

> Unsubscribe
>
> > On Jun 28, 2019, at 11:00, riak-users-requ...@lists.basho.com wrote:
> >
> > Send riak-users mailing list submissions to
> >riak-users@lists.basho.com
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> >http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> > or, via email, send a message with subject or body 'help' to
> >riak-users-requ...@lists.basho.com
> >
> > You can reach the person managing the list at
> >riak-users-ow...@lists.basho.com
> >
> > When replying, please edit your Subject line so it is more specific
> > than "Re: Contents of riak-users digest..."
> >
> >
> > Today's Topics:
> >
> >   1. Re: Riak 2.9.0 - Update Available (Martin Sumner)
> >   2. Re: Riak 2.9.0 - Update Available (Martin Sumner)
> >   3. Re: Riak 2.9.0 - Update Available (Russell Brown)
> >   4. Re: Riak 2.9.0 - Update Available (Bryan Hunt)
> >
> >
> > ------
> >
> > Message: 1
> > Date: Fri, 28 Jun 2019 09:34:58 +0100
> > From: Martin Sumner 
> > To: riak-users@lists.basho.com
> > Subject: Re: Riak 2.9.0 - Update Available
> > Message-ID:
> >
> > Content-Type: text/plain; charset="utf-8"
> >
> > There is now a second update available for 2.9.0:
> > https://github.com/basho/riak/tree/riak-2.9.0p2.
> >
> > This patch, like the patch before, resolves a memory management issue in
> > leveled, which this time could be triggered by sending many large objects
> > in a short period of time.  The underlying problem is described a bit
> > further here https://github.com/martinsumner/leveled/issues/285, and is
> > resolved by leveled working more sympathetically with the beam binary
> > memory management.
> >
> > Switching to the patched version is not urgent unless you are using the
> > leveled backend, and may send a large number of large objects in a burst.
> >
> > Updated packages are available (thanks to Nick Adams at TI Tokyo) -
> > https://files.tiot.jp/riak/kv/2.9/2.9.0p2/
> >
> > Thanks again to the testing team at the NHS Spine project, Aaron Gibbon
> > (BJSS) and Ramen Sen, who discovered the problem.  The issue was
> discovered
> > in a handoff scenario where there were a tens of thousands of 2MB objects
> > stored in a portion of the keyspace at the end of the handoff - which led
> > to memory issues until either more PUTs were received (to force a persist
> > to disk) or a restart occurred..
> >
> > Regards
> >
> >
> > On Sat, 25 May 2019 at 09:35, Martin Sumner  >
> > wrote:
> >
> >> Unfortunately, Riak 2.9.0 was released with an issue whereby a race
> >> condition in heavy-PUT scenarios (e.g. handoffs), could cause a leak of
> >> file descriptors.
> >>
> >> The issue is described here -
> https://github.com/basho/riak_kv/issues/1699,
> >> and the underlying issue here -
> >> https://github.com/martinsumner/leveled/issues/278.
> >>
> >> There is a new patched version of the release available (2.9.0p1) at
> >> https://github.com/basho/riak/tree/riak-2.9.0p1.  This should be used
> in
> >> preference to the original release of 2.9.0.
> >>
> >> Updated packages are available (thanks to Nick Adams at TI Tokyo) -
> >> https://files.tiot.jp/riak/kv/2.9/2.9.0p1/
> >>
> >> Thanks also to the testing team at the NHS Spine project, Aaron Gibbon
> >> (BJSS) and Ramen Sen, who discovered the problem.
> >>
> >> Regards
> >>
> >> Martin
> >>
> >>
> >>
> >>
> >>
> > -- next part --
> > An HTML attachment was scrubbed...
> > URL: <
> http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20190628/66b6528b/attachment-0001.html
> >
> >
> > --
> >
> > Message: 2
> > Date: Fri, 28 Jun 2019 10:24:28 +0100
> > From: Martin Sumner 
> > To: b h 
> > Cc: riak-users@lists.basho.com
> > Subject: Re: Riak 2.9.0 - Update Available
> > Message-ID:
> >
> > Content-Type: text/plain; charset="utf-8"
> >
> > Bryan,
> >
> > We saw that Riak was using much more memory t

Re: riak-users Digest, Vol 118, Issue 1

[Ben Stineman]

Unsubscribe 

> On Jun 28, 2019, at 11:00, riak-users-requ...@lists.basho.com wrote:
> 
> Send riak-users mailing list submissions to
>riak-users@lists.basho.com
> 
> To subscribe or unsubscribe via the World Wide Web, visit
>http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> or, via email, send a message with subject or body 'help' to
>riak-users-requ...@lists.basho.com
> 
> You can reach the person managing the list at
>riak-users-ow...@lists.basho.com
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of riak-users digest..."
> 
> 
> Today's Topics:
> 
>   1. Re: Riak 2.9.0 - Update Available (Martin Sumner)
>   2. Re: Riak 2.9.0 - Update Available (Martin Sumner)
>   3. Re: Riak 2.9.0 - Update Available (Russell Brown)
>   4. Re: Riak 2.9.0 - Update Available (Bryan Hunt)
> 
> 
> --
> 
> Message: 1
> Date: Fri, 28 Jun 2019 09:34:58 +0100
> From: Martin Sumner 
> To: riak-users@lists.basho.com
> Subject: Re: Riak 2.9.0 - Update Available
> Message-ID:
>
> Content-Type: text/plain; charset="utf-8"
> 
> There is now a second update available for 2.9.0:
> https://github.com/basho/riak/tree/riak-2.9.0p2.
> 
> This patch, like the patch before, resolves a memory management issue in
> leveled, which this time could be triggered by sending many large objects
> in a short period of time.  The underlying problem is described a bit
> further here https://github.com/martinsumner/leveled/issues/285, and is
> resolved by leveled working more sympathetically with the beam binary
> memory management.
> 
> Switching to the patched version is not urgent unless you are using the
> leveled backend, and may send a large number of large objects in a burst.
> 
> Updated packages are available (thanks to Nick Adams at TI Tokyo) -
> https://files.tiot.jp/riak/kv/2.9/2.9.0p2/
> 
> Thanks again to the testing team at the NHS Spine project, Aaron Gibbon
> (BJSS) and Ramen Sen, who discovered the problem.  The issue was discovered
> in a handoff scenario where there were a tens of thousands of 2MB objects
> stored in a portion of the keyspace at the end of the handoff - which led
> to memory issues until either more PUTs were received (to force a persist
> to disk) or a restart occurred..
> 
> Regards
> 
> 
> On Sat, 25 May 2019 at 09:35, Martin Sumner 
> wrote:
> 
>> Unfortunately, Riak 2.9.0 was released with an issue whereby a race
>> condition in heavy-PUT scenarios (e.g. handoffs), could cause a leak of
>> file descriptors.
>> 
>> The issue is described here - https://github.com/basho/riak_kv/issues/1699,
>> and the underlying issue here -
>> https://github.com/martinsumner/leveled/issues/278.
>> 
>> There is a new patched version of the release available (2.9.0p1) at
>> https://github.com/basho/riak/tree/riak-2.9.0p1.  This should be used in
>> preference to the original release of 2.9.0.
>> 
>> Updated packages are available (thanks to Nick Adams at TI Tokyo) -
>> https://files.tiot.jp/riak/kv/2.9/2.9.0p1/
>> 
>> Thanks also to the testing team at the NHS Spine project, Aaron Gibbon
>> (BJSS) and Ramen Sen, who discovered the problem.
>> 
>> Regards
>> 
>> Martin
>> 
>> 
>> 
>> 
>> 
> -- next part --
> An HTML attachment was scrubbed...
> URL: 
> <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20190628/66b6528b/attachment-0001.html>
> 
> --
> 
> Message: 2
> Date: Fri, 28 Jun 2019 10:24:28 +0100
> From: Martin Sumner 
> To: b h 
> Cc: riak-users@lists.basho.com
> Subject: Re: Riak 2.9.0 - Update Available
> Message-ID:
>
> Content-Type: text/plain; charset="utf-8"
> 
> Bryan,
> 
> We saw that Riak was using much more memory than was expected at the end of
> the handoffs.  Using `riak-admin top` we could see that this wasn't process
> memory, but binaries.  Firstly did some work via attach looping over
> processes and running GC to confirm that this wasn't a failure to collect
> garbage - the references to memory were real.  Then did a bit of work in
> attach writing some functions to analyse process_info/2 for each process
> (looking at binary and memory), and discovered that there were penciller
> processes that had lots of references to lots of large binaries (and this
> accounted for all the unexpected memory use), and where the penciller was
> the only process with a reference to the binary.  This made no sense
> initially

Re: Riak 2.9.0 - Update Available

[Bryan Hunt]

Top quality spelunking - always fun to read - thanks Martin !

> On 28 Jun 2019, at 10:24, Martin Sumner  wrote:
> 
> Bryan,
> 
> We saw that Riak was using much more memory than was expected at the end of 
> the handoffs.  Using `riak-admin top` we could see that this wasn't process 
> memory, but binaries.  Firstly did some work via attach looping over 
> processes and running GC to confirm that this wasn't a failure to collect 
> garbage - the references to memory were real.  Then did a bit of work in 
> attach writing some functions to analyse process_info/2 for each process 
> (looking at binary and memory), and discovered that there were penciller 
> processes that had lots of references to lots of large binaries (and this 
> accounted for all the unexpected memory use), and where the penciller was the 
> only process with a reference to the binary.  This made no sense initially as 
> the penciller should only have small binaries (metadata).  Then looked at the 
> running state of the penciller processes and could see no large binaries in 
> the state, but could see that a lot of the active keys in the penciller were 
> keys that were known to have large object values (but small amounts of 
> metadata) - and that the size of the object values were the same as the size 
> of the binary references found on the penciller process via process_info/2.. 
> 
> I then recalled the first part of this: 
> https://dieswaytoofast.blogspot.com/2012/12/erlang-binaries-and-garbage-collection.html
>  
> .
>   It was obvious that in extracting the metadata the beam was naturally 
> retaining a reference to the whole binary, as long as the sub-binary was 
> retained by the a process (the Penciller).  Forcing a binary copy resolved 
> this referencing issue.  It was nice that the same tools used to detect the 
> issue, made it quite easy to write a test to confirm resolution - 
> https://github.com/martinsumner/leveled/blob/master/test/end_to_end/riak_SUITE.erl#L1214-L1239
>  
> .
> 
> The memory leak section of Fred Herbert's http://www.erlang-in-anger.com/ 
>  is great reading for helping with these 
> types of issues. 
> 
> Thanks
> 
> Martin
> 
> 
> On Fri, 28 Jun 2019 at 09:46, b h  > wrote:
> Nice work - I've read issue / PR - how did you discover / track it down - 
> tools or just reading the code ? 
> 
> On Fri, 28 Jun 2019 at 09:35, Martin Sumner  > wrote:
> There is now a second update available for 2.9.0: 
> https://github.com/basho/riak/tree/riak-2.9.0p2 
> .
> 
> This patch, like the patch before, resolves a memory management issue in 
> leveled, which this time could be triggered by sending many large objects in 
> a short period of time.  The underlying problem is described a bit further 
> here https://github.com/martinsumner/leveled/issues/285 
> , and is resolved by 
> leveled working more sympathetically with the beam binary memory management. 
> 
> Switching to the patched version is not urgent unless you are using the 
> leveled backend, and may send a large number of large objects in a burst.  
> 
> Updated packages are available (thanks to Nick Adams at TI Tokyo) - 
> https://files.tiot.jp/riak/kv/2.9/2.9.0p2/ 
> 
> 
> Thanks again to the testing team at the NHS Spine project, Aaron Gibbon 
> (BJSS) and Ramen Sen, who discovered the problem.  The issue was discovered 
> in a handoff scenario where there were a tens of thousands of 2MB objects 
> stored in a portion of the keyspace at the end of the handoff - which led to 
> memory issues until either more PUTs were received (to force a persist to 
> disk) or a restart occurred..
> 
> Regards
> 
> 
> On Sat, 25 May 2019 at 09:35, Martin Sumner  > wrote:
> Unfortunately, Riak 2.9.0 was released with an issue whereby a race condition 
> in heavy-PUT scenarios (e.g. handoffs), could cause a leak of file 
> descriptors.
> 
> The issue is described here - https://github.com/basho/riak_kv/issues/1699 
> , and the underlying issue here 
> - https://github.com/martinsumner/leveled/issues/278 
> .
> 
> There is a new patched version of the release available (2.9.0p1) at 
> https://github.com/basho/riak/tree/riak-2.9.0p1 
> .  This should be used in 
> preference to the original release of 2.9.0.
> 
> Updated packages are available (thanks to Nick Adams at TI Tokyo) - 
> https://files.tiot.jp/riak/kv/2.9/2.9.0p1/ 
> 

Re: Riak 2.9.0 - Update Available

[Russell Brown via riak-users]

Good job on finding and fixing so fast.

I have to ask. What's with the naming scheme? Why not 2.9.2 instead of 
2.9.0p2?


Cheers

Russell

On 28/06/2019 10:24, Martin Sumner wrote:

Bryan,

We saw that Riak was using much more memory than was expected at the 
end of the handoffs.  Using `riak-admin top` we could see that this 
wasn't process memory, but binaries. Firstly did some work via attach 
looping over processes and running GC to confirm that this wasn't a 
failure to collect garbage - the references to memory were real.  Then 
did a bit of work in attach writing some functions to analyse 
process_info/2 for each process (looking at binary and memory), and 
discovered that there were penciller processes that had lots of 
references to lots of large binaries (and this accounted for all the 
unexpected memory use), and where the penciller was the only process 
with a reference to the binary.  This made no sense initially as the 
penciller should only have small binaries (metadata).  Then looked at 
the running state of the penciller processes and could see no large 
binaries in the state, but could see that a lot of the active keys in 
the penciller were keys that were known to have large object values 
(but small amounts of metadata) - and that the size of the object 
values were the same as the size of the binary references found on the 
penciller process via process_info/2..


I then recalled the first part of this: 
https://dieswaytoofast.blogspot.com/2012/12/erlang-binaries-and-garbage-collection.html. 
It was obvious that in extracting the metadata the beam was naturally 
retaining a reference to the whole binary, as long as the sub-binary 
was retained by the a process (the Penciller).  Forcing a binary copy 
resolved this referencing issue.  It was nice that the same tools used 
to detect the issue, made it quite easy to write a test to confirm 
resolution - 
https://github.com/martinsumner/leveled/blob/master/test/end_to_end/riak_SUITE.erl#L1214-L1239.


The memory leak section of Fred Herbert's 
http://www.erlang-in-anger.com/ is great reading for helping with 
these types of issues.


Thanks

Martin


On Fri, 28 Jun 2019 at 09:46, b h > wrote:


Nice work - I've read issue / PR - how did you discover / track it
down - tools or just reading the code ?

On Fri, 28 Jun 2019 at 09:35, Martin Sumner
mailto:martin.sum...@adaptip.co.uk>>
wrote:

There is now a second update available for 2.9.0:
https://github.com/basho/riak/tree/riak-2.9.0p2.

This patch, like the patch before, resolves a memory
management issue in leveled, which this time could be
triggered by sending many large objects in a short period of
time.  The underlying problem is described a bit further here
https://github.com/martinsumner/leveled/issues/285, and is
resolved by leveled working more sympathetically with the beam
binary memory management.

Switching to the patched version is not urgent unless you are
using the leveled backend, and may send a large number of
large objects in a burst.

Updated packages are available (thanks to Nick Adams at TI
Tokyo) - https://files.tiot.jp/riak/kv/2.9/2.9.0p2/

Thanks again to the testing team at the NHS Spine project,
Aaron Gibbon (BJSS) and Ramen Sen, who discovered the
problem.  The issue was discovered in a handoff scenario where
there were a tens of thousands of 2MB objects stored in a
portion of the keyspace at the end of the handoff - which led
to memory issues until either more PUTs were received (to
force a persist to disk) or a restart occurred..

Regards


On Sat, 25 May 2019 at 09:35, Martin Sumner
mailto:martin.sum...@adaptip.co.uk>> wrote:

Unfortunately, Riak 2.9.0 was released with an issue
whereby a race condition in heavy-PUT scenarios (e.g.
handoffs), could cause a leak of file descriptors.

The issue is described here -
https://github.com/basho/riak_kv/issues/1699, and the
underlying issue here -
https://github.com/martinsumner/leveled/issues/278.

There is a new patched version of the release available
(2.9.0p1) at
https://github.com/basho/riak/tree/riak-2.9.0p1. This
should be used in preference to the original release of 2.9.0.

Updated packages are available (thanks to Nick Adams at TI
Tokyo) - https://files.tiot.jp/riak/kv/2.9/2.9.0p1/

Thanks also to the testing team at the NHS Spine project,
Aaron Gibbon (BJSS) and Ramen Sen, who discovered the problem.

Regards

Martin




___
riak-users mailing list
riak-users@lists.basho.com 

Re: Riak 2.9.0 - Update Available

[Martin Sumner]

Bryan,

We saw that Riak was using much more memory than was expected at the end of
the handoffs.  Using `riak-admin top` we could see that this wasn't process
memory, but binaries.  Firstly did some work via attach looping over
processes and running GC to confirm that this wasn't a failure to collect
garbage - the references to memory were real.  Then did a bit of work in
attach writing some functions to analyse process_info/2 for each process
(looking at binary and memory), and discovered that there were penciller
processes that had lots of references to lots of large binaries (and this
accounted for all the unexpected memory use), and where the penciller was
the only process with a reference to the binary.  This made no sense
initially as the penciller should only have small binaries (metadata).
Then looked at the running state of the penciller processes and could see
no large binaries in the state, but could see that a lot of the active keys
in the penciller were keys that were known to have large object values (but
small amounts of metadata) - and that the size of the object values were
the same as the size of the binary references found on the penciller
process via process_info/2..

I then recalled the first part of this:
https://dieswaytoofast.blogspot.com/2012/12/erlang-binaries-and-garbage-collection.html.
It was obvious that in extracting the metadata the beam was naturally
retaining a reference to the whole binary, as long as the sub-binary was
retained by the a process (the Penciller).  Forcing a binary copy resolved
this referencing issue.  It was nice that the same tools used to detect the
issue, made it quite easy to write a test to confirm resolution -
https://github.com/martinsumner/leveled/blob/master/test/end_to_end/riak_SUITE.erl#L1214-L1239
.

The memory leak section of Fred Herbert's http://www.erlang-in-anger.com/ is
great reading for helping with these types of issues.

Thanks

Martin


On Fri, 28 Jun 2019 at 09:46, b h  wrote:

> Nice work - I've read issue / PR - how did you discover / track it down -
> tools or just reading the code ?
>
> On Fri, 28 Jun 2019 at 09:35, Martin Sumner 
> wrote:
>
>> There is now a second update available for 2.9.0:
>> https://github.com/basho/riak/tree/riak-2.9.0p2.
>>
>> This patch, like the patch before, resolves a memory management issue in
>> leveled, which this time could be triggered by sending many large objects
>> in a short period of time.  The underlying problem is described a bit
>> further here https://github.com/martinsumner/leveled/issues/285, and is
>> resolved by leveled working more sympathetically with the beam binary
>> memory management.
>>
>> Switching to the patched version is not urgent unless you are using the
>> leveled backend, and may send a large number of large objects in a burst.
>>
>> Updated packages are available (thanks to Nick Adams at TI Tokyo) -
>> https://files.tiot.jp/riak/kv/2.9/2.9.0p2/
>>
>> Thanks again to the testing team at the NHS Spine project, Aaron Gibbon
>> (BJSS) and Ramen Sen, who discovered the problem.  The issue was discovered
>> in a handoff scenario where there were a tens of thousands of 2MB objects
>> stored in a portion of the keyspace at the end of the handoff - which led
>> to memory issues until either more PUTs were received (to force a persist
>> to disk) or a restart occurred..
>>
>> Regards
>>
>>
>> On Sat, 25 May 2019 at 09:35, Martin Sumner 
>> wrote:
>>
>>> Unfortunately, Riak 2.9.0 was released with an issue whereby a race
>>> condition in heavy-PUT scenarios (e.g. handoffs), could cause a leak of
>>> file descriptors.
>>>
>>> The issue is described here -
>>> https://github.com/basho/riak_kv/issues/1699, and the underlying issue
>>> here - https://github.com/martinsumner/leveled/issues/278.
>>>
>>> There is a new patched version of the release available (2.9.0p1) at
>>> https://github.com/basho/riak/tree/riak-2.9.0p1.  This should be used
>>> in preference to the original release of 2.9.0.
>>>
>>> Updated packages are available (thanks to Nick Adams at TI Tokyo) -
>>> https://files.tiot.jp/riak/kv/2.9/2.9.0p1/
>>>
>>> Thanks also to the testing team at the NHS Spine project, Aaron Gibbon
>>> (BJSS) and Ramen Sen, who discovered the problem.
>>>
>>> Regards
>>>
>>> Martin
>>>
>>>
>>>
>>>
>>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak 2.9.0 - Update Available

[Martin Sumner]

There is now a second update available for 2.9.0:
https://github.com/basho/riak/tree/riak-2.9.0p2.

This patch, like the patch before, resolves a memory management issue in
leveled, which this time could be triggered by sending many large objects
in a short period of time.  The underlying problem is described a bit
further here https://github.com/martinsumner/leveled/issues/285, and is
resolved by leveled working more sympathetically with the beam binary
memory management.

Switching to the patched version is not urgent unless you are using the
leveled backend, and may send a large number of large objects in a burst.

Updated packages are available (thanks to Nick Adams at TI Tokyo) -
https://files.tiot.jp/riak/kv/2.9/2.9.0p2/

Thanks again to the testing team at the NHS Spine project, Aaron Gibbon
(BJSS) and Ramen Sen, who discovered the problem.  The issue was discovered
in a handoff scenario where there were a tens of thousands of 2MB objects
stored in a portion of the keyspace at the end of the handoff - which led
to memory issues until either more PUTs were received (to force a persist
to disk) or a restart occurred..

Regards


On Sat, 25 May 2019 at 09:35, Martin Sumner 
wrote:

> Unfortunately, Riak 2.9.0 was released with an issue whereby a race
> condition in heavy-PUT scenarios (e.g. handoffs), could cause a leak of
> file descriptors.
>
> The issue is described here - https://github.com/basho/riak_kv/issues/1699,
> and the underlying issue here -
> https://github.com/martinsumner/leveled/issues/278.
>
> There is a new patched version of the release available (2.9.0p1) at
> https://github.com/basho/riak/tree/riak-2.9.0p1.  This should be used in
> preference to the original release of 2.9.0.
>
> Updated packages are available (thanks to Nick Adams at TI Tokyo) -
> https://files.tiot.jp/riak/kv/2.9/2.9.0p1/
>
> Thanks also to the testing team at the NHS Spine project, Aaron Gibbon
> (BJSS) and Ramen Sen, who discovered the problem.
>
> Regards
>
> Martin
>
>
>
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak KV 2.9.0 released at Code BEAM STO 2019

[Tom Santero]

Fantastic news! Nice work, everyone.

Two questions:

1. Why the version jump all the way to 2.9? Is it because it's closer to
3.0? :)
2. Any update on getting off of R16B02? Is there an ETA to get to OTP 20+?

On Fri, May 17, 2019 at 11:05 AM Nicholas Adams 
wrote:

> Dear All,
>
> I am extremely pleased to announce with Martin Sumner at Code BEAM STO
> 2019 that Riak KV 2.9.0 has officially been released!
>
>
>
> GitHub
>
> https://github.com/basho/riak/tree/riak-2.9.0
>
>
>
> Packages
>
> https://files.tiot.jp/riak/kv/2.9/2.9.0/
>
>
>
> Many thanks to everybody who contributed to this great achievement.
>
>
>
> As always any issues or questions, please post to GitHub, this mailing
> list or the Slack channel.
>
>
>
> Best regards,
>
>
>
> Nicholas Adams
>
> Director
>
> TI Tokyo
>
> https://www.tiot.jp/en/
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak KV 2.9.0 released at Code BEAM STO 2019

[DeadZen]

w00t! You guys rock

On Fri, May 17, 2019 at 12:27 PM Russell Brown via riak-users <
riak-users@lists.basho.com> wrote:

> Congratulations all! Great news!
>
> On 17/05/2019 16:04, Nicholas Adams wrote:
> >
> > Dear All,
> >
> > I am extremely pleased to announce with Martin Sumner at Code BEAM STO
> > 2019 that Riak KV 2.9.0 has officially been released!
> >
> > GitHub
> >
> > https://github.com/basho/riak/tree/riak-2.9.0
> >
> > Packages
> >
> > https://files.tiot.jp/riak/kv/2.9/2.9.0/
> >
> > Many thanks to everybody who contributed to this great achievement.
> >
> > As always any issues or questions, please post to GitHub, this mailing
> > list or the Slack channel.
> >
> > Best regards,
> >
> > Nicholas Adams
> >
> > Director
> >
> > TI Tokyo
> >
> > https://www.tiot.jp/en/
> >
> >
> > ___
> > riak-users mailing list
> > riak-users@lists.basho.com
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
-- 
/* Sincerely
-- */
Pedram N. - CTO

It is a mistake to look too far ahead. Only one link of the chain of
destiny can be handled at a time. - Winston Churchill
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak KV 2.9.0 released at Code BEAM STO 2019

[Russell Brown via riak-users]

Congratulations all! Great news!

On 17/05/2019 16:04, Nicholas Adams wrote:


Dear All,

I am extremely pleased to announce with Martin Sumner at Code BEAM STO 
2019 that Riak KV 2.9.0 has officially been released!


GitHub

https://github.com/basho/riak/tree/riak-2.9.0

Packages

https://files.tiot.jp/riak/kv/2.9/2.9.0/

Many thanks to everybody who contributed to this great achievement.

As always any issues or questions, please post to GitHub, this mailing 
list or the Slack channel.


Best regards,

Nicholas Adams

Director

TI Tokyo

https://www.tiot.jp/en/


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

RE: mounting leveldb with noatime problem

[Nicholas Adams]

Hi Farhad,
Try this https://www.tiot.jp/riak-docs/riak/kv/2.2.1/using/performance/#mounts 
for a sample command. Alternately, if you are looking to set it permanently via 
fstab, look at 
https://www.tiot.jp/riak-docs/riak/kv/2.0.9/setup/planning/backend/bitcask/#tips-tricks
 It’s for Bitcask but the same optimization is appropriate for LevelDB.

Best regards,

Nicholas
From: riak-users  On Behalf Of Farhad
Sent: 26 April 2019 22:55
To: riak-users@lists.basho.com
Subject: mounting leveldb with noatime problem


Dears,

I am newbie both in Riak and ubuntu, when i type "sudo riak-admin diag" I get 
this message:
"Data directory /var/lib/riak/leveldb is not mounted with 'noatime'.
 Please remount its disk with the 'noatime' flag to improve performance."

I only have one partition on my system I am using Ubuntu 18.04.2.

Can you give me the required command to handle this warning.


--
Yours sincerely
Farhad
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

RE: Having trouble setting up riak-cs using s3cmd to test

[Nicholas Adams]

Hi Greg,
I’m guessing that something isn’t running. Try the following (without the back 
checks) and see if you get “pong” for each one:
`riak ping`
`riak-cs ping`
`stanchion ping`

You may or may not need `sudo` to execute the above depending on the user you 
are using.

Incidentally, your s3cfg file looks incredibly short. Here is my standard one:

access_key = admin.key
bucket_location = US
cloudfront_host = cloudfront.amazonaws.com
cloudfront_resource = /2010-07-15/distribution
default_mime_type = binary/octet-stream
delete_removed = False
dry_run = False
encoding = UTF-8
encrypt = False
follow_symlinks = False
force = False
get_continue = False
gpg_command = /usr/bin/gpg
gpg_decrypt = %(gpg_command)s -d --verbose 
--no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o 
%(output_file)s %(input_file)s
gpg_encrypt = %(gpg_command)s -c --verbose 
--no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o 
%(output_file)s %(input_file)s
gpg_passphrase =
guess_mime_type = True
host_base = s3.amazonaws.com
host_bucket = %(bucket)s.s3.amazonaws.com
human_readable_sizes = False
list_md5 = False
log_target_prefix =
preserve_attrs = True
progress_meter = True
proxy_host = 127.0.0.1
proxy_port = 18080
recursive = False
recv_chunk = 4096
reduced_redundancy = False
secret_key = admin.secret
send_chunk = 4096
simpledb_host = sdb.amazonaws.com
skip_existing = False
socket_timeout = 10
urlencoding_mode = normal
use_https = False
verbosity = WARNING
signature_v2 = True

Of course, the admin.key and admin.secret are substituted out for appropriate 
values.

Finally, what version of s3cmd are you using? If 1.x then you should be fine. 
2.x is known to have a few difficulties with some functions but usually works. 
We are hoping to rectify that in Riak CS 2.2.0.

Hope this helps,

Nicholas

From: riak-users  On Behalf Of Greg Zoller
Sent: 23 April 2019 23:43
To: riak-users@lists.basho.com
Subject: Having trouble setting up riak-cs using s3cmd to test

Hello,

I'm following the instructions for the Docker-ized riak-cs.  The Docker 
container runs and (appears) healthy.

I am now testing the install with s3cmd as described in the instructions.  I 
installed s3cmd and am using this .s3cfg file:  (the 2 keys I got from the 
running container's log output as instructed)

[default]
access_key = Q9NKDXZJGAD2CZF3T4B7
host_base = s3.amazonaws.dev
host_bucket = %(bucket)s.s3.amazonaws.dev
proxy_host = localhost
proxy_port = 8080
secret_key = 2aR_IthHHYQpFGx8xP0z46ZyoQUsfzvSsTl4bA==
signature_v2 = True

When I run a simple command like "s3cmd ls" I get this output:

WARNING: Retrying failed request: / (No status line received - the server has 
closed the connection)
WARNING: Waiting 3 sec...
WARNING: Retrying failed request: / (No status line received - the server has 
closed the connection)
WARNING: Waiting 6 sec...
^CSee ya!

Any ideas what I'm doing wrong?
Thanks,
Greg
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: riak 2.2.3, allow_mult=false but still siblings

[Russell Brown via riak-users]

That's actually very useful, I hope you are OK if that I re-add back in 
the list so that this thread can be indexed/used in future.



On 15/04/2019 15:14, ジョハンガル wrote:
After -X DELETE types/???/buckets/???/props the properties reverted to 
the bucket type properties!

​
Thank you a lot!
​

-Original Message-
*From:* "Russell Brown"
*To:* "ジョハンガル";
*Cc:*
*Sent:* 2019-04-15 (月) 19:10:24 (GMT+09:00)
*Subject:* Re: riak 2.2.3, allow_mult=false but still siblings

I'm sorry, I don't know, I've not worked much on that code.


Here is how I assume it works


1, create a bucket type T (allow_mult=true by default)

2, create a bucket B of type T (allow_mult "inherited" from T)

3, change property allow_mult of B to false (it is false on B only, not T)


Default buckets have a default set of properties, allow_mult=false

Typed buckets have a default set of properties, allow_mult=true


buckets (named) inherit their Types properties, unless you change the
bucket properties.


On 15/04/2019 09:41, ジョハンガル wrote:
> So if I understand well
> There are basic bucket properties who are ignored
I don't understand "who are ignored" bucket properties are not ignored
> /buckets/???/props
> ​
> Then there are bucket types
> riak-admin bucket-type ...​
> ​
> Then there are bucket type properties under that type
> /types/???/buckets/???/props
> ​
> Are /types/??A/buckets/???/props and /types/??B/buckets/???/props
> different

Only if you changed the properties of A or B on creation of the type.
And further you can change the properties of buckets that or type A or B
(unless the types has immutable properties (like the allow_mult property
of a datatyped bucket))


> ​
> Just to make sure, I am quite confused.
> ​
> If I run -X DELETE /types/???/buckets/???/props would it revert it to
> the default bucket type properties??

I don't think so, no, I don't think you can delete bucket props that
way. You need to set the properties you want


If it helps, I'm confused too. What do the docs say on the matter?


Cheers


Russell

> ​
>
> -Original Message-
> *From:* "Russell Brown"
> *To:* "ジョハンガル";
> *Cc:*
> *Sent:* 2019-04-15 (月) 17:14:34 (GMT+09:00)
> *Subject:* Re: riak 2.2.3, allow_mult=false but still siblings
>
> That bucket has allow_mult=true, so the siblings are expected. How the
> bucket props managed to be changed from the bucket-type defaults is
> worth investigating though.
>
>
> On 15/04/2019 08:49, ジョハンガル wrote:
> > Sorry for the late answer!
> > ​
> > ​​
> > ​
> > Is it a case of bucket props overriding the type?
> > We deleted all the bucket props recently. (curl -X DELETE  /props)
> >
> > -Original Message-
> > *From:* "Russell Brown"
> > *To:* "ジョハンガル";
> > *Cc:*
> > *Sent:* 2019-04-13 (土) 05:05:33 (GMT+09:00)
> > *Subject:* Re: riak 2.2.3, allow_mult=false but still siblings
> >
> > Can I see the bucket properties for the bucket in question, please?
> > Buckets can override their type's properties, iirc
> >
> >
> > Cheers
> >
> > Russell
> >
> >
> > On 12/04/2019 13:36, ジョハンガル wrote:
> > > for the bucket type definition:
> > > ​
> > > ​
> > > For the headers
> > > ​
> > > ​
> > > These buckets formally allowed siblings (more of a default thing 
than

> > > anything).
> > > Following sibling explosion problems we modified all buckets that
> > > received updates from a single source to not allow siblings anymore.
> > > During some time we had bucket properties and types used at the same
> > > type, following the previously mentioned (property broadcast bug)
> > > repeatedly bringing our machines down, we identified the 
problem, made

> > > sure everything run with >2.0 clients, reset (deleted) all bucket
> > > properties and made sure to use types. Then the cluster became 
stable.

> > > Then while monitoring I noticed these siblings that shouldn't be.
> > > ​
> > > The entire content of these buckets is batch regenerated multiple
> > > times by minute (~1 to 10). There is very little total content 
(a few

> > > megabytes in that bucket) and the machines used are ridiculously
> > > overprovisionned (many cores 64gb ram machines).
> > > ​
> > >
> > > -Original Message-
> > > *From:* "Russell Brown"
> > > *To:* "ジョハンガル";
> > > ;
> > > *Cc:*
> > > *Sent:* 2019-04-12 (金) 20:47:56 (GMT+09:00)
> > > *Subject:* Re: riak 2.2.3, allow_mult=false but still siblings
> > >
>

Re: riak 2.2.3, allow_mult=false but still siblings

[Russell Brown via riak-users]

Can you let us see the bucket type definition, please?


Can you show me the headers from the curl command that returns siblings, 
please?



I want to say that what you are seeing is unpossible (from what I 
remember of the code.) But I don't remember the 2.2.3 release process, 
and I'd like to see some more evidence before I look at the code again.



I wonder if you remember the history of the bucket/bucket type in 
question. Has it had changes to allow_mult etc?



Cheers

Russell

On 12/04/2019 12:29, ジョハンガル wrote:

Hello,
​
I would be thankful is somebody could help me with some weird 
abnormalities.

​
curl https://~~~/types/THE-TYPE/bucket/~~~/keys/~~~ returns siblings
​
However the corresponding type has "allow_mult" set to false...
​​
The phenomenon appear with both 
"last_write_wins=true;dvv_enabled=false" and 
"last_write_wins=false;dvv_enabled=true".
The backend is leveldb (in a multi configuration), secondary indices 
are used.

​
The cluster is an old cluster (pre 2.0) that received rolling updates 
to 2.0 (remove node, reinstall riak with new config, read node).

Currently 2.2.3
We used to hit that problem a lot until we found out about it and 
fixed the problem https://github.com/basho/yokozuna/issues/389

I don't if it has any kind of relevancy though.
​
Does somebody have some kind of hint? Or diagnostic command I might 
run? Am I missing something obvious?

​
Best regards,
​


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

RE: [ANN] Riak 2.9.0 - Release Candidate 7 Available

[Nicholas Adams]

Dear All,
Just following on from Martin to say that package builds are available from 
https://files.tiot.jp/riak/kv/2.9/2.9.0rc7/
Please test well and report any issues on GitHub.

Best regards,

Nicholas

From: riak-users  On Behalf Of Martin Sumner
Sent: 03 April 2019 04:13
To: riak-users@lists.basho.com
Subject: [ANN] Riak 2.9.0 - Release Candidate 7 Available


Release Candidate 7 for 2.9.0 is now available on github.  It is available

to build from source here - https://github.com/basho/riak/tree/riak-2.9.0rc7.

All usual caveats apply.  All usual promises that this will be the last release 
candidate are duly made.

Regards

Martin


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

RE: [ANN] Riak 2.9.0 - Release Candidate 6 Available

[Nicholas Adams]

Dear All,
Just a brief update to say that rc6 packages are available for most of the 
usual operating systems at https://files.tiot.jp/riak/kv/2.9/2.9.0rc6/

I will restate that all the usual caveats apply. This is a release candidate 
intended for testing and non-production environments only. Please report any 
bugs found via GitHub https://github.com/basho/riak/tree/riak-2.9.0rc6

Best regards,

Nicholas

From: riak-users  On Behalf Of Martin Sumner
Sent: 19 March 2019 07:03
To: riak-users@lists.basho.com
Subject: [ANN] Riak 2.9.0 - Release Candidate 6 Available


Release Candidate 6 for 2.9.0 is now available on github.  It is available

to build from source here - https://github.com/basho/riak/tree/riak-2.9.0rc6.



Apologies for the churn of release candidates on 2.9.0.  As part of the move to 
2.9 there has been a big update to the mochiweb/webmachine code, getting that 
more in line with current upstream - and this has thrown up some issues.



All usual caveats arise.  This is a release intended for testing in

non-production environments.



Apologies for the ongoing delays getting this release through the test

process.  I'm hopeful this will be the last candidate before we release 2.9.0.



Regards



Martin
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: [ANN] Riak 2.9.0 - Release Candidate 2 Available

[Zsolt Laky]

Dear All,

Does anyone have experience on building RIAK on Power9 architecture? We want to 
have 2.2.6 built but it fails at compiling leveldb.

I will appreciate a pointer to get in touch with.

Thanks in advance and kind regards,
Zsolt


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

RE: [ANN] Riak 2.9.0 - Release Candidate 5 Available

[Nicholas Adams]

Dear All,
Just following up to say that initial rc5 package builds for Rhel/CentOS 6/7 
and Ubuntu Bionic are now available from 
https://files.tiot.jp/riak/kv/2.9/2.9.0rc5/
Packages for other operating systems will be available at the same link by the 
end of the week.

Best regards,

Nicholas

From: riak-users  On Behalf Of Martin Sumner
Sent: 05 March 2019 01:17
To: riak-users@lists.basho.com
Subject: [ANN] Riak 2.9.0 - Release Candidate 5 Available

Release Candidate 5 for 2.9.0 is now available on github.  It is available to 
build from source here - https://github.com/basho/riak/tree/riak-2.9.0rc5.

Apologies for the churn of release candidates on 2.9.0.  This is as a result of 
issues discovered in the leveled refactoring in RC2, having side-effects during 
big handoff events with high transfer limits.

All usual caveats arise.  This is a release intended for testing in 
non-production environments.

Apologies for the ongoing delays getting this release through the test process.

Regards

Martin

P.S. A reminder that work on Release 3.0 is progressing, with a workshop being 
held this week to try and accelerate progress.  Follow the #gothmeet2019 
channel on Riak slack for updates
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

RE: Leveled and auto-expiry (ジョハンガル)

[Martin Sumner]

Johan,

Leveled does have TTL support for objects, but this is not added yet into
riak_kv.

I've raised an issue to look at this -
https://github.com/basho/riak_kv/issues/1684.

As mentioned in the issue, there is some thinking required first to make
sure that expiry works well with anti-entropy.

Things are a bit hectic for the next couple of weeks, but it is something I
want to tackle.  Without the AAE problem it is easy to do, I'm not sure yet
how big a problem it is going to be to tackle the AAE problem - but I have
some ideas.

Regards

Martin
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Wants/Choose claim functions

[Guido Medina]

Hi Luke,

That old e-mail is what I was looking for, many thanks !!!
I had already found the FAQ

Guido.

On 19/02/2019 14:35, Luke Bakken wrote:

Hi Guido,

Searching google for "riak wants_claim_fun" turns up this document.
Please see the section "Q: A node left the cluster before handing off
all data. How can I resolve this?"

https://docs.basho.com/riak/kv/2.2.3/developing/faq/

This may be helpful as well -

http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-November/038435.html

On Tue, Feb 19, 2019 at 3:22 AM Guido Medina  wrote:

Hi,

Can someone please point me out to the guide explaining how to change wants and 
choose claim functions for Riak distribution % distribution?

We would like to set these permanently and trigger a cluster redistribution, I 
just can't find that documentation anymore, I was able to find this but I can't 
remember how to use it:

{wants_claim_fun, {riak_core_claim, wants_claim_v3}},
{choose_claim_fun, {riak_core_claim, choose_claim_v3}}


Kind regards,
Guido.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Wants/Choose claim functions

[Luke Bakken]

Hi Guido,

Searching google for "riak wants_claim_fun" turns up this document.
Please see the section "Q: A node left the cluster before handing off
all data. How can I resolve this?"

https://docs.basho.com/riak/kv/2.2.3/developing/faq/

This may be helpful as well -

http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-November/038435.html

On Tue, Feb 19, 2019 at 3:22 AM Guido Medina  wrote:
>
> Hi,
>
> Can someone please point me out to the guide explaining how to change wants 
> and choose claim functions for Riak distribution % distribution?
>
> We would like to set these permanently and trigger a cluster redistribution, 
> I just can't find that documentation anymore, I was able to find this but I 
> can't remember how to use it:
>
> {wants_claim_fun, {riak_core_claim, wants_claim_v3}},
> {choose_claim_fun, {riak_core_claim, choose_claim_v3}}
>
>
> Kind regards,
> Guido.
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

RE: [ANN] Riak 2.9.0 - Release Candidate 2 Available

[Nicholas Adams]

Dear All,
Just a follow up to Martin’s email to say that packages are now available at 
https://files.tiot.jp/riak/kv/2.9/2.9.0rc2/ and that more will be coming soon.

Please be reminded that this is a release candidate and is probably not 
suitable for use in production. However, please use or abuse it in as many ways 
as possible in your test environments and report any issues on 
https://github.com/basho/riak/issues accordingly.

Best regards,

Nicholas

From: riak-users  On Behalf Of Martin Sumner
Sent: 15 February 2019 21:10
To: riak-users@lists.basho.com
Subject: [ANN] Riak 2.9.0 - Release Candidate 2 Available

Release Candidate 2 for 2.9.0 is now available on github.  It is available to 
build from source here - https://github.com/basho/riak/tree/riak-2.9.0rc2.

The changes from Release Candidate 1 are described here - 
https://github.com/basho/riak/blob/develop-2.9/doc/Release%202.9%20Series%20-%20Overview.md#release-292-rc2.

The changes are mainly in two parts:

- Improvements and fixes to the leveled backend following further 
property-based testing;
- An uplift of mochiweb (the HTTP handler used by Riak), to take in general bug 
fixes, but also to allow for the receive buffer to be configurable to 
workaround an issue with receiving large HTTP headers 
(https://github.com/basho/riak_api/issues/123).

Thanks to all those who contributed to the release.   My current expectation is 
that the release will be finalised before the end of February.  Apologies again 
for the delay.

Regards

Martin (@masleeds)
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: [ANN] Riak 2.9.0 - Release Candidate 1 Available

[Bryan Hunt]

Yeah - it used to be the case that it would be typical to rolling upgrade 
replace, leave, individual cluster nodes - but that was the era when :

a). It cost $5000 per node for a riak_repl (enterprise) license
b) Most folk ran on physical hardware 

Now with most deployments (Bet366/NHS being notable exceptions) running on 
cloud, and no license fee to pay - it’s a no brainer to go the route of create 
new cluster, replicate, verify, switch over, archive old data.

B

> On 1 Feb 2019, at 14:08, Guido Medina  wrote:
> 
> We could probably go the "replace" way, but maybe we take this as a good 
> opportunity to increase our ring size, it is still something we are 
> considering.
> 
> Resizing the cluster while operating daily is no joke, too much data and Riak 
> becomes extremely slow when we have to add a new node, so it would either go 
> the "replace" way or replicate and switch to the new cluster.
> 
> Thanks for all the answers ;-)
> Guido.
> 
> On 01/02/2019 13:50, Nicholas Adams wrote:
>> If he has an extra node he can add then a replace would be much better. I 
>> provided this example under the assumption that he had no additional 
>> resources to play with.
>>  
>> Nicholas
>>  
>> From: riak-users  
>> <mailto:riak-users-boun...@lists.basho.com> On Behalf Of Fred Dushin
>> Sent: 01 February 2019 22:47
>> To: riak-users@lists.basho.com <mailto:riak-users@lists.basho.com>
>> Subject: Re: [ANN] Riak 2.9.0 - Release Candidate 1 Available
>>  
>> Wouldn't it be better to do a `riak-admin replace`?  Leave could be 
>> problematic if there are other nodes in the cluster that are 
>> under-provisioned (disk space, for example).  Plus a leave and add would 
>> move the data around the cluster twice, for each node in the cluster, 
>> whereas a replace would just move the data to the new node once, no?
>>  
>> -Fred
>> 
>> 
>> On Feb 1, 2019, at 8:32 AM, Nicholas Adams > <mailto:nicholas.ad...@tiot.jp>> wrote:
>>  
>> Hi Guido,
>> Although Martin would be the most qualified to comment on this, I believe 
>> you should be able to do a slow migration.
>>  
>> Choose target node.
>> Make target node leave cluster as in a full “riak-admin cluster leave”, 
>> commit and wait for transfers to finish.
>> Set up target node with leveled and TicTac AAE.
>> Have node rejoin cluster and wait for transfers to finish.
>> Repeat with every single node in the cluster until all have been done.
>>  
>> Unless you are using specific features restricted to your current backend 
>> then Riak will usually put up with multiple backends in the same cluster.
>>  
>> Failing that, I’d go with Bryan’s suggestion to use MDC to replicate from 
>> your existing cluster to a separate cluster that is using the leveled 
>> backend and TicTac AAE.
>>  
>> Either way, be sure to try in a dev environment first and only proceed when 
>> you are happy with the process.
>>  
>> Best regards,
>>  
>> Nicholas
>>  
>> From: riak-users > <mailto:riak-users-boun...@lists.basho.com>> On Behalf Of Bryan Hunt
>> Sent: 01 February 2019 19:22
>> To: Guido Medina mailto:gmed...@temetra.com>>
>> Cc: riak-users@lists.basho.com <mailto:riak-users@lists.basho.com>
>> Subject: Re: [ANN] Riak 2.9.0 - Release Candidate 1 Available
>>  
>> Replication would be the optimum solution - in theory anything can be 
>> implemented - but it would be enormously painful in comparison to simply 
>> standing up a new cluster.
>> 
>> 
>> 
>> On 1 Feb 2019, at 10:14, Guido Medina > <mailto:gmed...@temetra.com>> wrote:
>>  
>> Hi all,
>> 
>> Nice work on the upcoming 2.9.0 release, I have a quick question:
>> Will it be possible to switch from the eleveldb to the new leveled backend 
>> and Tictac AAE for an existing cluster?
>> 
>> In case it is not possible we are thinking to use the new replication and 
>> move to a brand new cluster.
>> 
>> Kind regards,
>> Guido.
>> 
>> 
>> On 30/01/2019 15:23, Nicholas Adams wrote:
>> Dear All,
>> Following on from Martin’s email, the initial builds of the KV 2.9.0rc1 
>> packages are complete. Currently we are limited to Redhat/CentOS 6 and 7 as 
>> well as Ubuntu Bionic. We shall be adding more packages over the next few 
>> days. Please see https://files.tiot.jp/riak/kv/2.9/2.9.0rc1/ 
>> <https://files.tiot.jp/riak/kv/2.9/2.9.0rc1/> to download.
>>  
>> Disclaimer: This is a release candidate. Please do not u

Re: [ANN] Riak 2.9.0 - Release Candidate 1 Available

[Guido Medina]

We could probably go the "replace" way, but maybe we take this as a good 
opportunity to increase our ring size, it is still something we are 
considering.


Resizing the cluster while operating daily is no joke, too much data and 
Riak becomes extremely slow when we have to add a new node, so it would 
either go the "replace" way or replicate and switch to the new cluster.


Thanks for all the answers ;-)
Guido.

On 01/02/2019 13:50, Nicholas Adams wrote:


If he has an extra node he can add then a replace would be much 
better. I provided this example under the assumption that he had no 
additional resources to play with.


Nicholas

*From:* riak-users  *On Behalf Of 
*Fred Dushin

*Sent:* 01 February 2019 22:47
*To:* riak-users@lists.basho.com
*Subject:* Re: [ANN] Riak 2.9.0 - Release Candidate 1 Available

Wouldn't it be better to do a `riak-admin replace`?  Leave could be 
problematic if there are other nodes in the cluster that are 
under-provisioned (disk space, for example).  Plus a leave and add 
would move the data around the cluster twice, for each node in the 
cluster, whereas a replace would just move the data to the new node 
once, no?


-Fred



On Feb 1, 2019, at 8:32 AM, Nicholas Adams mailto:nicholas.ad...@tiot.jp>> wrote:

Hi Guido,

Although Martin would be the most qualified to comment on this, I
believe you should be able to do a slow migration.

 1. Choose target node.
 2. Make target node leave cluster as in a full “riak-admin
cluster leave”, commit and wait for transfers to finish.
 3. Set up target node with leveled and TicTac AAE.
 4. Have node rejoin cluster and wait for transfers to finish.
 5. Repeat with every single node in the cluster until all have
been done.

Unless you are using specific features restricted to your current
backend then Riak will usually put up with multiple backends in
the same cluster.

Failing that, I’d go with Bryan’s suggestion to use MDC to
replicate from your existing cluster to a separate cluster that is
using the leveled backend and TicTac AAE.

Either way, be sure to try in a dev environment first and only
proceed when you are happy with the process.

Best regards,

Nicholas

*From:*riak-users mailto:riak-users-boun...@lists.basho.com>>*On Behalf Of*Bryan Hunt
*Sent:*01 February 2019 19:22
*To:*Guido Medina mailto:gmed...@temetra.com>>
*Cc:*riak-users@lists.basho.com <mailto:riak-users@lists.basho.com>
*Subject:*Re: [ANN] Riak 2.9.0 - Release Candidate 1 Available

Replication would be the optimum solution - in theory anything can
be implemented - but it would be enormously painful in comparison
to simply standing up a new cluster.




On 1 Feb 2019, at 10:14, Guido Medina mailto:gmed...@temetra.com>> wrote:

Hi all,

Nice work on the upcoming 2.9.0 release, I have a quick question:
Will it be possible to switch from the eleveldb to the new
leveled backend and Tictac AAE for an existing cluster?

In case it is not possible we are thinking to use the new
replication and move to a brand new cluster.

Kind regards,
Guido.


On 30/01/2019 15:23, Nicholas Adams wrote:

Dear All,

Following on from Martin’s email, the initial builds of
the KV 2.9.0rc1 packages are complete. Currently we are
limited to Redhat/CentOS 6 and 7 as well as Ubuntu Bionic.
We shall be adding more packages over the next few days.
Please seehttps://files.tiot.jp/riak/kv/2.9/2.9.0rc1/to
download.

Disclaimer: This is a release candidate. Please do not use
in production as we expect bugs and abnormal functionality
may occur in certain conditions. However, please test this
lots and place any issues on github so that they can be
fixed before the final release.

Best regards,

Nicholas

*From:*riak-users
<mailto:riak-users-boun...@lists.basho.com>*On Behalf
Of*Martin Sumner
*Sent:*30 January 2019 21:07
*To:*riak-users@lists.basho.com
<mailto:riak-users@lists.basho.com>
*Subject:*[ANN] Riak 2.9.0 - Release Candidate 1 Available

All,

There is now a publicly available release candidate for
Riak 2.9.0 for users to test in their own environment. The
release candidate is available to build from source here -
https://github.com/basho/riak/tree/riak-2.9.0rc1

There is only one significant change to Riak 2.2.6 for
users maintaining existing configuration settings.  The
release adds the `vnode_soft_limits` feature that aims to
reduce high percentile PUT latency, by chec

Re: [ANN] Riak 2.9.0 - Release Candidate 1 Available

[Fred Dushin]

Lol.  Good point.

I always assume people have spares lying around..  My bad.

-Fred

> On Feb 1, 2019, at 8:50 AM, Nicholas Adams  wrote:
> 
> If he has an extra node he can add then a replace would be much better. I 
> provided this example under the assumption that he had no additional 
> resources to play with.
>  
> Nicholas
>  
> From: riak-users  On Behalf Of Fred Dushin
> Sent: 01 February 2019 22:47
> To: riak-users@lists.basho.com
> Subject: Re: [ANN] Riak 2.9.0 - Release Candidate 1 Available
>  
> Wouldn't it be better to do a `riak-admin replace`?  Leave could be 
> problematic if there are other nodes in the cluster that are 
> under-provisioned (disk space, for example).  Plus a leave and add would move 
> the data around the cluster twice, for each node in the cluster, whereas a 
> replace would just move the data to the new node once, no?
>  
> -Fred
> 
> 
> On Feb 1, 2019, at 8:32 AM, Nicholas Adams  <mailto:nicholas.ad...@tiot.jp>> wrote:
>  
> Hi Guido,
> Although Martin would be the most qualified to comment on this, I believe you 
> should be able to do a slow migration.
>  
> Choose target node.
> Make target node leave cluster as in a full “riak-admin cluster leave”, 
> commit and wait for transfers to finish.
> Set up target node with leveled and TicTac AAE.
> Have node rejoin cluster and wait for transfers to finish.
> Repeat with every single node in the cluster until all have been done.
>  
> Unless you are using specific features restricted to your current backend 
> then Riak will usually put up with multiple backends in the same cluster.
>  
> Failing that, I’d go with Bryan’s suggestion to use MDC to replicate from 
> your existing cluster to a separate cluster that is using the leveled backend 
> and TicTac AAE.
>  
> Either way, be sure to try in a dev environment first and only proceed when 
> you are happy with the process.
>  
> Best regards,
>  
> Nicholas
>  
> From: riak-users  <mailto:riak-users-boun...@lists.basho.com>> On Behalf Of Bryan Hunt
> Sent: 01 February 2019 19:22
> To: Guido Medina mailto:gmed...@temetra.com>>
> Cc: riak-users@lists.basho.com <mailto:riak-users@lists.basho.com>
> Subject: Re: [ANN] Riak 2.9.0 - Release Candidate 1 Available
>  
> Replication would be the optimum solution - in theory anything can be 
> implemented - but it would be enormously painful in comparison to simply 
> standing up a new cluster.
> 
> 
> 
> On 1 Feb 2019, at 10:14, Guido Medina  <mailto:gmed...@temetra.com>> wrote:
>  
> Hi all,
> 
> Nice work on the upcoming 2.9.0 release, I have a quick question:
> Will it be possible to switch from the eleveldb to the new leveled backend 
> and Tictac AAE for an existing cluster?
> 
> In case it is not possible we are thinking to use the new replication and 
> move to a brand new cluster.
> 
> Kind regards,
> Guido.
> 
> 
> On 30/01/2019 15:23, Nicholas Adams wrote:
> Dear All,
> Following on from Martin’s email, the initial builds of the KV 2.9.0rc1 
> packages are complete. Currently we are limited to Redhat/CentOS 6 and 7 as 
> well as Ubuntu Bionic. We shall be adding more packages over the next few 
> days. Please see https://files.tiot.jp/riak/kv/2.9/2.9.0rc1/ 
> <https://files.tiot.jp/riak/kv/2.9/2.9.0rc1/> to download.
>  
> Disclaimer: This is a release candidate. Please do not use in production as 
> we expect bugs and abnormal functionality may occur in certain conditions. 
> However, please test this lots and place any issues on github so that they 
> can be fixed before the final release.
>  
> Best regards,
>  
> Nicholas
>  
> From: riak-users  
> <mailto:riak-users-boun...@lists.basho.com> On Behalf Of Martin Sumner
> Sent: 30 January 2019 21:07
> To: riak-users@lists.basho.com <mailto:riak-users@lists.basho.com>
> Subject: [ANN] Riak 2.9.0 - Release Candidate 1 Available
>  
> All,
>  
> There is now a publicly available release candidate for Riak 2.9.0 for users 
> to test in their own environment.  The release candidate is available to 
> build from source here - https://github.com/basho/riak/tree/riak-2.9.0rc1 
> <https://github.com/basho/riak/tree/riak-2.9.0rc1>
>  
> There is only one significant change to Riak 2.2.6 for users maintaining 
> existing configuration settings.  The release adds the `vnode_soft_limits` 
> feature that aims to reduce high percentile PUT latency, by checking the 
> outstanding work queue for a vnode before selecting it as a coordinator of a 
> PUT.
>  
> With additional configuration, there are two major features added in riak 
> 2.9.0:
>  
> - Support for the leveled backen

RE: [ANN] Riak 2.9.0 - Release Candidate 1 Available

[Nicholas Adams]

If he has an extra node he can add then a replace would be much better. I 
provided this example under the assumption that he had no additional resources 
to play with.

Nicholas

From: riak-users  On Behalf Of Fred Dushin
Sent: 01 February 2019 22:47
To: riak-users@lists.basho.com
Subject: Re: [ANN] Riak 2.9.0 - Release Candidate 1 Available

Wouldn't it be better to do a `riak-admin replace`?  Leave could be problematic 
if there are other nodes in the cluster that are under-provisioned (disk space, 
for example).  Plus a leave and add would move the data around the cluster 
twice, for each node in the cluster, whereas a replace would just move the data 
to the new node once, no?

-Fred


On Feb 1, 2019, at 8:32 AM, Nicholas Adams 
mailto:nicholas.ad...@tiot.jp>> wrote:

Hi Guido,
Although Martin would be the most qualified to comment on this, I believe you 
should be able to do a slow migration.


  1.  Choose target node.
  2.  Make target node leave cluster as in a full “riak-admin cluster leave”, 
commit and wait for transfers to finish.
  3.  Set up target node with leveled and TicTac AAE.
  4.  Have node rejoin cluster and wait for transfers to finish.
  5.  Repeat with every single node in the cluster until all have been done.

Unless you are using specific features restricted to your current backend then 
Riak will usually put up with multiple backends in the same cluster.

Failing that, I’d go with Bryan’s suggestion to use MDC to replicate from your 
existing cluster to a separate cluster that is using the leveled backend and 
TicTac AAE.

Either way, be sure to try in a dev environment first and only proceed when you 
are happy with the process.

Best regards,

Nicholas

From: riak-users 
mailto:riak-users-boun...@lists.basho.com>> 
On Behalf Of Bryan Hunt
Sent: 01 February 2019 19:22
To: Guido Medina mailto:gmed...@temetra.com>>
Cc: riak-users@lists.basho.com<mailto:riak-users@lists.basho.com>
Subject: Re: [ANN] Riak 2.9.0 - Release Candidate 1 Available

Replication would be the optimum solution - in theory anything can be 
implemented - but it would be enormously painful in comparison to simply 
standing up a new cluster.



On 1 Feb 2019, at 10:14, Guido Medina 
mailto:gmed...@temetra.com>> wrote:

Hi all,

Nice work on the upcoming 2.9.0 release, I have a quick question:
Will it be possible to switch from the eleveldb to the new leveled backend and 
Tictac AAE for an existing cluster?

In case it is not possible we are thinking to use the new replication and move 
to a brand new cluster.

Kind regards,
Guido.


On 30/01/2019 15:23, Nicholas Adams wrote:
Dear All,
Following on from Martin’s email, the initial builds of the KV 2.9.0rc1 
packages are complete. Currently we are limited to Redhat/CentOS 6 and 7 as 
well as Ubuntu Bionic. We shall be adding more packages over the next few days. 
Please see https://files.tiot.jp/riak/kv/2.9/2.9.0rc1/ to download.

Disclaimer: This is a release candidate. Please do not use in production as we 
expect bugs and abnormal functionality may occur in certain conditions. 
However, please test this lots and place any issues on github so that they can 
be fixed before the final release.

Best regards,

Nicholas

From: riak-users 
<mailto:riak-users-boun...@lists.basho.com> 
On Behalf Of Martin Sumner
Sent: 30 January 2019 21:07
To: riak-users@lists.basho.com<mailto:riak-users@lists.basho.com>
Subject: [ANN] Riak 2.9.0 - Release Candidate 1 Available

All,

There is now a publicly available release candidate for Riak 2.9.0 for users to 
test in their own environment.  The release candidate is available to build 
from source here - https://github.com/basho/riak/tree/riak-2.9.0rc1

There is only one significant change to Riak 2.2.6 for users maintaining 
existing configuration settings.  The release adds the `vnode_soft_limits` 
feature that aims to reduce high percentile PUT latency, by checking the 
outstanding work queue for a vnode before selecting it as a coordinator of a 
PUT.

With additional configuration, there are two major features added in riak 2.9.0:

- Support for the leveled backend (as an alternative to bitcask or eleveldb), 
to provide for improved throughput in some use cases and lower tail latency in 
some use cases 
-https://github.com/martinsumner/riak_testing_notes/blob/master/Release%202.9%20-%20Choosing%20a%20Backend.md;
- Support for a new anti-entropy mechanism (Tictac AAE) as an alternative for 
the existing anti-entropy mechanism (to provide both intra-cluster and 
inter-cluster data repair, with greater flexibility and reliability).

The release also adds support for:
- The riak_core node_worker_pool - which provides a mechanism for queueing 
background jobs and queries to control the resource consumed on the node by 
different queries.  No pre-existing features will use the node-worker_pool.
- AAE folds which allow for both cluster-wide AAE queries (e.g. produce a 
merkle tree represen

Re: [ANN] Riak 2.9.0 - Release Candidate 1 Available

[Fred Dushin]

Wouldn't it be better to do a `riak-admin replace`?  Leave could be problematic 
if there are other nodes in the cluster that are under-provisioned (disk space, 
for example).  Plus a leave and add would move the data around the cluster 
twice, for each node in the cluster, whereas a replace would just move the data 
to the new node once, no?

-Fred

> On Feb 1, 2019, at 8:32 AM, Nicholas Adams  wrote:
> 
> Hi Guido,
> Although Martin would be the most qualified to comment on this, I believe you 
> should be able to do a slow migration.
>  
> Choose target node.
> Make target node leave cluster as in a full “riak-admin cluster leave”, 
> commit and wait for transfers to finish.
> Set up target node with leveled and TicTac AAE.
> Have node rejoin cluster and wait for transfers to finish.
> Repeat with every single node in the cluster until all have been done.
>  
> Unless you are using specific features restricted to your current backend 
> then Riak will usually put up with multiple backends in the same cluster.
>  
> Failing that, I’d go with Bryan’s suggestion to use MDC to replicate from 
> your existing cluster to a separate cluster that is using the leveled backend 
> and TicTac AAE.
>  
> Either way, be sure to try in a dev environment first and only proceed when 
> you are happy with the process.
>  
> Best regards,
>  
> Nicholas
>  
> From: riak-users  On Behalf Of Bryan Hunt
> Sent: 01 February 2019 19:22
> To: Guido Medina 
> Cc: riak-users@lists.basho.com
> Subject: Re: [ANN] Riak 2.9.0 - Release Candidate 1 Available
>  
> Replication would be the optimum solution - in theory anything can be 
> implemented - but it would be enormously painful in comparison to simply 
> standing up a new cluster.
> 
> 
> On 1 Feb 2019, at 10:14, Guido Medina  <mailto:gmed...@temetra.com>> wrote:
>  
> Hi all,
> 
> Nice work on the upcoming 2.9.0 release, I have a quick question:
> Will it be possible to switch from the eleveldb to the new leveled backend 
> and Tictac AAE for an existing cluster?
> 
> In case it is not possible we are thinking to use the new replication and 
> move to a brand new cluster.
> 
> Kind regards,
> Guido.
> 
> On 30/01/2019 15:23, Nicholas Adams wrote:
> Dear All,
> Following on from Martin’s email, the initial builds of the KV 2.9.0rc1 
> packages are complete. Currently we are limited to Redhat/CentOS 6 and 7 as 
> well as Ubuntu Bionic. We shall be adding more packages over the next few 
> days. Please see https://files.tiot.jp/riak/kv/2.9/2.9.0rc1/ 
> <https://files.tiot.jp/riak/kv/2.9/2.9.0rc1/> to download.
>  
> Disclaimer: This is a release candidate. Please do not use in production as 
> we expect bugs and abnormal functionality may occur in certain conditions. 
> However, please test this lots and place any issues on github so that they 
> can be fixed before the final release.
>  
> Best regards,
>  
> Nicholas
>  
> From: riak-users  
> <mailto:riak-users-boun...@lists.basho.com> On Behalf Of Martin Sumner
> Sent: 30 January 2019 21:07
> To: riak-users@lists.basho.com <mailto:riak-users@lists.basho.com>
> Subject: [ANN] Riak 2.9.0 - Release Candidate 1 Available
>  
> All,
>  
> There is now a publicly available release candidate for Riak 2.9.0 for users 
> to test in their own environment.  The release candidate is available to 
> build from source here - https://github.com/basho/riak/tree/riak-2.9.0rc1 
> <https://github.com/basho/riak/tree/riak-2.9.0rc1>
>  
> There is only one significant change to Riak 2.2.6 for users maintaining 
> existing configuration settings.  The release adds the `vnode_soft_limits` 
> feature that aims to reduce high percentile PUT latency, by checking the 
> outstanding work queue for a vnode before selecting it as a coordinator of a 
> PUT.
>  
> With additional configuration, there are two major features added in riak 
> 2.9.0:
>  
> - Support for the leveled backend (as an alternative to bitcask or eleveldb), 
> to provide for improved throughput in some use cases and lower tail latency 
> in some use cases 
> -https://github.com/martinsumner/riak_testing_notes/blob/master/Release%202.9%20-%20Choosing%20a%20Backend.md
>  
> <https://github.com/martinsumner/riak_testing_notes/blob/master/Release%202.9%20-%20Choosing%20a%20Backend.md>;
> - Support for a new anti-entropy mechanism (Tictac AAE) as an alternative for 
> the existing anti-entropy mechanism (to provide both intra-cluster and 
> inter-cluster data repair, with greater flexibility and reliability).
>  
> The release also adds support for:
> - The riak_core node_worker_pool - which provides a mechanism for queueing 
> background jo

RE: [ANN] Riak 2.9.0 - Release Candidate 1 Available

[Nicholas Adams]

Hi Guido,
Although Martin would be the most qualified to comment on this, I believe you 
should be able to do a slow migration.


  1.  Choose target node.
  2.  Make target node leave cluster as in a full “riak-admin cluster leave”, 
commit and wait for transfers to finish.
  3.  Set up target node with leveled and TicTac AAE.
  4.  Have node rejoin cluster and wait for transfers to finish.
  5.  Repeat with every single node in the cluster until all have been done.

Unless you are using specific features restricted to your current backend then 
Riak will usually put up with multiple backends in the same cluster.

Failing that, I’d go with Bryan’s suggestion to use MDC to replicate from your 
existing cluster to a separate cluster that is using the leveled backend and 
TicTac AAE.

Either way, be sure to try in a dev environment first and only proceed when you 
are happy with the process.

Best regards,

Nicholas

From: riak-users  On Behalf Of Bryan Hunt
Sent: 01 February 2019 19:22
To: Guido Medina 
Cc: riak-users@lists.basho.com
Subject: Re: [ANN] Riak 2.9.0 - Release Candidate 1 Available

Replication would be the optimum solution - in theory anything can be 
implemented - but it would be enormously painful in comparison to simply 
standing up a new cluster.


On 1 Feb 2019, at 10:14, Guido Medina 
mailto:gmed...@temetra.com>> wrote:

Hi all,

Nice work on the upcoming 2.9.0 release, I have a quick question:
Will it be possible to switch from the eleveldb to the new leveled backend and 
Tictac AAE for an existing cluster?

In case it is not possible we are thinking to use the new replication and move 
to a brand new cluster.

Kind regards,
Guido.

On 30/01/2019 15:23, Nicholas Adams wrote:
Dear All,
Following on from Martin’s email, the initial builds of the KV 2.9.0rc1 
packages are complete. Currently we are limited to Redhat/CentOS 6 and 7 as 
well as Ubuntu Bionic. We shall be adding more packages over the next few days. 
Please see https://files.tiot.jp/riak/kv/2.9/2.9.0rc1/ to download.

Disclaimer: This is a release candidate. Please do not use in production as we 
expect bugs and abnormal functionality may occur in certain conditions. 
However, please test this lots and place any issues on github so that they can 
be fixed before the final release.

Best regards,

Nicholas

From: riak-users 
<mailto:riak-users-boun...@lists.basho.com> 
On Behalf Of Martin Sumner
Sent: 30 January 2019 21:07
To: riak-users@lists.basho.com<mailto:riak-users@lists.basho.com>
Subject: [ANN] Riak 2.9.0 - Release Candidate 1 Available

All,

There is now a publicly available release candidate for Riak 2.9.0 for users to 
test in their own environment.  The release candidate is available to build 
from source here - https://github.com/basho/riak/tree/riak-2.9.0rc1

There is only one significant change to Riak 2.2.6 for users maintaining 
existing configuration settings.  The release adds the `vnode_soft_limits` 
feature that aims to reduce high percentile PUT latency, by checking the 
outstanding work queue for a vnode before selecting it as a coordinator of a 
PUT.

With additional configuration, there are two major features added in riak 2.9.0:

- Support for the leveled backend (as an alternative to bitcask or eleveldb), 
to provide for improved throughput in some use cases and lower tail latency in 
some use cases 
-https://github.com/martinsumner/riak_testing_notes/blob/master/Release%202.9%20-%20Choosing%20a%20Backend.md;
- Support for a new anti-entropy mechanism (Tictac AAE) as an alternative for 
the existing anti-entropy mechanism (to provide both intra-cluster and 
inter-cluster data repair, with greater flexibility and reliability).

The release also adds support for:
- The riak_core node_worker_pool - which provides a mechanism for queueing 
background jobs and queries to control the resource consumed on the node by 
different queries.  No pre-existing features will use the node-worker_pool.
- AAE folds which allow for both cluster-wide AAE queries (e.g. produce a 
merkle tree representing all or a partial range of the cluster data), and 
administrative queries (e.g. discovering object sizes and sibling counts within 
the database depending on bucket name, key range and last modified date).  AAE 
folds depend on the Tictac AAE feature being activated.
- An interface to request re-replication of an object (where variances between 
clusters have been discovered).

Further details of the release, and the release plan in general can be found 
here - 
https://github.com/basho/riak/blob/develop-2.9/doc/Release%202.9%20Series%20-%20Overview.md.

It is hoped that there will be a short period of about 1 month before the 
release candidate will be converted into a formal release.  The period will 
allow for more testing by Riak users, and also there will be further enhanced 
testing of the new modules with the help of Quviq (http://www.quviq.com/).

There will follow additional releases un

Re: [ANN] Riak 2.9.0 - Release Candidate 1 Available

[Bryan Hunt]

Replication would be the optimum solution - in theory anything can be 
implemented - but it would be enormously painful in comparison to simply 
standing up a new cluster.

> On 1 Feb 2019, at 10:14, Guido Medina  wrote:
> 
> Hi all,
> 
> Nice work on the upcoming 2.9.0 release, I have a quick question:
> Will it be possible to switch from the eleveldb to the new leveled backend 
> and Tictac AAE for an existing cluster?
> 
> In case it is not possible we are thinking to use the new replication and 
> move to a brand new cluster.
> 
> Kind regards,
> Guido.
> 
> On 30/01/2019 15:23, Nicholas Adams wrote:
>> Dear All,
>> Following on from Martin’s email, the initial builds of the KV 2.9.0rc1 
>> packages are complete. Currently we are limited to Redhat/CentOS 6 and 7 as 
>> well as Ubuntu Bionic. We shall be adding more packages over the next few 
>> days. Please see https://files.tiot.jp/riak/kv/2.9/2.9.0rc1/ 
>> <https://files.tiot.jp/riak/kv/2.9/2.9.0rc1/> to download.
>>  
>> Disclaimer: This is a release candidate. Please do not use in production as 
>> we expect bugs and abnormal functionality may occur in certain conditions. 
>> However, please test this lots and place any issues on github so that they 
>> can be fixed before the final release.
>>  
>> Best regards,
>>  
>> Nicholas
>>  
>> From: riak-users  
>> <mailto:riak-users-boun...@lists.basho.com> On Behalf Of Martin Sumner
>> Sent: 30 January 2019 21:07
>> To: riak-users@lists.basho.com <mailto:riak-users@lists.basho.com>
>> Subject: [ANN] Riak 2.9.0 - Release Candidate 1 Available
>>  
>> All,
>>  
>> There is now a publicly available release candidate for Riak 2.9.0 for users 
>> to test in their own environment.  The release candidate is available to 
>> build from source here - https://github.com/basho/riak/tree/riak-2.9.0rc1 
>> <https://github.com/basho/riak/tree/riak-2.9.0rc1>
>>  
>> There is only one significant change to Riak 2.2.6 for users maintaining 
>> existing configuration settings.  The release adds the `vnode_soft_limits` 
>> feature that aims to reduce high percentile PUT latency, by checking the 
>> outstanding work queue for a vnode before selecting it as a coordinator of a 
>> PUT.
>>  
>> With additional configuration, there are two major features added in riak 
>> 2.9.0:
>>  
>> - Support for the leveled backend (as an alternative to bitcask or 
>> eleveldb), to provide for improved throughput in some use cases and lower 
>> tail latency in some use cases 
>> -https://github.com/martinsumner/riak_testing_notes/blob/master/Release%202.9%20-%20Choosing%20a%20Backend.md
>>  
>> <https://github.com/martinsumner/riak_testing_notes/blob/master/Release%202.9%20-%20Choosing%20a%20Backend.md>;
>> - Support for a new anti-entropy mechanism (Tictac AAE) as an alternative 
>> for the existing anti-entropy mechanism (to provide both intra-cluster and 
>> inter-cluster data repair, with greater flexibility and reliability).
>>  
>> The release also adds support for:
>> - The riak_core node_worker_pool - which provides a mechanism for queueing 
>> background jobs and queries to control the resource consumed on the node by 
>> different queries.  No pre-existing features will use the node-worker_pool.
>> - AAE folds which allow for both cluster-wide AAE queries (e.g. produce a 
>> merkle tree representing all or a partial range of the cluster data), and 
>> administrative queries (e.g. discovering object sizes and sibling counts 
>> within the database depending on bucket name, key range and last modified 
>> date).  AAE folds depend on the Tictac AAE feature being activated.
>> - An interface to request re-replication of an object (where variances 
>> between clusters have been discovered).
>>  
>> Further details of the release, and the release plan in general can be found 
>> here - 
>> https://github.com/basho/riak/blob/develop-2.9/doc/Release%202.9%20Series%20-%20Overview.md
>>  
>> <https://github.com/basho/riak/blob/develop-2.9/doc/Release%202.9%20Series%20-%20Overview.md>.
>>  
>> It is hoped that there will be a short period of about 1 month before the 
>> release candidate will be converted into a formal release.  The period will 
>> allow for more testing by Riak users, and also there will be further 
>> enhanced testing of the new modules with the help of Quviq 
>> (http://www.quviq.com/ <http://www.quviq.com/>).  
>>  
>> There will follow additional releases under the 2.9 banner, targeted at 
>&

Re: [ANN] Riak 2.9.0 - Release Candidate 1 Available

[Guido Medina]

Hi all,

Nice work on the upcoming 2.9.0 release, I have a quick question:
Will it be possible to switch from the eleveldb to the new leveled 
backend and Tictac AAE for an existing cluster?


In case it is not possible we are thinking to use the new replication 
and move to a brand new cluster.


Kind regards,
Guido.

On 30/01/2019 15:23, Nicholas Adams wrote:


Dear All,

Following on from Martin’s email, the initial builds of the KV 
2.9.0rc1 packages are complete. Currently we are limited to 
Redhat/CentOS 6 and 7 as well as Ubuntu Bionic. We shall be adding 
more packages over the next few days. Please see 
https://files.tiot.jp/riak/kv/2.9/2.9.0rc1/ to download.


Disclaimer: This is a release candidate. Please do not use in 
production as we expect bugs and abnormal functionality may occur in 
certain conditions. However, please test this lots and place any 
issues on github so that they can be fixed before the final release.


Best regards,

Nicholas

*From:* riak-users  *On Behalf Of 
*Martin Sumner

*Sent:* 30 January 2019 21:07
*To:* riak-users@lists.basho.com
*Subject:* [ANN] Riak 2.9.0 - Release Candidate 1 Available

All,

There is now a publicly available release candidate for Riak 2.9.0 for 
users to test in their own environment.  The release candidate is 
available to build from source here - 
https://github.com/basho/riak/tree/riak-2.9.0rc1


There is only one significant change to Riak 2.2.6 for users 
maintaining existing configuration settings.  The release adds the 
`vnode_soft_limits` feature that aims to reduce high percentile PUT 
latency, by checking the outstanding work queue for a vnode before 
selecting it as a coordinator of a PUT.


With additional configuration, there are two major features added in 
riak 2.9.0:


- Support for the leveled backend (as an alternative to bitcask or 
eleveldb), to provide for improved throughput in some use cases and 
lower tail latency in some use cases - 
https://github.com/martinsumner/riak_testing_notes/blob/master/Release%202.9%20-%20Choosing%20a%20Backend.md;


- Support for a new anti-entropy mechanism (Tictac AAE) as an 
alternative for the existing anti-entropy mechanism (to provide both 
intra-cluster and inter-cluster data repair, with greater flexibility 
and reliability).


The release also adds support for:

- The riak_core node_worker_pool - which provides a mechanism for 
queueing background jobs and queries to control the resource consumed 
on the node by different queries.  No pre-existing features will use 
the node-worker_pool.


- AAE folds which allow for both cluster-wide AAE queries (e.g. 
produce a merkle tree representing all or a partial range of the 
cluster data), and administrative queries (e.g. discovering object 
sizes and sibling counts within the database depending on bucket name, 
key range and last modified date).  AAE folds depend on the Tictac AAE 
feature being activated.


- An interface to request re-replication of an object (where variances 
between clusters have been discovered).


Further details of the release, and the release plan in general can be 
found here - 
https://github.com/basho/riak/blob/develop-2.9/doc/Release%202.9%20Series%20-%20Overview.md.


It is hoped that there will be a short period of about 1 month before 
the release candidate will be converted into a formal release.  The 
period will allow for more testing by Riak users, and also there will 
be further enhanced testing of the new modules with the help of Quviq 
(http://www.quviq.com/).


There will follow additional releases under the 2.9 banner, targeted 
at enhancing both new and existing inter-cluster replication 
features.  In parallel to this, work will continue on Release 3.0 
which is intended to modernise the OTP/rebar platform used by Riak.


Thanks to all those who contributed to the release.  Apologies to 
those who have been kept waiting over the past few months as 
finalising the changes and completing the testing has dragged on.


Regards

Martin


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

RE: [ANN] Riak 2.9.0 - Release Candidate 1 Available

[Nicholas Adams]

Dear All,
Following on from Martin’s email, the initial builds of the KV 2.9.0rc1 
packages are complete. Currently we are limited to Redhat/CentOS 6 and 7 as 
well as Ubuntu Bionic. We shall be adding more packages over the next few days. 
Please see https://files.tiot.jp/riak/kv/2.9/2.9.0rc1/ to download.

Disclaimer: This is a release candidate. Please do not use in production as we 
expect bugs and abnormal functionality may occur in certain conditions. 
However, please test this lots and place any issues on github so that they can 
be fixed before the final release.

Best regards,

Nicholas

From: riak-users  On Behalf Of Martin Sumner
Sent: 30 January 2019 21:07
To: riak-users@lists.basho.com
Subject: [ANN] Riak 2.9.0 - Release Candidate 1 Available

All,

There is now a publicly available release candidate for Riak 2.9.0 for users to 
test in their own environment.  The release candidate is available to build 
from source here - https://github.com/basho/riak/tree/riak-2.9.0rc1

There is only one significant change to Riak 2.2.6 for users maintaining 
existing configuration settings.  The release adds the `vnode_soft_limits` 
feature that aims to reduce high percentile PUT latency, by checking the 
outstanding work queue for a vnode before selecting it as a coordinator of a 
PUT.

With additional configuration, there are two major features added in riak 2.9.0:

- Support for the leveled backend (as an alternative to bitcask or eleveldb), 
to provide for improved throughput in some use cases and lower tail latency in 
some use cases - 
https://github.com/martinsumner/riak_testing_notes/blob/master/Release%202.9%20-%20Choosing%20a%20Backend.md;
- Support for a new anti-entropy mechanism (Tictac AAE) as an alternative for 
the existing anti-entropy mechanism (to provide both intra-cluster and 
inter-cluster data repair, with greater flexibility and reliability).

The release also adds support for:
- The riak_core node_worker_pool - which provides a mechanism for queueing 
background jobs and queries to control the resource consumed on the node by 
different queries.  No pre-existing features will use the node-worker_pool.
- AAE folds which allow for both cluster-wide AAE queries (e.g. produce a 
merkle tree representing all or a partial range of the cluster data), and 
administrative queries (e.g. discovering object sizes and sibling counts within 
the database depending on bucket name, key range and last modified date).  AAE 
folds depend on the Tictac AAE feature being activated.
- An interface to request re-replication of an object (where variances between 
clusters have been discovered).

Further details of the release, and the release plan in general can be found 
here - 
https://github.com/basho/riak/blob/develop-2.9/doc/Release%202.9%20Series%20-%20Overview.md.

It is hoped that there will be a short period of about 1 month before the 
release candidate will be converted into a formal release.  The period will 
allow for more testing by Riak users, and also there will be further enhanced 
testing of the new modules with the help of Quviq (http://www.quviq.com/).

There will follow additional releases under the 2.9 banner, targeted at 
enhancing both new and existing inter-cluster replication features.  In 
parallel to this, work will continue on Release 3.0 which is intended to 
modernise the OTP/rebar platform used by Riak.

Thanks to all those who contributed to the release.  Apologies to those who 
have been kept waiting over the past few months as finalising the changes and 
completing the testing has dragged on.

Regards

Martin



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Looking for the official repository for riak packages

[Henning Verbeek]

Thanks, that helps me!

Cheers,
hank

On Fri, Jan 11, 2019, 14:10 Nicholas Adams  Hi Hank,
> We are looking into bringing new packages onto packagecloud but when this
> will become available is not confirmed. In the meantime, TI Tokyo will
> continue to make new packages available on https://files.tiot.jp/riak/ .
> There are plans underway to make everything a little more transparent but
> we will have to wait a little longer for them to come to fruition.
>
> Best regards,
>
> Nicholas
>
> -Original Message-
> From: riak-users  On Behalf Of
> Henning Verbeek
> Sent: 10 January 2019 20:42
> To: riak-users 
> Subject: Looking for the official repository for riak packages
>
> Hello,
>
> I'm trying to understand what is the plan for official riak releases and
> where to find packages.
>
> In the 2.2.6 announcement, Nicholas Adams provided
> https://files.tiot.jp/riak/kv/2.2/2.2.6/ as the location for binary
> packages. Is this correct? Is there a plan to publish packages on
> packagecloud again?
>
> Thanks and regards,
> Hank
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Looking for the official repository for riak packages

[Steve Roberts]

Hi Hank

TI Tokyo has packages for most distributions. There are also Riak CentOS 
packages (and the Erlang environment packages) available at:


https://www.erlang-solutions.com/resources/download.html

Steve

On 11/01/19 13:10, Nicholas Adams wrote:

Hi Hank,
We are looking into bringing new packages onto packagecloud but when this will 
become available is not confirmed. In the meantime, TI Tokyo will continue to 
make new packages available on https://files.tiot.jp/riak/ . There are plans 
underway to make everything a little more transparent but we will have to wait 
a little longer for them to come to fruition.

Best regards,

Nicholas

-Original Message-
From: riak-users  On Behalf Of Henning 
Verbeek
Sent: 10 January 2019 20:42
To: riak-users 
Subject: Looking for the official repository for riak packages

Hello,

I'm trying to understand what is the plan for official riak releases and where 
to find packages.

In the 2.2.6 announcement, Nicholas Adams provided 
https://files.tiot.jp/riak/kv/2.2/2.2.6/ as the location for binary packages. 
Is this correct? Is there a plan to publish packages on packagecloud again?

Thanks and regards,
Hank

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


--
Steve Roberts
+44-7887-852-920


Erlang Solutions cares about your data and privacy; please find all details 
about the basis for communicating
with you and the way we process your data in our Privacy Policy 
(https://www.erlang-solutions.com/privacy-policy.html)

You can update your email preferences or opt-out from receiving Marketing 
emails (http://www2.erlang-solutions.com/emailpreference)


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

RE: Looking for the official repository for riak packages

[Nicholas Adams]

Hi Hank,
We are looking into bringing new packages onto packagecloud but when this will 
become available is not confirmed. In the meantime, TI Tokyo will continue to 
make new packages available on https://files.tiot.jp/riak/ . There are plans 
underway to make everything a little more transparent but we will have to wait 
a little longer for them to come to fruition.

Best regards,

Nicholas

-Original Message-
From: riak-users  On Behalf Of Henning 
Verbeek
Sent: 10 January 2019 20:42
To: riak-users 
Subject: Looking for the official repository for riak packages

Hello,

I'm trying to understand what is the plan for official riak releases and where 
to find packages.

In the 2.2.6 announcement, Nicholas Adams provided 
https://files.tiot.jp/riak/kv/2.2/2.2.6/ as the location for binary packages. 
Is this correct? Is there a plan to publish packages on packagecloud again?

Thanks and regards,
Hank

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: High disk usage on node

[István]

Hi,

64 is a bit low, I guess 128 would be better to avoid such situation.

I.

On Thu, Nov 1, 2018 at 2:26 PM Travis Kirstine <
tkirst...@firstbasesolutions.com> wrote:

> I’m running riak (v2.14) in a 5 node cluster and for some reason one of
> the nodes has higher disk usage than the other nodes.  The problem seems to
> be related to how riak distributes the partitions, in my case I’m using the
> default 64, riak has given each node 12 partition except one node that gets
> 16 (4x12+16=64).  As a result the node with 16 partitions has filled the
> disk and become ‘un-reachable’.
>
>
>
> I have a node on standby with roughly the same disk space as the failed
> node, my concern is that if a add it to the cluster it will overflow as
> well.
>
>
>
> How do I recover the failed node and add a new node without destroying the
> cluster….. BTW just to make things more fun the new node is at a newer
> version of riak so I need to perform a rolling upgrade at the same time.
>
>
>
> Any help would be greatly appreciated!
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>


-- 
the sun shines for all
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

RE: High disk usage on node

[Travis Kirstine]

Fred / Nicholas,

Thanks for your input!  We are running riak (leveldb with AAE) inside of VMs, 
as a result we may be able to simply copy the VM to a new server and add more 
storage to the logical volume.  So our plan would be to:


-  Shutdown the failed node

-  Copy the VM and allocate storage

-  Restart the node

-  Wait for transfers to complete.


Just to be clear riak will only redistribute the partitions if the failed node 
is removed from the cluster?  My worry is that it may take 3-4 days to copy the 
VM at which time the failed riak node would be offline, if riak attempts to 
redistribute the partitions all nodes would fill and it would be a nightmare!

Thanks

From: riak-users [mailto:riak-users-boun...@lists.basho.com] On Behalf Of Fred 
Dushin
Sent: Thursday, November 01, 2018 10:30 AM
To: riak-users@lists.basho.com
Subject: Re: High disk usage on node

I think your best bet is to do a force-replace (and then a manual repair, if 
you are not using AAE) with a node that has higher capacity than your current 
standby.  You are correct that replacing with your standby will fail when you 
run repairs and end up running out of space.

I think you do NOT want to do a remove (or force remove) of the node that has 
run out of space, as you will likely just end up pushing those partitions to 
other nodes that are already stressed for capacity.

I would not tempt fate by doing an add on a cluster with a node that is down 
for the count.  Fix that node first, and then add capacity to the cluster.  But 
others here might have more experience with expanding clusters that are in ill 
health.  For example, it might be possible to do a replace without a manual 
repair (which will leave a bunch of near-empty partitions on your new node), 
and then do an add with the node you took out of the cluster.  You would then 
need to track down where all the partitions got moved to in the cluster, and 
then do a manual repair of those partitions.  (Or I suppose if you are using 
AAE, wait for a full set of exchanges to complete, but if were me I'd turn off 
AAE and run the repairs manually, so you can monitor which partitions actually 
got repaired.)

If you are moving to the latest version of Riak (2.2.6), then you may find that 
the v2 claim algorithm does a better job of distributing (ownership) partitions 
more evenly around the cluster, but you should upgrade all your nodes first 
(before running your cap add).

Also make use of `riak_core_claim_sim:help()` in the Riak console.  You can get 
pretty good insight into what your cap adds will look like, without having to 
do a cluster plan.  As Drew has mentioned, you might not want to try the 
claim_v3 algorithms if you are on an already under-provisioned cluster, as 
claim_v3 may move partitions around to already heavily-loaded nodes as part of 
its ownership handoff, which in your case could be catastrophic, as you 
currently have keys that have only two replicas (and presumably are already 
growing in size with secondary partitions), so losing a node or two in the rest 
of your cluster due to disk space could result in data loss (4eva).

-Fred

PS> I see Sick has posted a temporary workaround if you can add extra disk 
capacity to your node (NAS, etc), and that seems like a good route to go, if 
you can do that.  Otherwise, I'd find a machine with more disk and do the 
replace as described above.


On Nov 1, 2018, at 9:25 AM, Travis Kirstine 
mailto:tkirst...@firstbasesolutions.com>> 
wrote:

I’m running riak (v2.14) in a 5 node cluster and for some reason one of the 
nodes has higher disk usage than the other nodes.  The problem seems to be 
related to how riak distributes the partitions, in my case I’m using the 
default 64, riak has given each node 12 partition except one node that gets 16 
(4x12+16=64).  As a result the node with 16 partitions has filled the disk and 
become ‘un-reachable’.

I have a node on standby with roughly the same disk space as the failed node, 
my concern is that if a add it to the cluster it will overflow as well.

How do I recover the failed node and add a new node without destroying the 
cluster….. BTW just to make things more fun the new node is at a newer version 
of riak so I need to perform a rolling upgrade at the same time.

Any help would be greatly appreciated!
___
riak-users mailing list
riak-users@lists.basho.com<mailto:riak-users@lists.basho.com>
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: High disk usage on node

[Fred Dushin]

I think your best bet is to do a force-replace (and then a manual repair, if 
you are not using AAE) with a node that has higher capacity than your current 
standby.  You are correct that replacing with your standby will fail when you 
run repairs and end up running out of space.

I think you do NOT want to do a remove (or force remove) of the node that has 
run out of space, as you will likely just end up pushing those partitions to 
other nodes that are already stressed for capacity.

I would not tempt fate by doing an add on a cluster with a node that is down 
for the count.  Fix that node first, and then add capacity to the cluster.  But 
others here might have more experience with expanding clusters that are in ill 
health.  For example, it might be possible to do a replace without a manual 
repair (which will leave a bunch of near-empty partitions on your new node), 
and then do an add with the node you took out of the cluster.  You would then 
need to track down where all the partitions got moved to in the cluster, and 
then do a manual repair of those partitions.  (Or I suppose if you are using 
AAE, wait for a full set of exchanges to complete, but if were me I'd turn off 
AAE and run the repairs manually, so you can monitor which partitions actually 
got repaired.)

If you are moving to the latest version of Riak (2.2.6), then you may find that 
the v2 claim algorithm does a better job of distributing (ownership) partitions 
more evenly around the cluster, but you should upgrade all your nodes first 
(before running your cap add).

Also make use of `riak_core_claim_sim:help()` in the Riak console.  You can get 
pretty good insight into what your cap adds will look like, without having to 
do a cluster plan.  As Drew has mentioned, you might not want to try the 
claim_v3 algorithms if you are on an already under-provisioned cluster, as 
claim_v3 may move partitions around to already heavily-loaded nodes as part of 
its ownership handoff, which in your case could be catastrophic, as you 
currently have keys that have only two replicas (and presumably are already 
growing in size with secondary partitions), so losing a node or two in the rest 
of your cluster due to disk space could result in data loss (4eva).

-Fred

PS> I see Sick has posted a temporary workaround if you can add extra disk 
capacity to your node (NAS, etc), and that seems like a good route to go, if 
you can do that.  Otherwise, I'd find a machine with more disk and do the 
replace as described above.

> On Nov 1, 2018, at 9:25 AM, Travis Kirstine 
>  wrote:
> 
> I’m running riak (v2.14) in a 5 node cluster and for some reason one of the 
> nodes has higher disk usage than the other nodes.  The problem seems to be 
> related to how riak distributes the partitions, in my case I’m using the 
> default 64, riak has given each node 12 partition except one node that gets 
> 16 (4x12+16=64).  As a result the node with 16 partitions has filled the disk 
> and become ‘un-reachable’.
>  
> I have a node on standby with roughly the same disk space as the failed node, 
> my concern is that if a add it to the cluster it will overflow as well.
>  
> How do I recover the failed node and add a new node without destroying the 
> cluster….. BTW just to make things more fun the new node is at a newer 
> version of riak so I need to perform a rolling upgrade at the same time.
>  
> Any help would be greatly appreciated!
> ___
> riak-users mailing list
> riak-users@lists.basho.com 
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com 
> 
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

RE: High disk usage on node

[Nicholas Adams]

Hi Travis,
I see you have encountered the disk space pickle. Just for the record, the 
safest way to run Riak in production is to keep all resources (CPU, RAM, 
Network, Disk Space, I/O etc.) below 70% utilization on all nodes at all times. 
The reason behind this is to compensate for when one or more nodes go down. 
When this does happen, the remaining nodes have to carry all the load of the 
offline node(s) on top of their existing load and therefore need to have 
sufficient free resources available to do it. If you are running right on the 
limit for any resource then you need to expect issues like this or worse to 
happen on a regular basis.

Potential initial prevention
When you join all the nodes together to form your cluster, you get to run 
`riak-admin cluster plan` and it will show you how things will turn out. If you 
like this plan, run `riak-admin cluster commit` and partitions will be moved 
around accordingly. If not, you can cancel the plan and generate a new one and 
keep doing so until you are happy with the distribution. Sometimes with 
unfortunate divisions of partitions/nodes/ring size then one node gets its fair 
share and then some but often a replan will make it as painless as possible.

Temporary escape from current situation
Before beginning here, let me mention that this method does have the potential 
to go horribly wrong so proceed at your own risk. With regards to the server 
with the filled disk, you can follow the method underneath:

  *   Stop Riak
  *   Attach additional storage (USB, additional disks, NAS, whatever)
  *   Copy partitions from the data directory of, presumably bitcask, to the 
additional storage
  *   Once the copy has been completed, delete the data from the regular node's 
hard disk
  *   Create a symlink from the external storage to where you just deleted the 
data from
  *   Repeat until you have freed up sufficient disk space (new stuff may be 
copied here, so make sure you do have enough space)
  *   Start Riak

The above should bring your server back in touch with the cluster. Monitor 
transfers and once they have all finished, add your new node to the cluster. 
After this new node has been added and all transfers have finished, take the 
previously full node offline and reverse the steps above until you are able to 
remove the additional storage.

Note: running a mixed version cluster for a prolonged period of time is not 
recommended in production. Out of preference, I would suggest installing the 
same version of Riak on the new node, going through the above and then looking 
at upgrading the cluster once everything is stable.

Good luck,

Nicholas
From: riak-users  On Behalf Of Travis 
Kirstine
Sent: 01 November 2018 22:25
To: riak-users@lists.basho.com
Subject: High disk usage on node

I'm running riak (v2.14) in a 5 node cluster and for some reason one of the 
nodes has higher disk usage than the other nodes.  The problem seems to be 
related to how riak distributes the partitions, in my case I'm using the 
default 64, riak has given each node 12 partition except one node that gets 16 
(4x12+16=64).  As a result the node with 16 partitions has filled the disk and 
become 'un-reachable'.

I have a node on standby with roughly the same disk space as the failed node, 
my concern is that if a add it to the cluster it will overflow as well.

How do I recover the failed node and add a new node without destroying the 
cluster. BTW just to make things more fun the new node is at a newer 
version of riak so I need to perform a rolling upgrade at the same time.

Any help would be greatly appreciated!
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

RE: RIAK Manchester Workshop

[Martin Sumner]

Andrew,

It would be good at the Manchester workshop to talk about some of the work
we've been doing for the next release.

- A short reminder of the bugs and fixes that made it into 2.2.5 and 2.2.6;
- Report on the testing and production-readiness of leveled - Quviq's work
on property-based testing leveled, riak_test and latest volume testing
results
- FSM and kv_vnode changes to utilise HEAD requests
- Overview of TictacAAE and its production readiness as an alternative
anti-entropy mechanism

The above are all items, that at the NHS that we're looking to build
urgently into a release for our production systems.  We could talk for a
long time about this, but hopefully can hold it down to one hour.

There's then some other items we've been working on, but aren't quite
release-ready at the moment:

- Soft-limits (aggressive vnode queue monitoring)
- Mapfold (range-based object folds)
- End-to-end repl replacement (and questions about whether some of the repl
control logic should be removed from such replacements)

There would be at least 30 minutes required to go through these changes for
information and feedback.

Another agenda item we would like, would be to have a discussion on how to
get the OTP 20/21 migration moving again.

Regards

Martin
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Node Recovery Questions

[sean mcevoy]

Hi Martin, List,

Just an update to let ye know how things went and what we learned.

We did the force-replace procedure to bring the new node into the cluster
in place of the old one. I attached to the riak erlang shell and with a
little hacking was able to get all the bitcask handles and then do a
bitcask:fold/3 to count keys. This showed that only a small percentage of
all keys were present on the new node, even after the handoffs and
transfers had completed.

Following the instructions at the bottom of this page:
https://docs.basho.com/riak/kv/2.2.0/using/repair-recovery/repairs/
I attached to the erlang shell again and ran these commands (replacing the
IP with our actual IP) to force repairs on all vnodes:

{ok, Ring} = riak_core_ring_manager:get_my_ring().
Partitions = [P || {P, 'dev1@127.0.0.1'} <-
riak_core_ring:all_owners(Ring)].
[riak_kv_vnode:repair(P) || P <- Partitions].

The progress was most easily monitored with: riak-admin handoff summary
and once complete the new node had the expected number of keys.

Counting the keys is more than a bit hacky and occasionally caused a seg
fault if there was background traffic, so I don't recommend it in general.
But it did allow us to verify where the data was in our test env and then
we could trust the procedure without counting keys in production.
Monitoring the size of the bitcask directory is a lot lower resolution but
it is at least safe, the results were similar in test & production so it
was sufficient to verify the above procedure.

So in short, when replacing a node the force-replace procedure doesn't
actually cause data to be synched to the new node. The above erlang shell
commands do force a sync.

Thanks for the support!
//Sean.

On Thu, Aug 9, 2018 at 11:25 PM sean mcevoy  wrote:

> Hi Martin,
> Thanks for taking the time.
> Yes, by "size of the bitcask directory" I mean I did a "du -h
> --max-depth=1 bitcask", so I think that would cover all the vnodes. We
> don't use any other backends.
> Those answers are helpful, will get back to this in a few days and see
> what I can determine about where our data physically lies. Might have more
> questions then.
> Cheers,
> //Sean.
>
> On Wed, Aug 8, 2018 at 6:05 PM, Martin Sumner  > wrote:
>
>> Based on a quick read of the code, compaction in bitcask is performed
>> only on "readable" files, and the current active file for writing is
>> excluded from that list.  With default settings, that active file can grow
>> to 2GB.  So it is possible that if objects had been replaced/deleted many
>> times within the active file, that space will not be recovered if all the
>> replacements amount to < 2GB per vnode.  So at these small data sizes - you
>> may get a relatively significant discrepancy between an old and recovered
>> node in terms of disk space usage.
>>
>> On 8 August 2018 at 17:37, Martin Sumner 
>> wrote:
>>
>>> Sean,
>>>
>>> Some partial answers to your questions.
>>>
>>> I don't believe force-replace itself will sync anything up - it just
>>> reassigns ownership (hence handoff happens very quickly).
>>>
>>> Read repair would synchronise a portion of the data.  So if 10% of you
>>> data is read regularly, this might explain some of what you see.
>>>
>>> AAE should also repair your data.  But if nothing has happened for 4
>>> days, then that doesn't seem to be the case.  It would be worth checking
>>> the aae-status page (
>>> http://docs.basho.com/riak/kv/2.2.3/using/admin/riak-admin/#aae-status)
>>> to confirm things are happening.
>>>
>>> I don't know if there are any minimum levels of data before bitcask will
>>> perform compaction.  There's nothing obvious in the code that wouldn't be
>>> triggered way before 90%.  I don't know if it will merge on the active file
>>> (the one currently being written to), but that is 2GB max size (configured
>>> through bitcask.max_file_size).
>>>
>>> When you say the size of the bitcask directory - is this the size shared
>>> across all vnodes on the node?  I guess if each vnode has a single file
>>> <2GB, and there are multiple vnodes - something unexpected might happen
>>> here?  If bitcask does indeed not merge the file active for writing.
>>>
>>> In terms of distribution around the cluster, if you have an n_val of 3
>>> you should normally expect to see a relatively even distribution of the
>>> data on failure (certainly not it all going to one).  Worst case scenario
>>> is that 3 nodes get all the load from that one failed node.
>>>
>>> When a vnode is inaccessible, 3 (assuming n=3) fallback vnodes are
>>> selected to handle the load for that 1 vnode (as that vnode would normally
>>> be in 3 preflists, and commonly a different node will be asked to start a
>>> vnode for each preflist).
>>>
>>>
>>> I will try and dig later into bitcask merge/compaction code, to see if I
>>> spot anything else.
>>>
>>> Martin
>>>
>>>
>>>
>>
>
___
riak-users mailing list
riak-users@lists.basho.com

Re: Node Recovery Questions

[sean mcevoy]

Hi Martin,
Thanks for taking the time.
Yes, by "size of the bitcask directory" I mean I did a "du -h --max-depth=1
bitcask", so I think that would cover all the vnodes. We don't use any
other backends.
Those answers are helpful, will get back to this in a few days and see what
I can determine about where our data physically lies. Might have more
questions then.
Cheers,
//Sean.

On Wed, Aug 8, 2018 at 6:05 PM, Martin Sumner 
wrote:

> Based on a quick read of the code, compaction in bitcask is performed only
> on "readable" files, and the current active file for writing is excluded
> from that list.  With default settings, that active file can grow to 2GB.
> So it is possible that if objects had been replaced/deleted many times
> within the active file, that space will not be recovered if all the
> replacements amount to < 2GB per vnode.  So at these small data sizes - you
> may get a relatively significant discrepancy between an old and recovered
> node in terms of disk space usage.
>
> On 8 August 2018 at 17:37, Martin Sumner 
> wrote:
>
>> Sean,
>>
>> Some partial answers to your questions.
>>
>> I don't believe force-replace itself will sync anything up - it just
>> reassigns ownership (hence handoff happens very quickly).
>>
>> Read repair would synchronise a portion of the data.  So if 10% of you
>> data is read regularly, this might explain some of what you see.
>>
>> AAE should also repair your data.  But if nothing has happened for 4
>> days, then that doesn't seem to be the case.  It would be worth checking
>> the aae-status page (http://docs.basho.com/riak/kv
>> /2.2.3/using/admin/riak-admin/#aae-status) to confirm things are
>> happening.
>>
>> I don't know if there are any minimum levels of data before bitcask will
>> perform compaction.  There's nothing obvious in the code that wouldn't be
>> triggered way before 90%.  I don't know if it will merge on the active file
>> (the one currently being written to), but that is 2GB max size (configured
>> through bitcask.max_file_size).
>>
>> When you say the size of the bitcask directory - is this the size shared
>> across all vnodes on the node?  I guess if each vnode has a single file
>> <2GB, and there are multiple vnodes - something unexpected might happen
>> here?  If bitcask does indeed not merge the file active for writing.
>>
>> In terms of distribution around the cluster, if you have an n_val of 3
>> you should normally expect to see a relatively even distribution of the
>> data on failure (certainly not it all going to one).  Worst case scenario
>> is that 3 nodes get all the load from that one failed node.
>>
>> When a vnode is inaccessible, 3 (assuming n=3) fallback vnodes are
>> selected to handle the load for that 1 vnode (as that vnode would normally
>> be in 3 preflists, and commonly a different node will be asked to start a
>> vnode for each preflist).
>>
>>
>> I will try and dig later into bitcask merge/compaction code, to see if I
>> spot anything else.
>>
>> Martin
>>
>>
>>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Node Recovery Questions

[Martin Sumner]

Based on a quick read of the code, compaction in bitcask is performed only
on "readable" files, and the current active file for writing is excluded
from that list.  With default settings, that active file can grow to 2GB.
So it is possible that if objects had been replaced/deleted many times
within the active file, that space will not be recovered if all the
replacements amount to < 2GB per vnode.  So at these small data sizes - you
may get a relatively significant discrepancy between an old and recovered
node in terms of disk space usage.

On 8 August 2018 at 17:37, Martin Sumner 
wrote:

> Sean,
>
> Some partial answers to your questions.
>
> I don't believe force-replace itself will sync anything up - it just
> reassigns ownership (hence handoff happens very quickly).
>
> Read repair would synchronise a portion of the data.  So if 10% of you
> data is read regularly, this might explain some of what you see.
>
> AAE should also repair your data.  But if nothing has happened for 4 days,
> then that doesn't seem to be the case.  It would be worth checking the
> aae-status page (http://docs.basho.com/riak/kv/2.2.3/using/admin/riak-
> admin/#aae-status) to confirm things are happening.
>
> I don't know if there are any minimum levels of data before bitcask will
> perform compaction.  There's nothing obvious in the code that wouldn't be
> triggered way before 90%.  I don't know if it will merge on the active file
> (the one currently being written to), but that is 2GB max size (configured
> through bitcask.max_file_size).
>
> When you say the size of the bitcask directory - is this the size shared
> across all vnodes on the node?  I guess if each vnode has a single file
> <2GB, and there are multiple vnodes - something unexpected might happen
> here?  If bitcask does indeed not merge the file active for writing.
>
> In terms of distribution around the cluster, if you have an n_val of 3 you
> should normally expect to see a relatively even distribution of the data on
> failure (certainly not it all going to one).  Worst case scenario is that 3
> nodes get all the load from that one failed node.
>
> When a vnode is inaccessible, 3 (assuming n=3) fallback vnodes are
> selected to handle the load for that 1 vnode (as that vnode would normally
> be in 3 preflists, and commonly a different node will be asked to start a
> vnode for each preflist).
>
>
> I will try and dig later into bitcask merge/compaction code, to see if I
> spot anything else.
>
> Martin
>
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

RE: Node Recovery Questions

[Martin Sumner]

Sean,

Some partial answers to your questions.

I don't believe force-replace itself will sync anything up - it just
reassigns ownership (hence handoff happens very quickly).

Read repair would synchronise a portion of the data.  So if 10% of you data
is read regularly, this might explain some of what you see.

AAE should also repair your data.  But if nothing has happened for 4 days,
then that doesn't seem to be the case.  It would be worth checking the
aae-status page (
http://docs.basho.com/riak/kv/2.2.3/using/admin/riak-admin/#aae-status) to
confirm things are happening.

I don't know if there are any minimum levels of data before bitcask will
perform compaction.  There's nothing obvious in the code that wouldn't be
triggered way before 90%.  I don't know if it will merge on the active file
(the one currently being written to), but that is 2GB max size (configured
through bitcask.max_file_size).

When you say the size of the bitcask directory - is this the size shared
across all vnodes on the node?  I guess if each vnode has a single file
<2GB, and there are multiple vnodes - something unexpected might happen
here?  If bitcask does indeed not merge the file active for writing.

In terms of distribution around the cluster, if you have an n_val of 3 you
should normally expect to see a relatively even distribution of the data on
failure (certainly not it all going to one).  Worst case scenario is that 3
nodes get all the load from that one failed node.

When a vnode is inaccessible, 3 (assuming n=3) fallback vnodes are selected
to handle the load for that 1 vnode (as that vnode would normally be in 3
preflists, and commonly a different node will be asked to start a vnode for
each preflist).


I will try and dig later into bitcask merge/compaction code, to see if I
spot anything else.

Martin
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Failed to start riak_kv_multi_backend

[Bryan Hunt]

Amol,

Although a repair tool exists for eleveldb the simplest solutions is to simply 
delete the corrupted partition and run a repair job. 

1. Stop the node

2. Locate the directory you have configured for eleveldb data storage and 
delete the subdirectory 479555224749202520035584085735030365824602865664

3. Start the node

4. Follow these instructions : 
https://docs.basho.com/riak/kv/2.0.0/using/repair-recovery/repairs/#repairing-partitions

PS - It’s better to run a cluster with 5 nodes. 

Best Regards,

Bryan Hunt



> On 12 Jul 2018, at 14:32, Amol Zambare(HO)  
> wrote:
> 
> Hello all,
> 
> We have 4 node cluster, one node is unable to start because of below error
> 
> Failed to start riak_kv_multi_backend backend for index 
> 479555224749202520035584085735030365824602865664 error: 
> [{riak_kv_eleveldb_backend,{db_open,"Corruption: truncated record at end of 
> file"}}]
> 
> 
> we have try to start after deleting anti entropy still getting same error, 
> kindly guild us is there any way to repair this node.
> 
> Thanks. 
> 
> 
> *
> 
> PRIVILEGED AND CONFIDENTIAL COMMUNICATION 
> 
> This e-mail transmission, and any documents, files or previous e-mail 
> messages attached to it, may contain confidential information that is legally 
> privileged. If you are not the intended recipient or a person responsible for 
> delivering it to the intended recipient, you are hereby notified that any 
> disclosure, copying, distribution or use of any of the information contained 
> in or attached to this transmission is STRICTLY PROHIBITED. If you have 
> received this transmission in error, please: 
> (1) immediately notify me by reply e-mail, or by telephone call; and 
> (2) destroy the original transmission and its attachments without reading or 
> saving in any manner.
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: rolling upgrades riak kv OSS 2.14 to 2.2.3

[Fred Dushin]

There are some changes to the format of AAE tree data, if you have AAE enabled. 
 IIRC, AAE tree upgrades will not kick in until all nodes in the cluster are 
upgraded to 2.2 or later, but they will kick in automatically, once full 
upgrade is detected.  You can disable the new AAE format, but this is not 
recommended.

http://docs.basho.com/riak/kv/2.2.3/setup/upgrading/version/

-Fred

> On Jul 9, 2018, at 2:49 PM, Travis Kirstine 
>  wrote:
> 
> Can I perform a rolling upgrade as described in the docs
>  
> -  Stop riak
> -  Backup /etc /data /basho-patches
> -  Remove /basho-patches
> -  Upgrade from 2.14 -> 2.2.3
> -  Start riak
> -  Verify the version 
> -  Wait for the service to start and complete hinted handoffs
>  
> Thanks 
> ___
> riak-users mailing list
> riak-users@lists.basho.com 
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com 
> 
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Partitions repair isn't working anymore?

[Russell Brown]

> On 1 Jun 2018, at 10:07, Guido Medina  wrote:
> 
> The last command should have been like the following:
>   • {ok, Ring} = riak_core_ring_manager:get_my_ring().
>   • Partitions = [P || {P, 'r...@node4.domain.com'} <- 
> riak_core_ring:all_owners(Ring)].
>   • [riak_kv_vnode:repair(P) || P <- Partitions].
> My bad, I think I copied the command from the wrong instructions before.

And does that work?

From your mail you said that 
>>> )3> [riak_search_vnode:repair(P) || P <- Partitions].
>>> ** exception error: undefined function riak_search_vnode:repair/1
>>> 

Does not work. This I would expect, and I was asking, why do you run this? Are 
you using legacy search?

If you run 

> • [riak_kv_vnode:repair(P) || P <- Partitions].

Does it work?

Cheers

Russell

> 
> Guido.
> 
> On 01/06/18 09:37, Russell Brown wrote:
>> I don’t see a call to `riak_search_vnode:repair` in those docs
>> 
>> Do you still run legacy riak search (i.e. not yokozuna/solr)?
>> 
>> 
>>> On 1 Jun 2018, at 09:35, Guido Medina 
>>>  wrote:
>>> 
>>> Sorry, not repairing a single partition but all partitions per node:
>>> 
>>> https://docs.basho.com/riak/kv/2.2.3/using/repair-recovery/repairs/#repairing-all-partitions-on-a-node
>>> 
>>> 
>>> On 01/06/18 09:34, Guido Medina wrote:
>>> 
 Hi Russell,
 
 I was repairing each node as specified in this guide 
 https://docs.basho.com/riak/kv/2.2.3/using/repair-recovery/repairs/#repairing-a-single-partition
 
 
 Guido.
 
 On 01/06/18 09:16, Russell Brown wrote:
 
> riak_search has been removed from riak-2.2.5. Looks like some vestige 
> survived. 
> 
> Can you tell me what command you ran, it looks to me from the output 
> below that you’re connected to node and typing commands in the console?
> 
> Is this some snippet that you attach and run?
> 
> Cheers
> 
> Russell
> 
> 
> 
>> On 1 Jun 2018, at 09:07, Guido Medina 
>> 
>>  wrote:
>> 
>> Hi all,
>> 
>> We started the partitions repair a couple of weeks ago, so far so good 
>> (3 nodes out of 7 done), then we started getting this error:
>> 
>> 
>>> (r...@node4.domain.com
>>> 
>>> )3> [riak_search_vnode:repair(P) || P <- Partitions].
>>> ** exception error: undefined function riak_search_vnode:repair/1
>>> 
>>> 
>> The first two steps for the node repair executed fine:
>> 
>> 
>>> {ok, Ring} = riak_core_ring_manager:get_my_ring().
>>> Partitions = [P || {P, '
>>> 
>>> r...@node4.domain.com
>>> 
>>> '} <- riak_core_ring:all_owners(Ring)].
>>> 
>>> 
>> We are running on 2.2.5
>> Guido.
>> ___
>> riak-users mailing list
>> 
>> 
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>> ___
>>> riak-users mailing list
>>> 
>>> riak-users@lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> 
> 


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Partitions repair isn't working anymore?

[Guido Medina]

The last command should have been like the following:

 * {ok, Ring} = riak_core_ring_manager:get_my_ring().
 * Partitions = [P || {P, 'r...@node4.domain.com'} <-
   riak_core_ring:all_owners(Ring)].
 * *[riak_kv_vnode:repair(P) || P <- Partitions].*

My bad, I think I copied the command from the wrong instructions before.

Guido.

On 01/06/18 09:37, Russell Brown wrote:

I don’t see a call to `riak_search_vnode:repair` in those docs

Do you still run legacy riak search (i.e. not yokozuna/solr)?


On 1 Jun 2018, at 09:35, Guido Medina  wrote:

Sorry, not repairing a single partition but all partitions per node:
https://docs.basho.com/riak/kv/2.2.3/using/repair-recovery/repairs/#repairing-all-partitions-on-a-node

On 01/06/18 09:34, Guido Medina wrote:

Hi Russell,

I was repairing each node as specified in this guide 
https://docs.basho.com/riak/kv/2.2.3/using/repair-recovery/repairs/#repairing-a-single-partition

Guido.

On 01/06/18 09:16, Russell Brown wrote:

riak_search has been removed from riak-2.2.5. Looks like some vestige survived.

Can you tell me what command you ran, it looks to me from the output below that 
you’re connected to node and typing commands in the console?

Is this some snippet that you attach and run?

Cheers

Russell



On 1 Jun 2018, at 09:07, Guido Medina 
  wrote:

Hi all,

We started the partitions repair a couple of weeks ago, so far so good (3 nodes 
out of 7 done), then we started getting this error:


(r...@node4.domain.com
)3> [riak_search_vnode:repair(P) || P <- Partitions].
** exception error: undefined function riak_search_vnode:repair/1


The first two steps for the node repair executed fine:


{ok, Ring} = riak_core_ring_manager:get_my_ring().
Partitions = [P || {P, '
r...@node4.domain.com
'} <- riak_core_ring:all_owners(Ring)].


We are running on 2.2.5
Guido.
___
riak-users mailing list

riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Partitions repair isn't working anymore?

[Guido Medina]

What did I copied and pasted, sorry about that, it must be that it is 
Friday or something...my apologies.


On 01/06/18 09:37, Russell Brown wrote:

I don’t see a call to `riak_search_vnode:repair` in those docs

Do you still run legacy riak search (i.e. not yokozuna/solr)?


On 1 Jun 2018, at 09:35, Guido Medina  wrote:

Sorry, not repairing a single partition but all partitions per node:
https://docs.basho.com/riak/kv/2.2.3/using/repair-recovery/repairs/#repairing-all-partitions-on-a-node

On 01/06/18 09:34, Guido Medina wrote:

Hi Russell,

I was repairing each node as specified in this guide 
https://docs.basho.com/riak/kv/2.2.3/using/repair-recovery/repairs/#repairing-a-single-partition

Guido.

On 01/06/18 09:16, Russell Brown wrote:

riak_search has been removed from riak-2.2.5. Looks like some vestige survived.

Can you tell me what command you ran, it looks to me from the output below that 
you’re connected to node and typing commands in the console?

Is this some snippet that you attach and run?

Cheers

Russell



On 1 Jun 2018, at 09:07, Guido Medina 
  wrote:

Hi all,

We started the partitions repair a couple of weeks ago, so far so good (3 nodes 
out of 7 done), then we started getting this error:


(r...@node4.domain.com
)3> [riak_search_vnode:repair(P) || P <- Partitions].
** exception error: undefined function riak_search_vnode:repair/1


The first two steps for the node repair executed fine:


{ok, Ring} = riak_core_ring_manager:get_my_ring().
Partitions = [P || {P, '
r...@node4.domain.com
'} <- riak_core_ring:all_owners(Ring)].


We are running on 2.2.5
Guido.
___
riak-users mailing list

riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Partitions repair isn't working anymore?

[Russell Brown]

I don’t see a call to `riak_search_vnode:repair` in those docs

Do you still run legacy riak search (i.e. not yokozuna/solr)?

> On 1 Jun 2018, at 09:35, Guido Medina  wrote:
> 
> Sorry, not repairing a single partition but all partitions per node:
> https://docs.basho.com/riak/kv/2.2.3/using/repair-recovery/repairs/#repairing-all-partitions-on-a-node
> 
> On 01/06/18 09:34, Guido Medina wrote:
>> Hi Russell,
>> 
>> I was repairing each node as specified in this guide 
>> https://docs.basho.com/riak/kv/2.2.3/using/repair-recovery/repairs/#repairing-a-single-partition
>> 
>> Guido.
>> 
>> On 01/06/18 09:16, Russell Brown wrote:
>>> riak_search has been removed from riak-2.2.5. Looks like some vestige 
>>> survived. 
>>> 
>>> Can you tell me what command you ran, it looks to me from the output below 
>>> that you’re connected to node and typing commands in the console?
>>> 
>>> Is this some snippet that you attach and run?
>>> 
>>> Cheers
>>> 
>>> Russell
>>> 
>>> 
 On 1 Jun 2018, at 09:07, Guido Medina 
  wrote:
 
 Hi all,
 
 We started the partitions repair a couple of weeks ago, so far so good (3 
 nodes out of 7 done), then we started getting this error:
 
> (r...@node4.domain.com
> )3> [riak_search_vnode:repair(P) || P <- Partitions].
> ** exception error: undefined function riak_search_vnode:repair/1
> 
 The first two steps for the node repair executed fine:
 
> {ok, Ring} = riak_core_ring_manager:get_my_ring().
> Partitions = [P || {P, '
> r...@node4.domain.com
> '} <- riak_core_ring:all_owners(Ring)].
> 
 We are running on 2.2.5
 Guido.
 ___
 riak-users mailing list
 
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> 
> 
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Partitions repair isn't working anymore?

[Guido Medina]

Sorry, not repairing a single partition but all partitions per node:
https://docs.basho.com/riak/kv/2.2.3/using/repair-recovery/repairs/#repairing-all-partitions-on-a-node

On 01/06/18 09:34, Guido Medina wrote:

Hi Russell,

I was repairing each node as specified in this guide 
https://docs.basho.com/riak/kv/2.2.3/using/repair-recovery/repairs/#repairing-a-single-partition


Guido.

On 01/06/18 09:16, Russell Brown wrote:

riak_search has been removed from riak-2.2.5. Looks like some vestige survived.

Can you tell me what command you ran, it looks to me from the output below that 
you’re connected to node and typing commands in the console?

Is this some snippet that you attach and run?

Cheers

Russell


On 1 Jun 2018, at 09:07, Guido Medina  wrote:

Hi all,

We started the partitions repair a couple of weeks ago, so far so good (3 nodes 
out of 7 done), then we started getting this error:

(r...@node4.domain.com)3> [riak_search_vnode:repair(P) || P <- Partitions].
** exception error: undefined function riak_search_vnode:repair/1

The first two steps for the node repair executed fine:

{ok, Ring} = riak_core_ring_manager:get_my_ring().
Partitions = [P || {P, 'r...@node4.domain.com'} <- 
riak_core_ring:all_owners(Ring)].

We are running on 2.2.5
Guido.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Partitions repair isn't working anymore?

[Guido Medina]

Hi Russell,

I was repairing each node as specified in this guide 
https://docs.basho.com/riak/kv/2.2.3/using/repair-recovery/repairs/#repairing-a-single-partition


Guido.

On 01/06/18 09:16, Russell Brown wrote:

riak_search has been removed from riak-2.2.5. Looks like some vestige survived.

Can you tell me what command you ran, it looks to me from the output below that 
you’re connected to node and typing commands in the console?

Is this some snippet that you attach and run?

Cheers

Russell


On 1 Jun 2018, at 09:07, Guido Medina  wrote:

Hi all,

We started the partitions repair a couple of weeks ago, so far so good (3 nodes 
out of 7 done), then we started getting this error:

(r...@node4.domain.com)3> [riak_search_vnode:repair(P) || P <- Partitions].
** exception error: undefined function riak_search_vnode:repair/1

The first two steps for the node repair executed fine:

{ok, Ring} = riak_core_ring_manager:get_my_ring().
Partitions = [P || {P, 'r...@node4.domain.com'} <- 
riak_core_ring:all_owners(Ring)].

We are running on 2.2.5
Guido.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

RE: Partitions repair isn't working anymore?

[Nicholas Adams]

From the mail history, I'd guess he's following the standard repair process 
listed on 
https://docs.basho.com/riak/kv/2.1.4/using/repair-recovery/repairs/#repairing-all-partitions-on-a-node


-Original Message-
From: riak-users  On Behalf Of Russell Brown
Sent: 01 June 2018 17:16
To: Guido Medina 
Cc: riak-users 
Subject: Re: Partitions repair isn't working anymore?

riak_search has been removed from riak-2.2.5. Looks like some vestige survived. 

Can you tell me what command you ran, it looks to me from the output below that 
you’re connected to node and typing commands in the console?

Is this some snippet that you attach and run?

Cheers

Russell

> On 1 Jun 2018, at 09:07, Guido Medina  wrote:
> 
> Hi all,
> 
> We started the partitions repair a couple of weeks ago, so far so good (3 
> nodes out of 7 done), then we started getting this error:
>> (r...@node4.domain.com)3> [riak_search_vnode:repair(P) || P <- Partitions].
>> ** exception error: undefined function riak_search_vnode:repair/1
> 
> The first two steps for the node repair executed fine:
>> {ok, Ring} = riak_core_ring_manager:get_my_ring().
>> Partitions = [P || {P, 'r...@node4.domain.com'} <- 
>> riak_core_ring:all_owners(Ring)].
> 
> We are running on 2.2.5
> Guido.
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Partitions repair isn't working anymore?

[Russell Brown]

riak_search has been removed from riak-2.2.5. Looks like some vestige survived. 

Can you tell me what command you ran, it looks to me from the output below that 
you’re connected to node and typing commands in the console?

Is this some snippet that you attach and run?

Cheers

Russell

> On 1 Jun 2018, at 09:07, Guido Medina  wrote:
> 
> Hi all,
> 
> We started the partitions repair a couple of weeks ago, so far so good (3 
> nodes out of 7 done), then we started getting this error:
>> (r...@node4.domain.com)3> [riak_search_vnode:repair(P) || P <- Partitions].
>> ** exception error: undefined function riak_search_vnode:repair/1
> 
> The first two steps for the node repair executed fine:
>> {ok, Ring} = riak_core_ring_manager:get_my_ring().
>> Partitions = [P || {P, 'r...@node4.domain.com'} <- 
>> riak_core_ring:all_owners(Ring)].
> 
> We are running on 2.2.5
> Guido.
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

RE: [ANN] Riak 2.2.6 release

[Nicholas Adams]

Dear All,
Just following up on Russell's email. Packages for KV 2.2.6 are available for 
Debian, FreeBSD, OSX, RedHat and Ubuntu from 
https://files.tiot.jp/riak/kv/2.2/2.2.6/

Best regards,

Nicholas

-Original Message-
From: riak-users  On Behalf Of Russell Brown
Sent: 31 May 2018 18:19
To: riak-users 
Subject: [ANN] Riak 2.2.6 release

Hi,

Riak-2.2.6 has been tagged and will be/is available from the usual outlets (see 
follow up from Nicholas Adams.)

Riak-2.2.6 has no changes/diffences from riak-2.2.5.

When I tagged riak-2.2.5 I did not notice that there already existed a tag 
`2.1.7-225` for riak_kv. I created a new tag with the same name. Thankfully all 
the build/release tools used the new tag, but there is still risk, and 
confusion, due to the duplicate tag on riak_kv.

Our options were to delete both tags, and re-create the latter tag, which 
seemed wrong, or create a new tag for riak_kv (at the exact same
SHA) which is what we have done. The new tag is 2.1.7-226.

Creating this new tag meant updating the rebar.config for yokozuna and 
riak_repl, and riak itself, which led to new SHAs, and new tags for those 
repos. And that is how we came to have a riak-2.2.6 which is exactly the same 
code as riak-2.2.5.

I have updated the release wiki instructions to include a step that checks for 
existing branch/tag with the planned new tag name, so that we don’t do this 
again.

Cheers

Russell



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

RE: Riak One partition handoff stall

[Nicholas Adams]

Dear Gaurav,

Standard troubleshooting – stalled handoffs can often be fixed by “riak-admin 
transfer limit 0” to stop all transfers and once you have confirmed that all 
transfers have stopped, run “riak-admin transfer limit 2” to set it back to the 
default value.

Another one you might want to investigate is repairing the VNode you list. For 
Riak KV 1.4.12, you would refer to the steps listed in 
http://docs.basho.com/riak/1.4.12/ops/running/recovery/repairing-partitions/#Running-a-Repair
 under Repairing a Single Partition and substituting in the VNode value you 
have below.

From my work as a CSE under Basho originally and now under TI Tokyo, can I ask 
why you are regularly getting nodes to leave the cluster? This is not common 
practice in production environments.

Finally, Riak KV 1.4.12 has been obsolete for quite a few years, I would 
strongly recommend that you update to LTS status Riak KV 2.0.9 as that is 
supported as a direct upgrade from 1.4.12 – see 
https://docs.basho.com/riak/kv/2.0.9/setup/upgrading/ for details. Once on the 
2.0.x series, you can then look at a further upgrade to the 2.2.x series should 
you so wish.

Hope this helps,

Nicholas

From: riak-users <riak-users-boun...@lists.basho.com> On Behalf Of Gaurav Sood
Sent: 28 May 2018 22:11
To: Bryan Hunt <bryan.h...@erlang-solutions.com>
Cc: riak-users@lists.basho.com
Subject: Re: Riak One partition handoff stall

Thanks Bryan

Below is the ouput of command riak-admin vnode_status. May be data transfer has 
stopped on the claimant node.

Output of all commands is constant.

1)

 VNode: 342539446249430371453988632667878832731859189760
Backend: riak_kv_eleveldb_backend
Status:
[{stats,<<"   Compactions\nLevel  Files Size(MB) 
Time(sec) Read(MB) 
Write(MB)\n--\n  01 
   0 00 0\n">>},
 {read_block_error,<<"0">>},
 {fixed_indexes,true}]


2) 30GB data per server
4) I am not sure about the number of objects. Is there any way to get the count 
of objects.

On Mon, May 28, 2018 at 4:57 PM, Bryan Hunt 
<bryan.h...@erlang-solutions.com<mailto:bryan.h...@erlang-solutions.com>> wrote:
Are you constantly executing a particular riak command, in your system 
monitoring scripts, for example: `riak-admin vnode-status` ?

What size is your data per server ?

How many objects are you storing ?

---
Erlang Solutions cares about your data and privacy; please find all details 
about the basis for communicating with you and the way we process your data in 
our Privacy Policy.You can update your email preferences or opt-out from 
receiving Marketing emails here.


On 28 May 2018, at 08:29, Gaurav Sood 
<gaurav.s...@mediologysoftware.com<mailto:gaurav.s...@mediologysoftware.com>> 
wrote:

Hi All - Good Day!

I have a 7 Node Raik_KV cluster. Recently I have upgraded this cluster from 
1.4.2  to 1.4.12 on Ubuntu 16.04. After upgrading the cluster whenever I leave 
a node from cluster one partition hand off stalled every time & Active 
transfers shows 'waiting to handoff 1 partitions", to complete this process I 
need to reboot the riak service on all nodes one by one.

I am not sure if it's configuration problem. Here is the current state of 
cluster.

#output of riak-admin member-status
= Membership ==
Status RingPendingNode
---
leaving 0.0%  --  'riak@192.168.2.10<mailto:riak@192.168.2.10>'
valid  14.1%  --  'riak@192.168.2.11<mailto:riak@192.168.2.11>'
valid  14.1%  --  'riak@192.168.2.12<mailto:riak@192.168.2.12>'
valid  15.6%  --  'riak@192.168.2.13<mailto:riak@192.168.2.13>'
valid  14.1%  --  'riak@192.168.2.14<mailto:riak@192.168.2.14>'
valid  14.1%  --  'riak@192.168.2.15<mailto:riak@192.168.2.15>'
valid  14.1%  --  'riak@192.168.2.16<mailto:riak@192.168.2.16>'
valid  14.1%  --  'riak@192.168.2.17<mailto:riak@192.168.2.17>'
---
Valid:7 / Leaving:1 / Exiting:0 / Joining:0 / Down:0
#output of riak-admin transfers

'riak@192.168.2.10<mailto:riak@192.168.2.10>' waiting to handoff 1 partitions

Active Transfers:

(nothing here)


#Output of riak-admin ring_status
== Claimant ===
Claimant:  'riak@192.168.2.10<mailto:riak@192.168.2.10>'
Status: up
Ring Ready: true

== Ownership Handoff ==
No pending changes.

== Unreachable Nodes ==
All nodes are up and reachable

current Transfer Limit is 2.

Tha

Re: Riak One partition handoff stall

[Gaurav Sood]

Thanks Bryan

Below is the ouput of command riak-admin vnode_status. May be data transfer
has stopped on the claimant node.

Output of all commands is constant.

1)

 VNode: 342539446249430371453988632667878832731859189760
Backend: riak_kv_eleveldb_backend
Status:
[{stats,<<"   Compactions\nLevel  Files
Size(MB) Time(sec) Read(MB) Write(MB)\n---
---\n  010 0
0 0\n">>},
 {read_block_error,<<"0">>},
 {fixed_indexes,true}]


2) 30GB data per server
4) I am not sure about the number of objects. Is there any way to get the
count of objects.

On Mon, May 28, 2018 at 4:57 PM, Bryan Hunt  wrote:

> Are you constantly executing a particular riak command, in your system
> monitoring scripts, for example: `riak-admin vnode-status` ?
>
> What size is your data per server ?
>
> How many objects are you storing ?
>
> ---
> Erlang Solutions cares about your data and privacy; please find all
> details about the basis for communicating with you and the way we process
> your data in our Privacy Policy.You can update your email preferences or
> opt-out from receiving Marketing emails here.
>
> On 28 May 2018, at 08:29, Gaurav Sood 
> wrote:
>
> Hi All - Good Day!
>
> I have a 7 Node Raik_KV cluster. Recently I have upgraded this cluster
> from 1.4.2  to 1.4.12 on Ubuntu 16.04. After upgrading the cluster whenever
> I leave a node from cluster one partition hand off stalled every time &
> Active transfers shows 'waiting to handoff 1 partitions", to complete this
> process I need to reboot the riak service on all nodes one by one.
>
> I am not sure if it's configuration problem. Here is the current state of
> cluster.
>
> *#output of riak-admin member-status*
> = Membership
> ==
> Status RingPendingNode
> 
> ---
> leaving 0.0%  --  'riak@192.168.2.10'
> valid  14.1%  --  'riak@192.168.2.11'
> valid  14.1%  --  'riak@192.168.2.12'
> valid  15.6%  --  'riak@192.168.2.13'
> valid  14.1%  --  'riak@192.168.2.14'
> valid  14.1%  --  'riak@192.168.2.15'
> valid  14.1%  --  'riak@192.168.2.16'
> valid  14.1%  --  'riak@192.168.2.17'
> 
> ---
> Valid:7 / Leaving:1 / Exiting:0 / Joining:0 / Down:0
>
> *#output of riak-admin transfers*
>
> 'riak@192.168.2.10' waiting to handoff 1 partitions
>
> Active Transfers:
>
> (nothing here)
>
>
> *#Output of riak-admin ring_status*
> == Claimant ==
> =
> Claimant:  'riak@192.168.2.10'
> Status: up
> Ring Ready: true
>
> == Ownership Handoff
> ==
> No pending changes.
>
> == Unreachable Nodes
> ==
> All nodes are up and reachable
>
> *current Transfer Limit is 2.*
>
> Thanks
> Gaurav
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak One partition handoff stall

[Bryan Hunt]

Are you constantly executing a particular riak command, in your system 
monitoring scripts, for example: `riak-admin vnode-status` ? 

What size is your data per server ? 

How many objects are you storing ? 

---
Erlang Solutions cares about your data and privacy; please find all details 
about the basis for communicating with you and the way we process your data in 
our Privacy Policy.You can update your email preferences or opt-out from 
receiving Marketing emails here.

> On 28 May 2018, at 08:29, Gaurav Sood  
> wrote:
> 
> Hi All - Good Day!
> 
> I have a 7 Node Raik_KV cluster. Recently I have upgraded this cluster from 
> 1.4.2  to 1.4.12 on Ubuntu 16.04. After upgrading the cluster whenever I 
> leave a node from cluster one partition hand off stalled every time & Active 
> transfers shows 'waiting to handoff 1 partitions", to complete this process I 
> need to reboot the riak service on all nodes one by one. 
> 
> I am not sure if it's configuration problem. Here is the current state of 
> cluster.
> 
> #output of riak-admin member-status
> = Membership 
> ==
> Status RingPendingNode
> ---
> leaving 0.0%  --  'riak@192.168.2.10 '
> valid  14.1%  --  'riak@192.168.2.11 '
> valid  14.1%  --  'riak@192.168.2.12 '
> valid  15.6%  --  'riak@192.168.2.13 '
> valid  14.1%  --  'riak@192.168.2.14 '
> valid  14.1%  --  'riak@192.168.2.15 '
> valid  14.1%  --  'riak@192.168.2.16 '
> valid  14.1%  --  'riak@192.168.2.17 '
> ---
> Valid:7 / Leaving:1 / Exiting:0 / Joining:0 / Down:0
> 
> #output of riak-admin transfers
> 
> 'riak@192.168.2.10 ' waiting to handoff 1 partitions
> 
> Active Transfers:
> 
> (nothing here)
> 
> 
> #Output of riak-admin ring_status
> == Claimant 
> ===
> Claimant:  'riak@192.168.2.10 '
> Status: up
> Ring Ready: true
> 
> == Ownership Handoff 
> ==
> No pending changes.
> 
> == Unreachable Nodes 
> ==
> All nodes are up and reachable
> 
> current Transfer Limit is 2.
> 
> Thanks
> Gaurav
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

RE: RIAK repl enhancements

[Andrew.Deane]

Hi,

Yeah. FB and PRs back to main for both 2 and 3.

We intend to push forward and switch focus to 3 after this phase. OTP20 will be 
targeted, as we have seen large performance increases in provisional tests.

Andy.


Andrew Deane

Systems Development Manager - Core Systems

Hillside (Technology) Limited

andrew.de...@bet365.com

bet365.com


From: Bryan Hunt [mailto:bryan.h...@erlang-solutions.com]
Sent: 23 May 2018 13:39
To: Andrew Deane
Cc: riak-users@lists.basho.com
Subject: Re: RIAK repl enhancements

Andrew,

Your proposed changes sound like very well considered and welcome improvements.

What is going to be the strategy for merging?

Do you intend to publish to feature branches and make pull requests main 
development branches with community review?

Are the changes going to be targeted at the existing 2.* line or 3.0?

Progress towards 3.* (switch to modern Erlang/rebar3/leveled/new AAE) has been 
slow, it may be expeditious to target the 2.* line.

Bryan


On 23 May 2018, at 13:17, 
<andrew.de...@bet365.com<mailto:andrew.de...@bet365.com>> 
<andrew.de...@bet365.com<mailto:andrew.de...@bet365.com>> wrote:

Hi,

A quick note to highlight bet365 have published a summary of the replication 
work currently in final stages of dev.

http://bet365techblog.com/riak-update

These enhancements will be pushed to the canonical repo in the coming weeks, at 
which point we’ll also publish a more in-depth explanation of the changes.

Thanks,
Andy.

Andrew Deane

Systems Development Manager - Core Systems

Hillside (Technology) Limited

andrew.de...@bet365.com<mailto:andrew.de...@bet365.com>

bet365.com<http://bet365.com/>


___
riak-users mailing list
riak-users@lists.basho.com<mailto:riak-users@lists.basho.com>
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: RIAK repl enhancements

[Bryan Hunt]

Andrew, 

Your proposed changes sound like very well considered and welcome improvements. 

What is going to be the strategy for merging? 

Do you intend to publish to feature branches and make pull requests main 
development branches with community review?

Are the changes going to be targeted at the existing 2.* line or 3.0? 

Progress towards 3.* (switch to modern Erlang/rebar3/leveled/new AAE) has been 
slow, it may be expeditious to target the 2.* line. 

Bryan

> On 23 May 2018, at 13:17,   
> wrote:
> 
> Hi,
>  
> A quick note to highlight bet365 have published a summary of the replication 
> work currently in final stages of dev.
>  
> http://bet365techblog.com/riak-update 
>  
> These enhancements will be pushed to the canonical repo in the coming weeks, 
> at which point we’ll also publish a more in-depth explanation of the changes.
>  
> Thanks,
> Andy.
>  
> Andrew Deane
> Systems Development Manager - Core Systems
> Hillside (Technology) Limited
> andrew.de...@bet365.com 
> bet365.com 
>  
> ___
> riak-users mailing list
> riak-users@lists.basho.com 
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com 
> 
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

RE: N = 3 and RW = 2 not finding some keys

[Nicholas Adams]

Sorry for being late to the party but have you tried repairing all partitions? 
https://docs.basho.com/riak/kv/2.1.4/using/repair-recovery/repairs/#repairing-all-partitions-on-a-node

This forces every single piece of information to be read and thus causes read 
repair to kick in for any partitions that register less than 3 copies.

Best regards,

Nicholas

From: riak-users <riak-users-boun...@lists.basho.com> On Behalf Of Guido Medina
Sent: 18 May 2018 22:52
To: riak-users <riak-users@lists.basho.com>
Subject: Re: N = 3 and RW = 2 not finding some keys

Yes, but we haven't set back R = 2, we are currently with R = 3 until we 
resolve the issue.
I'm in the middle of writing something that will read every single key for the 
cluster using our relation DB IDs (each key has a counterpart on DB)

I'm guessing that 2i streaming won't help? we need to make sure we are fully 
consistent (for RW = 2) so I have to find a way to repair everything before 
turning R to 2.

After reading the object yes every request after was successful, but we even 
had a weird situation where after writing key with W = 2 we couldn't find with 
R = 2, not good for us.

Guido.
On 18/05/18 14:34, Bryan Hunt wrote:

Russell, Good question. I’m guessing they are iterating, and requesting a 
different object for each request?



Guido, given the behaviour you initially described, before applying the 
configuration I suggested - did you receive a successful response upon 
subsequent requests for the same object ?



On 18 May 2018, at 13:13, Russell Brown 
<russell.br...@icloud.com><mailto:russell.br...@icloud.com> wrote:



But why isn’t read repair “working”?



On 18 May 2018, at 11:07, Bryan Hunt 
<bryan.h...@erlang-solutions.com><mailto:bryan.h...@erlang-solutions.com> wrote:



Of course, AAE will eventually repair the missing object replicas but it seems 
like you need something more immediate.



On 18 May 2018, at 11:00, Bryan Hunt 
<bryan.h...@erlang-solutions.com><mailto:bryan.h...@erlang-solutions.com> wrote:



Hi Guido,



You should attempt to change the bucket property ‘notfound_ok’ from the default 
of ‘true' to ‘false'.



I.e



curl -XPUT 127.0.0.1:10018/buckets/foo/props -H "Content-Type: 
application/json" -d '{"props":{"notfound_ok": false}}'



This makes GET operations for non-existent keys slower as it forces an internal 
GET for each of the three copies.



https://docs.basho.com/riak/kv/2.1.1/developing/app-guide/replication-properties/#the-implications-of-notfound-ok



From what you describe, it sounds like only a single copy (out of the original 
three), somehow remain present in your cluster.



Best Regards,



Bryan Hunt



On 17 May 2018, at 15:42, Guido Medina 
<gmed...@temetra.com><mailto:gmed...@temetra.com> wrote:



Hi all,



After some big rebalance of our cluster some keys are not found anymore unless 
we set R = 3, we had N = 3 and R = W = 2



Is there any sort of repair that would correct such situation for Riak 2.2.3, 
this is really driving us nuts.



Any help will be truly appreciated.



Kind regards,

Guido.

___

riak-users mailing list

riak-users@lists.basho.com<mailto:riak-users@lists.basho.com>

http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





___

riak-users mailing list

riak-users@lists.basho.com<mailto:riak-users@lists.basho.com>

http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com







___

riak-users mailing list

riak-users@lists.basho.com<mailto:riak-users@lists.basho.com>

http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: N = 3 and RW = 2 not finding some keys

[Guido Medina]

Yes, but we haven't set back R = 2, we are currently with R = 3 until we 
resolve the issue.
I'm in the middle of writing something that will read every single key 
for the cluster using our relation DB IDs (each key has a counterpart on DB)


I'm guessing that 2i streaming won't help? we need to make sure we are 
fully consistent (for RW = 2) so I have to find a way to repair 
everything before turning R to 2.


After reading the object yes every request after was successful, but we 
even had a weird situation where after writing key with W = 2 we 
couldn't find with R = 2, not good for us.


Guido.

On 18/05/18 14:34, Bryan Hunt wrote:

Russell, Good question. I’m guessing they are iterating, and requesting a 
different object for each request?

Guido, given the behaviour you initially described, before applying the 
configuration I suggested - did you receive a successful response upon 
subsequent requests for the same object ?


On 18 May 2018, at 13:13, Russell Brown  wrote:

But why isn’t read repair “working”?


On 18 May 2018, at 11:07, Bryan Hunt  wrote:

Of course, AAE will eventually repair the missing object replicas but it seems 
like you need something more immediate.


On 18 May 2018, at 11:00, Bryan Hunt  wrote:

Hi Guido,

You should attempt to change the bucket property ‘notfound_ok’ from the default 
of ‘true' to ‘false'.

I.e

curl -XPUT 127.0.0.1:10018/buckets/foo/props -H "Content-Type: application/json" -d 
'{"props":{"notfound_ok": false}}'

This makes GET operations for non-existent keys slower as it forces an internal 
GET for each of the three copies.

https://docs.basho.com/riak/kv/2.1.1/developing/app-guide/replication-properties/#the-implications-of-notfound-ok

 From what you describe, it sounds like only a single copy (out of the original 
three), somehow remain present in your cluster.

Best Regards,

Bryan Hunt


On 17 May 2018, at 15:42, Guido Medina  wrote:

Hi all,

After some big rebalance of our cluster some keys are not found anymore unless 
we set R = 3, we had N = 3 and R = W = 2

Is there any sort of repair that would correct such situation for Riak 2.2.3, 
this is really driving us nuts.

Any help will be truly appreciated.

Kind regards,
Guido.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: N = 3 and RW = 2 not finding some keys

[Bryan Hunt]

Russell, Good question. I’m guessing they are iterating, and requesting a 
different object for each request? 

Guido, given the behaviour you initially described, before applying the 
configuration I suggested - did you receive a successful response upon 
subsequent requests for the same object ?

> On 18 May 2018, at 13:13, Russell Brown  wrote:
> 
> But why isn’t read repair “working”?
> 
>> On 18 May 2018, at 11:07, Bryan Hunt  wrote:
>> 
>> Of course, AAE will eventually repair the missing object replicas but it 
>> seems like you need something more immediate. 
>> 
>>> On 18 May 2018, at 11:00, Bryan Hunt  
>>> wrote:
>>> 
>>> Hi Guido, 
>>> 
>>> You should attempt to change the bucket property ‘notfound_ok’ from the 
>>> default of ‘true' to ‘false'.
>>> 
>>> I.e 
>>> 
>>> curl -XPUT 127.0.0.1:10018/buckets/foo/props -H "Content-Type: 
>>> application/json" -d '{"props":{"notfound_ok": false}}'
>>> 
>>> This makes GET operations for non-existent keys slower as it forces an 
>>> internal GET for each of the three copies.
>>> 
>>> https://docs.basho.com/riak/kv/2.1.1/developing/app-guide/replication-properties/#the-implications-of-notfound-ok
>>> 
>>> From what you describe, it sounds like only a single copy (out of the 
>>> original three), somehow remain present in your cluster.
>>> 
>>> Best Regards,
>>> 
>>> Bryan Hunt
>>> 
 On 17 May 2018, at 15:42, Guido Medina  wrote:
 
 Hi all,
 
 After some big rebalance of our cluster some keys are not found anymore 
 unless we set R = 3, we had N = 3 and R = W = 2
 
 Is there any sort of repair that would correct such situation for Riak 
 2.2.3, this is really driving us nuts.
 
 Any help will be truly appreciated.
 
 Kind regards,
 Guido.
 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>> 
>> 
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> 


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: N = 3 and RW = 2 not finding some keys

[Guido Medina]

A force replace done wrong causing some partitions to only have one copy.
And the cluster running extremely slow have forced us to disable AAE for 
now.


Guido.

On 18/05/18 13:13, Russell Brown wrote:

But why isn’t read repair “working”?


On 18 May 2018, at 11:07, Bryan Hunt  wrote:

Of course, AAE will eventually repair the missing object replicas but it seems 
like you need something more immediate.


On 18 May 2018, at 11:00, Bryan Hunt  wrote:

Hi Guido,

You should attempt to change the bucket property ‘notfound_ok’ from the default 
of ‘true' to ‘false'.

I.e

curl -XPUT 127.0.0.1:10018/buckets/foo/props -H "Content-Type: application/json" -d 
'{"props":{"notfound_ok": false}}'

This makes GET operations for non-existent keys slower as it forces an internal 
GET for each of the three copies.

https://docs.basho.com/riak/kv/2.1.1/developing/app-guide/replication-properties/#the-implications-of-notfound-ok

 From what you describe, it sounds like only a single copy (out of the original 
three), somehow remain present in your cluster.

Best Regards,

Bryan Hunt


On 17 May 2018, at 15:42, Guido Medina  wrote:

Hi all,

After some big rebalance of our cluster some keys are not found anymore unless 
we set R = 3, we had N = 3 and R = W = 2

Is there any sort of repair that would correct such situation for Riak 2.2.3, 
this is really driving us nuts.

Any help will be truly appreciated.

Kind regards,
Guido.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: N = 3 and RW = 2 not finding some keys

[Russell Brown]

But why isn’t read repair “working”?

> On 18 May 2018, at 11:07, Bryan Hunt  wrote:
> 
> Of course, AAE will eventually repair the missing object replicas but it 
> seems like you need something more immediate. 
> 
>> On 18 May 2018, at 11:00, Bryan Hunt  wrote:
>> 
>> Hi Guido, 
>> 
>> You should attempt to change the bucket property ‘notfound_ok’ from the 
>> default of ‘true' to ‘false'.
>> 
>> I.e 
>> 
>> curl -XPUT 127.0.0.1:10018/buckets/foo/props -H "Content-Type: 
>> application/json" -d '{"props":{"notfound_ok": false}}'
>> 
>> This makes GET operations for non-existent keys slower as it forces an 
>> internal GET for each of the three copies.
>> 
>> https://docs.basho.com/riak/kv/2.1.1/developing/app-guide/replication-properties/#the-implications-of-notfound-ok
>> 
>> From what you describe, it sounds like only a single copy (out of the 
>> original three), somehow remain present in your cluster.
>> 
>> Best Regards,
>> 
>> Bryan Hunt
>> 
>>> On 17 May 2018, at 15:42, Guido Medina  wrote:
>>> 
>>> Hi all,
>>> 
>>> After some big rebalance of our cluster some keys are not found anymore 
>>> unless we set R = 3, we had N = 3 and R = W = 2
>>> 
>>> Is there any sort of repair that would correct such situation for Riak 
>>> 2.2.3, this is really driving us nuts.
>>> 
>>> Any help will be truly appreciated.
>>> 
>>> Kind regards,
>>> Guido.
>>> ___
>>> riak-users mailing list
>>> riak-users@lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> 
> 
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: N = 3 and RW = 2 not finding some keys

[Bryan Hunt]

Of course, AAE will eventually repair the missing object replicas but it seems 
like you need something more immediate. 

> On 18 May 2018, at 11:00, Bryan Hunt  wrote:
> 
> Hi Guido, 
> 
> You should attempt to change the bucket property ‘notfound_ok’ from the 
> default of ‘true' to ‘false'.
> 
> I.e 
> 
> curl -XPUT 127.0.0.1:10018/buckets/foo/props -H "Content-Type: 
> application/json" -d '{"props":{"notfound_ok": false}}'
> 
> This makes GET operations for non-existent keys slower as it forces an 
> internal GET for each of the three copies.
> 
> https://docs.basho.com/riak/kv/2.1.1/developing/app-guide/replication-properties/#the-implications-of-notfound-ok
>  
> 
> 
> From what you describe, it sounds like only a single copy (out of the 
> original three), somehow remain present in your cluster.
> 
> Best Regards,
> 
> Bryan Hunt
> 
>> On 17 May 2018, at 15:42, Guido Medina > > wrote:
>> 
>> Hi all,
>> 
>> After some big rebalance of our cluster some keys are not found anymore 
>> unless we set R = 3, we had N = 3 and R = W = 2
>> 
>> Is there any sort of repair that would correct such situation for Riak 
>> 2.2.3, this is really driving us nuts.
>> 
>> Any help will be truly appreciated.
>> 
>> Kind regards,
>> Guido.
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com 
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> 

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: N = 3 and RW = 2 not finding some keys

[Bryan Hunt]

Hi Guido, 

You should attempt to change the bucket property ‘notfound_ok’ from the default 
of ‘true' to ‘false'.

I.e 

curl -XPUT 127.0.0.1:10018/buckets/foo/props -H "Content-Type: 
application/json" -d '{"props":{"notfound_ok": false}}'

This makes GET operations for non-existent keys slower as it forces an internal 
GET for each of the three copies.

https://docs.basho.com/riak/kv/2.1.1/developing/app-guide/replication-properties/#the-implications-of-notfound-ok
 


From what you describe, it sounds like only a single copy (out of the original 
three), somehow remain present in your cluster.

Best Regards,

Bryan Hunt

> On 17 May 2018, at 15:42, Guido Medina  wrote:
> 
> Hi all,
> 
> After some big rebalance of our cluster some keys are not found anymore 
> unless we set R = 3, we had N = 3 and R = W = 2
> 
> Is there any sort of repair that would correct such situation for Riak 2.2.3, 
> this is really driving us nuts.
> 
> Any help will be truly appreciated.
> 
> Kind regards,
> Guido.
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

RE: Riak CS Backup Solutions

[Nicholas Adams]

Dear Anthony,

As you specified in your question as to Basho's official method, may I refer 
you to https://docs.basho.com/riak/kv/2.1.4/using/cluster-operations/backing-up/

Backing up an entire Riak cluster in one operation is difficult because of the 
way Riak operates. It is impossible, without downtime, to say that no requests 
are currently in-flight between nodes or only written 2 out of 3 replicas, etc. 
For this reason we recommend users back up their nodes in a rolling fashion. 
Taking a node down, backing up the data/configuration/ring, and then restarting 
the node. Restoring from these backups is simple. Move the data directory 
backups, config, and ring back to their appropriate location and start the node.

But even with all those explanations, it's clear that for realtime distributed 
systems there's really not a truly "point in time" backup. This has been 
resolved in a Riak KV cluster via MDC replication. Should you want to consider 
MDC replication, we would highly recommend bi-directional replication as non 
bi-directional replication does not propagate tombstones which will lead to 
siblings on the standby cluster.

One thing to bear in mind is that Riak CS was never officially tested or 
approved with Riak KV 2.2.x series. As such, if you do not have an Enterprise 
Edition copy of Riak KV 2.1.4 or earlier, there is no officially approved way 
to run an MDC solution with Riak CS.

If MDC is not an option, one possible way to be able to replicate to a standby 
cluster is to do it at the application level. Maybe have the app communicate to 
2 CS nodes, the 1st CS node communicates to the production Riak KV cluster and 
the 2nd one communicates to the standby Riak KV cluster. This way, both the 
production and standby KV clusters get updated at around the same time.

tl;dr;
The only truly safe ways to back up a Riak cluster that is using Riak CS would 
be to:
1. Stop the cluster completely and do a full data back up via your preferred 
method e.g. snapshots, tarballs etc.
2. Licensing permitting, use MDC to replicate to another cluster.
3. Create something yourself to work at the application level that would mirror 
all commands.

Hope this helps,

Nicholas

From: riak-users <riak-users-boun...@lists.basho.com> On Behalf Of Bryan Hunt
Sent: 25 April 2018 22:50
To: riak-users <riak-users@lists.basho.com>
Subject: Re: Riak CS Backup Solutions

Anthony,

Your snapshot solution is not bad if you just want a daily version you can roll 
back to for disaster recovery.

I presume :

1. You’re running on VM’s
2. You stop the world at the time of the snapshot

Riak CS makes multiple (3 by default) copies of your data and generally 
speaking, stores them onto multiple physical servers with the following caveats 
:

1. Verify that the partitions are evenly spread across the machines, sometimes 
old Riak puts two copies of the data on a single physical machine.
2. Always run a cluster of 5 or more machines.
3. Take care that you’re not storing all your data on the same physical device 
(SAN server in the basement anyone?) ).

I suppose the reason you want backups is to mitigate the situation where an 
adversary gets inside your servers and begins silently deleting or tampering 
with data, in which case, snapshots are prudent.

If you wanted to do something more sophisticated, you could try :

1. Re-building your cluster using the soon to be released Riak 2.2.5.
2. Replicate off site to a physically remote location
3. More regularly perform snapshots there (as they won’t impact latencies on 
your main Riak CS cluster)

Does that make sense? Any questions?

Bryan





On 25 Apr 2018, at 14:30, Anthony Valenti 
<anthony.vale...@inmar.com<mailto:anthony.vale...@inmar.com>> wrote:


We are looking to improve on our Riak CS backup strategy.  We have had Riak CS 
in place for a while and we are currently taking a snapshot of each server.  
I'm not sure is this is the best method in terms of recoverability, space used, 
timeliness or the backups, etc.  What methods has anyone else used?  What is 
the basho recommended solution?

Thanks,
Anthony Valenti





Inmar Confidentiality Note:  This e-mail and any attachments are confidential 
and intended to be viewed and used solely by the intended recipient.  If you 
are not the intended recipient, be aware that any disclosure, dissemination, 
distribution, copying or use of this e-mail or any attachment is prohibited.  
If you received this e-mail in error, please notify us immediately by returning 
it to the sender and delete this copy and all attachments from your system and 
destroy any printed copies.  Thank you for your cooperation.


Notice of Protected Rights:  The removal of any copyright, trademark, or 
proprietary legend contained in this e-mail or any attachment is prohibited 
without the express, written permission of Inmar, Inc.  Furthermore, the 
intended recipient m

RE: [ANN] Riak 2.2.5 release

[Nicholas Adams]

Dear All,
Just a follow up to Russell's email to say the initial builds of Riak KV 2.2.5 
by TI Tokyo are now available at https://files.tiot.jp/riak/kv/2.2/2.2.5/ The 
same machines that built 2.2.5rc2 were used to build 2.2.5 so if rc2 worked for 
you, this should too.

As always, please offer feedback regarding any issues etc. encountered.

Best regards,

Nicholas

-Original Message-
From: riak-users  On Behalf Of Russell Brown
Sent: 26 April 2018 19:39
To: riak-users 
Subject: [ANN] Riak 2.2.5 release

Hi,

Thanks all for your patience and hard work. I just pushed the tag
riak-2.2.5 to the basho repo. Packages are being made.

Release notes are here 
https://github.com/basho/riak/blob/riak-2.2.5/RELEASE-NOTES.md

Cheers

Russell


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak CS Backup Solutions

[Bryan Hunt]

Anthony, 

Your snapshot solution is not bad if you just want a daily version you can roll 
back to for disaster recovery.

I presume : 

1. You’re running on VM’s 
2. You stop the world at the time of the snapshot

Riak CS makes multiple (3 by default) copies of your data and generally 
speaking, stores them onto multiple physical servers with the following caveats 
: 

1. Verify that the partitions are evenly spread across the machines, sometimes 
old Riak puts two copies of the data on a single physical machine.
2. Always run a cluster of 5 or more machines.
3. Take care that you’re not storing all your data on the same physical device 
(SAN server in the basement anyone?) ).

I suppose the reason you want backups is to mitigate the situation where an 
adversary gets inside your servers and begins silently deleting or tampering 
with data, in which case, snapshots are prudent. 

If you wanted to do something more sophisticated, you could try : 

1. Re-building your cluster using the soon to be released Riak 2.2.5.
2. Replicate off site to a physically remote location
3. More regularly perform snapshots there (as they won’t impact latencies on 
your main Riak CS cluster)

Does that make sense? Any questions? 

Bryan 




> On 25 Apr 2018, at 14:30, Anthony Valenti <anthony.vale...@inmar.com> wrote:
> 
> 
> We are looking to improve on our Riak CS backup strategy.  We have had Riak 
> CS in place for a while and we are currently taking a snapshot of each 
> server.  I'm not sure is this is the best method in terms of recoverability, 
> space used, timeliness or the backups, etc.  What methods has anyone else 
> used?  What is the basho recommended solution?
> 
> Thanks,
> Anthony Valenti
> 
> 
> 
> 
> 
>  
> Inmar Confidentiality Note:  This e-mail and any attachments are confidential 
> and intended to be viewed and used solely by the intended recipient.  If you 
> are not the intended recipient, be aware that any disclosure, dissemination, 
> distribution, copying or use of this e-mail or any attachment is prohibited.  
> If you received this e-mail in error, please notify us immediately by 
> returning it to the sender and delete this copy and all attachments from your 
> system and destroy any printed copies.  Thank you for your cooperation.
> 
> 
>  
> Notice of Protected Rights:  The removal of any copyright, trademark, or 
> proprietary legend contained in this e-mail or any attachment is prohibited 
> without the express, written permission of Inmar, Inc.  Furthermore, the 
> intended recipient must maintain all copyright notices, trademarks, and 
> proprietary legends within this e-mail and any attachments in their original 
> form and location if the e-mail or any attachments are reproduced, printed or 
> distributed.
> 
>  
> 
> 
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak-2.2.5 progress update

[Damien Krotkine]

Hi,

I have installed RC2 (thanks to Nicholas' rpms) on one of our rings (5 boxes, 
heavy continuous read, heavy continuous write, centos7, bitcask, expiration, no 
compaction), and everything is running smoothly (same metrics as before, no 
errors so far). The only minor user experience issue was 
https://github.com/basho/riak/issues/940



On Thu, Apr 12, 2018, at 05:26, Nicholas Adams wrote:
> Dear All,
> We have placed a basic set of package builds online at 
> https://files.tiot.jp/riak/kv/2.2/2.2.5rc2/ 
> These builds were very simply a clone, checkout, make locked-all and 
> make package locked-all so should provide the same functionality as 
> building Riak 2.2.5rc2 from source.
> Good luck and please share your feedback.
> 
> Best regards,
> 
> Nicholas 
> 
> -Original Message-
> From: riak-users <riak-users-boun...@lists.basho.com> On Behalf Of 
> Russell Brown
> Sent: 10 April 2018 18:53
> To: Bryan Hunt <bryan.h...@erlang-solutions.com>
> Cc: Damien Krotkine <dam...@krotkine.com>; riak-users  us...@lists.basho.com>
> Subject: Re: Riak-2.2.5 progress update
> 
> More update.
> 
> There were some minor changes that missed the rc1, so we added them
> 
> * yokozuna/746: [remove -XX:+UseStringCache]
> (https://github.com/basho/yokozuna/pull/746)
> * yokozuna/747: [Remove jvm directive from test too]
> (https://github.com/basho/yokozuna/pull/747)
> * riak_repl/782: [Change ETS queue table permissions to protected]
> (https://github.com/basho/riak_repl/pull/782)
> 
> And today tagged RC2 (riak-2.2.5rc2)
> 
> Same rules as before:
> 
> At this point I DID NOT tag all the dependencies so it is _VERY 
> IMPORTANT_ that the rebar.config.lock file is used when you build a 
> release to test (e.g. `make locked-all`)
> 
> If anyone (Tiot? ESL?) can/wants to build packages please post to this 
> list about location for those who'd rather test against packages.
> 
> We've got perf test results upcoming, and then we'll tag final.
> 
> Thanks for your patience, almost there now.
> 
> Cheers
> 
> Russell
> 
> On 29 Mar 2018, at 18:18, Bryan Hunt <bryan.h...@erlang-solutions.com> wrote:
> 
> > From our side, it's bank holiday weekend now, so we shall start building 
> > packages on Monday/Tuesday and share them out via package cloud. 
> > Will keep you updated. 
> > B
> > 
> >> On 29 Mar 2018, at 16:15, Russell Brown <russell.br...@icloud.com> wrote:
> >> 
> >> 
> > 
> > ___
> > riak-users mailing list
> > riak-users@lists.basho.com
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> 
> 
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> 
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

RE: Riak-2.2.5 progress update

[Nicholas Adams]

Dear All,
We have placed a basic set of package builds online at 
https://files.tiot.jp/riak/kv/2.2/2.2.5rc2/ 
These builds were very simply a clone, checkout, make locked-all and make 
package locked-all so should provide the same functionality as building Riak 
2.2.5rc2 from source.
Good luck and please share your feedback.

Best regards,

Nicholas 

-Original Message-
From: riak-users <riak-users-boun...@lists.basho.com> On Behalf Of Russell Brown
Sent: 10 April 2018 18:53
To: Bryan Hunt <bryan.h...@erlang-solutions.com>
Cc: Damien Krotkine <dam...@krotkine.com>; riak-users 
<riak-users@lists.basho.com>
Subject: Re: Riak-2.2.5 progress update

More update.

There were some minor changes that missed the rc1, so we added them

* yokozuna/746: [remove 
-XX:+UseStringCache](https://github.com/basho/yokozuna/pull/746)
* yokozuna/747: [Remove jvm directive from test 
too](https://github.com/basho/yokozuna/pull/747)
* riak_repl/782: [Change ETS queue table permissions to 
protected](https://github.com/basho/riak_repl/pull/782)

And today tagged RC2 (riak-2.2.5rc2)

Same rules as before:

At this point I DID NOT tag all the dependencies so it is _VERY IMPORTANT_ that 
the rebar.config.lock file is used when you build a release to test (e.g. `make 
locked-all`)

If anyone (Tiot? ESL?) can/wants to build packages please post to this list 
about location for those who'd rather test against packages.

We've got perf test results upcoming, and then we'll tag final.

Thanks for your patience, almost there now.

Cheers

Russell

On 29 Mar 2018, at 18:18, Bryan Hunt <bryan.h...@erlang-solutions.com> wrote:

> From our side, it's bank holiday weekend now, so we shall start building 
> packages on Monday/Tuesday and share them out via package cloud. 
> Will keep you updated. 
> B
> 
>> On 29 Mar 2018, at 16:15, Russell Brown <russell.br...@icloud.com> wrote:
>> 
>> 
> 
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak-2.2.5 progress update

[Russell Brown]

More update.

There were some minor changes that missed the rc1, so we added them

* yokozuna/746: [remove 
-XX:+UseStringCache](https://github.com/basho/yokozuna/pull/746)
* yokozuna/747: [Remove jvm directive from test 
too](https://github.com/basho/yokozuna/pull/747)
* riak_repl/782: [Change ETS queue table permissions to 
protected](https://github.com/basho/riak_repl/pull/782)

And today tagged RC2 (riak-2.2.5rc2)

Same rules as before:

At this point I DID NOT tag all the dependencies so it is _VERY IMPORTANT_ that 
the rebar.config.lock file is used when you build a release to test (e.g. `make 
locked-all`)

If anyone (Tiot? ESL?) can/wants to build packages please post to this list 
about location for those who’d rather test against packages.

We’ve got perf test results upcoming, and then we’ll tag final.

Thanks for your patience, almost there now.

Cheers

Russell

On 29 Mar 2018, at 18:18, Bryan Hunt  wrote:

> From our side, it’s bank holiday weekend now, so we shall start building 
> packages on Monday/Tuesday and share them out via package cloud. 
> Will keep you updated. 
> B
> 
>> On 29 Mar 2018, at 16:15, Russell Brown  wrote:
>> 
>> 
> 
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak-2.2.5 progress update

[Bryan Hunt]

From our side, it’s bank holiday weekend now, so we shall start building 
packages on Monday/Tuesday and share them out via package cloud. 
Will keep you updated. 
B

> On 29 Mar 2018, at 16:15, Russell Brown  wrote:
> 
> 

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak-2.2.5 progress update

[Russell Brown]

More update.

I just pushed riak-2.2.5rc1 to basho/riak.

At this point I DID NOT tag all the dependencies so it is _VERY IMPORTANT_ that 
the rebar.config.lock file is used when you build a release to test (e.g. `make 
locked-all`)

If anyone (Bryan Hunt, ESL?) can/wants to build packages please post to this 
list about location for those who’d rather test against packages.

Are we done? Not yet.

What to expect next:

1. Results of load tests from any who have said they can (Damien, Martin Cox, 
Martin Sumner?)
2. Docs and release notes (ESL, Tiot.jp, me?)
3. Final flakey riak tests (yokozuna and ensemble)
4. Tag all deps, tag final
5. Celebrate

It’s way early for celebrations, back slapping etc, but on a selfish, personal 
note, I really want to thank the NHS for enabling Ramen, Martin, and I to 
contribute to this project.

Cheers

Russell

On 29 Mar 2018, at 14:03, Damien Krotkine <dam...@krotkine.com> wrote:

> Hi,
> 
> I'm probably coming late to the party, but we have a big number of riak boxes 
> running at work, in meta-clusters, so some rings are redundantly storing 
> data. I could move one of them to the RC and compare its 
> performance/errors/whatever with the non upgraded rings, if you people think 
> it's useful. Poke me if that's useful.
> 
> Cheers
> 
> On Wed, Mar 28, 2018, at 09:07, martin@bet365.com wrote:
>> Awesome progress.
>> 
>> We're happy to run generalised load tests against RC and share results 
>> when completed - probably in the following week.
>> 
>> Cheers
>> 
>> From: riak-users [riak-users-boun...@lists.basho.com] on behalf of 
>> Russell Brown [russell.br...@icloud.com]
>> Sent: 27 March 2018 19:04
>> To: Fred Dushin
>> Cc: riak-users
>> Subject: Re: Riak-2.2.5 progress update
>> 
>> Hey Fred,
>> 
>> I can probably share my configs, yes. I have 4 configs!
>> 
>> current is riak-2.2.5, the develop-2.2 branch of basho/riak for all 
>> configs
>> previous is either riak-2.2.3 or riak-2.0.5 (and I have 4 configs as 
>> there has to be an EE config, for the repl upgrade tests, since 
>> riak-2.2.3 didn’t have repl, but riak_ee-2.2.3 did)
>> legacy is always 1.4.12
>> 
>> And since some tests have explicit version there are also explicit 
>> entries for 2.0.6, 2.0.4, 2.0.2, in each config.
>> 
>> Does that help?
>> 
>> Cheers
>> 
>> Russell
>> 
>> On 27 Mar 2018, at 18:51, Fred Dushin <f...@dushin.net> wrote:
>> 
>>> @russeldb do you have ordained values for `current`, `previous`, and 
>>> `legacy` to test against?
>>> 
>>> Always the :bane: of riak_test
>>> 
>>> -Fred
>>> 
>>>> On Mar 27, 2018, at 1:47 PM, Russell Brown <russell.br...@me.com> wrote:
>>>> 
>>>> Giddyup died when basho was shuttered. These test runs have all been on 
>>>> private infrastructure that doesn’t have a public facing interface.
>>>> 
>>>> I guess I could probably cut and paste logs into a gist for you. But I’d 
>>>> have to take the time to sanitize them, just in case. I’d rather not take 
>>>> that time unless absolutely necessary.
>>>> 
>>>> On 27 Mar 2018, at 18:43, Bryan Hunt <bryan.h...@erlang-solutions.com> 
>>>> wrote:
>>>> 
>>>>> Could you share URL for those outstanding failures please. B
>>>>> 
>>>>>> On 27 Mar 2018, at 16:37, Russell Brown <russell.br...@me.com> wrote:
>>>>>> 
>>>>>> Hi Again,
>>>>>> 
>>>>>> More progress update. All the PRs are merged (thanks Bryan Hunt (ESL)
>>>>>> and Nick/Martin (bet365)).
>>>>>> 
>>>>>> I’m planning on tagging Riak with riak-2.2.5RC1 this week.
>>>>>> 
>>>>>> We haven’t been able to run load tests. I’m not 100% sure that all
>>>>>> basho’s releases had extensive load tests run, though I know that
>>>>>> later releases had a perf team, and MvM always load tested leveldb.
>>>>>> 
>>>>>> The aim is to have those willing parties that deploy riak, and
>>>>>> therefore have perf testing already set up, to test and report
>>>>>> back. The NHS will run load tests and report results back. I hope that 
>>>>>> others can do the same.
>>>>>> To the end we’ll probably tag RC1-noperftest.
>>>>>> 
>>>>>> There are a few failing riak tests (was Riak ever relea

Re: Riak-2.2.5 progress update

[Damien Krotkine]

Hi,

I'm probably coming late to the party, but we have a big number of riak boxes 
running at work, in meta-clusters, so some rings are redundantly storing data. 
I could move one of them to the RC and compare its performance/errors/whatever 
with the non upgraded rings, if you people think it's useful. Poke me if that's 
useful.

Cheers

On Wed, Mar 28, 2018, at 09:07, martin@bet365.com wrote:
> Awesome progress.
> 
> We're happy to run generalised load tests against RC and share results 
> when completed - probably in the following week.
> 
> Cheers
> 
> From: riak-users [riak-users-boun...@lists.basho.com] on behalf of 
> Russell Brown [russell.br...@icloud.com]
> Sent: 27 March 2018 19:04
> To: Fred Dushin
> Cc: riak-users
> Subject: Re: Riak-2.2.5 progress update
> 
> Hey Fred,
> 
> I can probably share my configs, yes. I have 4 configs!
> 
> current is riak-2.2.5, the develop-2.2 branch of basho/riak for all 
> configs
> previous is either riak-2.2.3 or riak-2.0.5 (and I have 4 configs as 
> there has to be an EE config, for the repl upgrade tests, since 
> riak-2.2.3 didn’t have repl, but riak_ee-2.2.3 did)
> legacy is always 1.4.12
> 
> And since some tests have explicit version there are also explicit 
> entries for 2.0.6, 2.0.4, 2.0.2, in each config.
> 
> Does that help?
> 
> Cheers
> 
> Russell
> 
> On 27 Mar 2018, at 18:51, Fred Dushin <f...@dushin.net> wrote:
> 
> > @russeldb do you have ordained values for `current`, `previous`, and 
> > `legacy` to test against?
> >
> > Always the :bane: of riak_test
> >
> > -Fred
> >
> >> On Mar 27, 2018, at 1:47 PM, Russell Brown <russell.br...@me.com> wrote:
> >>
> >> Giddyup died when basho was shuttered. These test runs have all been on 
> >> private infrastructure that doesn’t have a public facing interface.
> >>
> >> I guess I could probably cut and paste logs into a gist for you. But I’d 
> >> have to take the time to sanitize them, just in case. I’d rather not take 
> >> that time unless absolutely necessary.
> >>
> >> On 27 Mar 2018, at 18:43, Bryan Hunt <bryan.h...@erlang-solutions.com> 
> >> wrote:
> >>
> >>> Could you share URL for those outstanding failures please. B
> >>>
> >>>> On 27 Mar 2018, at 16:37, Russell Brown <russell.br...@me.com> wrote:
> >>>>
> >>>> Hi Again,
> >>>>
> >>>> More progress update. All the PRs are merged (thanks Bryan Hunt (ESL)
> >>>> and Nick/Martin (bet365)).
> >>>>
> >>>> I’m planning on tagging Riak with riak-2.2.5RC1 this week.
> >>>>
> >>>> We haven’t been able to run load tests. I’m not 100% sure that all
> >>>> basho’s releases had extensive load tests run, though I know that
> >>>> later releases had a perf team, and MvM always load tested leveldb.
> >>>>
> >>>> The aim is to have those willing parties that deploy riak, and
> >>>> therefore have perf testing already set up, to test and report
> >>>> back. The NHS will run load tests and report results back. I hope that 
> >>>> others can do the same.
> >>>> To the end we’ll probably tag RC1-noperftest.
> >>>>
> >>>> There are a few failing riak tests (was Riak ever released without a
> >>>> failing riak-test?) If you have the time/capacity to run riak-test,
> >>>> and you’re interested in helping out, get in touch and I’ll help you
> >>>> get started.
> >>>>
> >>>> The failures, should one pique your interest:
> >>>>
> >>>> datatypes - riak667_mixed-eleveldb
> >>>> ensemble - ensemble_basic3-eleveldb ensemble_basic4-eleveldb
> >>>> yoko - yz_crdt-eleveldb yz_solr_upgrade_downgrade-eleveldb
> >>>>
> >>>> Let me know if you want to look into ensemble or yoko.
> >>>>
> >>>> Still aiming to have this tagged by end-of-week.
> >>>>
> >>>> Cheers
> >>>>
> >>>> Russell
> >>>>
> >>>> On 2 Mar 2018, at 09:44, Russell Brown <russell.br...@me.com> wrote:
> >>>>
> >>>>> Hi,
> >>>>>
> >>>>> Just an update on the progress of riak-2.2.5 release. I realize we said
> >>>>> "end of 2017" and then "end of Jan" and then "end of Feb" and he

Re: Riak-2.2.5 progress update

[Russell Brown]

Hey Fred,

I can probably share my configs, yes. I have 4 configs!

current is riak-2.2.5, the develop-2.2 branch of basho/riak for all configs
previous is either riak-2.2.3 or riak-2.0.5 (and I have 4 configs as there has 
to be an EE config, for the repl upgrade tests, since riak-2.2.3 didn’t have 
repl, but riak_ee-2.2.3 did)
legacy is always 1.4.12

And since some tests have explicit version there are also explicit entries for 
2.0.6, 2.0.4, 2.0.2, in each config.

Does that help?

Cheers

Russell

On 27 Mar 2018, at 18:51, Fred Dushin  wrote:

> @russeldb do you have ordained values for `current`, `previous`, and `legacy` 
> to test against?
> 
> Always the :bane: of riak_test
> 
> -Fred
> 
>> On Mar 27, 2018, at 1:47 PM, Russell Brown  wrote:
>> 
>> Giddyup died when basho was shuttered. These test runs have all been on 
>> private infrastructure that doesn’t have a public facing interface.
>> 
>> I guess I could probably cut and paste logs into a gist for you. But I’d 
>> have to take the time to sanitize them, just in case. I’d rather not take 
>> that time unless absolutely necessary.
>> 
>> On 27 Mar 2018, at 18:43, Bryan Hunt  wrote:
>> 
>>> Could you share URL for those outstanding failures please. B
>>> 
 On 27 Mar 2018, at 16:37, Russell Brown  wrote:
 
 Hi Again,
 
 More progress update. All the PRs are merged (thanks Bryan Hunt (ESL)
 and Nick/Martin (bet365)).
 
 I’m planning on tagging Riak with riak-2.2.5RC1 this week.
 
 We haven’t been able to run load tests. I’m not 100% sure that all
 basho’s releases had extensive load tests run, though I know that
 later releases had a perf team, and MvM always load tested leveldb.
 
 The aim is to have those willing parties that deploy riak, and
 therefore have perf testing already set up, to test and report
 back. The NHS will run load tests and report results back. I hope that 
 others can do the same.
 To the end we’ll probably tag RC1-noperftest.
 
 There are a few failing riak tests (was Riak ever released without a
 failing riak-test?) If you have the time/capacity to run riak-test,
 and you’re interested in helping out, get in touch and I’ll help you
 get started.
 
 The failures, should one pique your interest:
 
 datatypes - riak667_mixed-eleveldb
 ensemble - ensemble_basic3-eleveldb ensemble_basic4-eleveldb
 yoko - yz_crdt-eleveldb yz_solr_upgrade_downgrade-eleveldb
 
 Let me know if you want to look into ensemble or yoko.
 
 Still aiming to have this tagged by end-of-week.
 
 Cheers
 
 Russell
 
 On 2 Mar 2018, at 09:44, Russell Brown  wrote:
 
> Hi,
> 
> Just an update on the progress of riak-2.2.5 release. I realize we said
> "end of 2017" and then "end of Jan" and then "end of Feb" and here we
> are, 1st March, spring is upon us, and still no 2.2.5. I thought it best
> to at least keep you posted.
> 
> Why no release? Well, we're not quite finished yet. In terms of what is
> left:
> 
> - a few PRs need review and merge against the upstream Basho repo (see
> [1] if you want to help there);
> - Opening of PRs for gsets;
> - Docs for the changes;
> - Release notes;
> - Tagging;
> - A final round of testing (and fixing?) after all is merged;
> - Client support. This is crucial, but all the above work only has
> client support in the Basho erlang clients. I'm hoping the community
> that uses the Java/Python/Ruby etc clients can step up here. But we can 
> release Riak before all client work is done.
> 
> The optimist in me says 2 weeks.
> 
> What do I mean "release"?
> 
> From my point of view the release is the tags. After that I'm sincerely
> hoping ESL will continue to kindly build and host the actual artifacts.
> 
> What's in the release?
> 
> As a reminder, since it's been a while since StokeCon17, this release
> contains:
> 
> - Some developer clean-up around `make test` and `riak_test` to make
> them more reliable/trustworthy;
> - Open source MDC Repl (thanks bet365!);
> - A fix for a bug in riak core claim that led to unbalanced rings[2];
> - `node_confirms` a feature like `w` or `pw` but for physical diversity 
> in durability[3];
> - `participate in coverage` an admin setting that takes a node out of
> the coverage plan (for example after adding a node while transfers
> take place);
> - Riak repl fix, and change to unsafe default behaviour[4];
> - Addition of a GSet to riak data types;
> - Fix to repl stats[5].
> 
> Sorry if I missed anything. The release notes will have it all.
> 
> Work is already begun on Riak 3.0 with OTP20 support well under way.
> 

Re: Riak-2.2.5 progress update

[Fred Dushin]

@russeldb do you have ordained values for `current`, `previous`, and `legacy` 
to test against?

Always the :bane: of riak_test

-Fred

> On Mar 27, 2018, at 1:47 PM, Russell Brown  wrote:
> 
> Giddyup died when basho was shuttered. These test runs have all been on 
> private infrastructure that doesn’t have a public facing interface.
> 
> I guess I could probably cut and paste logs into a gist for you. But I’d have 
> to take the time to sanitize them, just in case. I’d rather not take that 
> time unless absolutely necessary.
> 
> On 27 Mar 2018, at 18:43, Bryan Hunt  wrote:
> 
>> Could you share URL for those outstanding failures please. B
>> 
>>> On 27 Mar 2018, at 16:37, Russell Brown  wrote:
>>> 
>>> Hi Again,
>>> 
>>> More progress update. All the PRs are merged (thanks Bryan Hunt (ESL)
>>> and Nick/Martin (bet365)).
>>> 
>>> I’m planning on tagging Riak with riak-2.2.5RC1 this week.
>>> 
>>> We haven’t been able to run load tests. I’m not 100% sure that all
>>> basho’s releases had extensive load tests run, though I know that
>>> later releases had a perf team, and MvM always load tested leveldb.
>>> 
>>> The aim is to have those willing parties that deploy riak, and
>>> therefore have perf testing already set up, to test and report
>>> back. The NHS will run load tests and report results back. I hope that 
>>> others can do the same.
>>> To the end we’ll probably tag RC1-noperftest.
>>> 
>>> There are a few failing riak tests (was Riak ever released without a
>>> failing riak-test?) If you have the time/capacity to run riak-test,
>>> and you’re interested in helping out, get in touch and I’ll help you
>>> get started.
>>> 
>>> The failures, should one pique your interest:
>>> 
>>> datatypes - riak667_mixed-eleveldb
>>> ensemble - ensemble_basic3-eleveldb ensemble_basic4-eleveldb
>>> yoko - yz_crdt-eleveldb yz_solr_upgrade_downgrade-eleveldb
>>> 
>>> Let me know if you want to look into ensemble or yoko.
>>> 
>>> Still aiming to have this tagged by end-of-week.
>>> 
>>> Cheers
>>> 
>>> Russell
>>> 
>>> On 2 Mar 2018, at 09:44, Russell Brown  wrote:
>>> 
 Hi,
 
 Just an update on the progress of riak-2.2.5 release. I realize we said
 "end of 2017" and then "end of Jan" and then "end of Feb" and here we
 are, 1st March, spring is upon us, and still no 2.2.5. I thought it best
 to at least keep you posted.
 
 Why no release? Well, we're not quite finished yet. In terms of what is
 left:
 
 - a few PRs need review and merge against the upstream Basho repo (see
 [1] if you want to help there);
 - Opening of PRs for gsets;
 - Docs for the changes;
 - Release notes;
 - Tagging;
 - A final round of testing (and fixing?) after all is merged;
 - Client support. This is crucial, but all the above work only has
 client support in the Basho erlang clients. I'm hoping the community
 that uses the Java/Python/Ruby etc clients can step up here. But we can 
 release Riak before all client work is done.
 
 The optimist in me says 2 weeks.
 
 What do I mean "release"?
 
 From my point of view the release is the tags. After that I'm sincerely
 hoping ESL will continue to kindly build and host the actual artifacts.
 
 What's in the release?
 
 As a reminder, since it's been a while since StokeCon17, this release
 contains:
 
 - Some developer clean-up around `make test` and `riak_test` to make
 them more reliable/trustworthy;
 - Open source MDC Repl (thanks bet365!);
 - A fix for a bug in riak core claim that led to unbalanced rings[2];
 - `node_confirms` a feature like `w` or `pw` but for physical diversity in 
 durability[3];
 - `participate in coverage` an admin setting that takes a node out of
 the coverage plan (for example after adding a node while transfers
 take place);
 - Riak repl fix, and change to unsafe default behaviour[4];
 - Addition of a GSet to riak data types;
 - Fix to repl stats[5].
 
 Sorry if I missed anything. The release notes will have it all.
 
 Work is already begun on Riak 3.0 with OTP20 support well under way.
 Some candidates for inclusion that we're working on are a new
 pure-Erlang backend[6], and a radical overhaul of AAE for both intra and 
 inter-cluster anti-entropy.
 
 Sorry for the delay, thanks for your patience. We’ll keep you posted.
 
 Cheers
 
 Russell
 
 Titus Systems - ti-sys.co.uk
 
 [1] Coverage:
 https://github.com/basho/riak_core/pull/917
 https://github.com/basho/riak_kv/pull/1664
 https://github.com/basho/riak_test/pull/1300
 
 Repl:
 https://github.com/basho/riak_repl/pull/777
 https://github.com/basho/riak_test/pull/1301
 
 Node Confirms:
 

Re: Riak-2.2.5 progress update

[Russell Brown]

Giddyup died when basho was shuttered. These test runs have all been on private 
infrastructure that doesn’t have a public facing interface.

I guess I could probably cut and paste logs into a gist for you. But I’d have 
to take the time to sanitize them, just in case. I’d rather not take that time 
unless absolutely necessary.

On 27 Mar 2018, at 18:43, Bryan Hunt  wrote:

> Could you share URL for those outstanding failures please. B
> 
>> On 27 Mar 2018, at 16:37, Russell Brown  wrote:
>> 
>> Hi Again,
>> 
>> More progress update. All the PRs are merged (thanks Bryan Hunt (ESL)
>> and Nick/Martin (bet365)).
>> 
>> I’m planning on tagging Riak with riak-2.2.5RC1 this week.
>> 
>> We haven’t been able to run load tests. I’m not 100% sure that all
>> basho’s releases had extensive load tests run, though I know that
>> later releases had a perf team, and MvM always load tested leveldb.
>> 
>> The aim is to have those willing parties that deploy riak, and
>> therefore have perf testing already set up, to test and report
>> back. The NHS will run load tests and report results back. I hope that 
>> others can do the same.
>> To the end we’ll probably tag RC1-noperftest.
>> 
>> There are a few failing riak tests (was Riak ever released without a
>> failing riak-test?) If you have the time/capacity to run riak-test,
>> and you’re interested in helping out, get in touch and I’ll help you
>> get started.
>> 
>> The failures, should one pique your interest:
>> 
>> datatypes - riak667_mixed-eleveldb
>> ensemble - ensemble_basic3-eleveldb ensemble_basic4-eleveldb
>> yoko - yz_crdt-eleveldb yz_solr_upgrade_downgrade-eleveldb
>> 
>> Let me know if you want to look into ensemble or yoko.
>> 
>> Still aiming to have this tagged by end-of-week.
>> 
>> Cheers
>> 
>> Russell
>> 
>> On 2 Mar 2018, at 09:44, Russell Brown  wrote:
>> 
>>> Hi,
>>> 
>>> Just an update on the progress of riak-2.2.5 release. I realize we said
>>> "end of 2017" and then "end of Jan" and then "end of Feb" and here we
>>> are, 1st March, spring is upon us, and still no 2.2.5. I thought it best
>>> to at least keep you posted.
>>> 
>>> Why no release? Well, we're not quite finished yet. In terms of what is
>>> left:
>>> 
>>> - a few PRs need review and merge against the upstream Basho repo (see
>>> [1] if you want to help there);
>>> - Opening of PRs for gsets;
>>> - Docs for the changes;
>>> - Release notes;
>>> - Tagging;
>>> - A final round of testing (and fixing?) after all is merged;
>>> - Client support. This is crucial, but all the above work only has
>>> client support in the Basho erlang clients. I'm hoping the community
>>> that uses the Java/Python/Ruby etc clients can step up here. But we can 
>>> release Riak before all client work is done.
>>> 
>>> The optimist in me says 2 weeks.
>>> 
>>> What do I mean "release"?
>>> 
>>> From my point of view the release is the tags. After that I'm sincerely
>>> hoping ESL will continue to kindly build and host the actual artifacts.
>>> 
>>> What's in the release?
>>> 
>>> As a reminder, since it's been a while since StokeCon17, this release
>>> contains:
>>> 
>>> - Some developer clean-up around `make test` and `riak_test` to make
>>> them more reliable/trustworthy;
>>> - Open source MDC Repl (thanks bet365!);
>>> - A fix for a bug in riak core claim that led to unbalanced rings[2];
>>> - `node_confirms` a feature like `w` or `pw` but for physical diversity in 
>>> durability[3];
>>> - `participate in coverage` an admin setting that takes a node out of
>>> the coverage plan (for example after adding a node while transfers
>>> take place);
>>> - Riak repl fix, and change to unsafe default behaviour[4];
>>> - Addition of a GSet to riak data types;
>>> - Fix to repl stats[5].
>>> 
>>> Sorry if I missed anything. The release notes will have it all.
>>> 
>>> Work is already begun on Riak 3.0 with OTP20 support well under way.
>>> Some candidates for inclusion that we're working on are a new
>>> pure-Erlang backend[6], and a radical overhaul of AAE for both intra and 
>>> inter-cluster anti-entropy.
>>> 
>>> Sorry for the delay, thanks for your patience. We’ll keep you posted.
>>> 
>>> Cheers
>>> 
>>> Russell
>>> 
>>> Titus Systems - ti-sys.co.uk
>>> 
>>> [1] Coverage:
>>> https://github.com/basho/riak_core/pull/917
>>> https://github.com/basho/riak_kv/pull/1664
>>> https://github.com/basho/riak_test/pull/1300
>>> 
>>> Repl:
>>> https://github.com/basho/riak_repl/pull/777
>>> https://github.com/basho/riak_test/pull/1301
>>> 
>>> Node Confirms:
>>> https://github.com/basho/riak_test/pull/1299
>>> https://github.com/basho/riak_test/pull/1298
>>> https://github.com/basho/riak-erlang-client/pull/371
>>> https://github.com/basho/riak_core/pull/915
>>> https://github.com/basho/riak-erlang-http-client/pull/69
>>> 
>>> [2]
>>> https://github.com/basho/riak_core/blob/develop-2.2/docs/claim-fixes.md
>>> [3]
>>> 

Re: Riak-2.2.5 progress update

[Bryan Hunt]

Could you share URL for those outstanding failures please. B

> On 27 Mar 2018, at 16:37, Russell Brown  wrote:
> 
> Hi Again,
> 
> More progress update. All the PRs are merged (thanks Bryan Hunt (ESL)
> and Nick/Martin (bet365)).
> 
> I’m planning on tagging Riak with riak-2.2.5RC1 this week.
> 
> We haven’t been able to run load tests. I’m not 100% sure that all
> basho’s releases had extensive load tests run, though I know that
> later releases had a perf team, and MvM always load tested leveldb.
> 
> The aim is to have those willing parties that deploy riak, and
> therefore have perf testing already set up, to test and report
> back. The NHS will run load tests and report results back. I hope that others 
> can do the same.
> To the end we’ll probably tag RC1-noperftest.
> 
> There are a few failing riak tests (was Riak ever released without a
> failing riak-test?) If you have the time/capacity to run riak-test,
> and you’re interested in helping out, get in touch and I’ll help you
> get started.
> 
> The failures, should one pique your interest:
> 
> datatypes - riak667_mixed-eleveldb
> ensemble - ensemble_basic3-eleveldb ensemble_basic4-eleveldb
> yoko - yz_crdt-eleveldb yz_solr_upgrade_downgrade-eleveldb
> 
> Let me know if you want to look into ensemble or yoko.
> 
> Still aiming to have this tagged by end-of-week.
> 
> Cheers
> 
> Russell
> 
> On 2 Mar 2018, at 09:44, Russell Brown  wrote:
> 
>> Hi,
>> 
>> Just an update on the progress of riak-2.2.5 release. I realize we said
>> "end of 2017" and then "end of Jan" and then "end of Feb" and here we
>> are, 1st March, spring is upon us, and still no 2.2.5. I thought it best
>> to at least keep you posted.
>> 
>> Why no release? Well, we're not quite finished yet. In terms of what is
>> left:
>> 
>> - a few PRs need review and merge against the upstream Basho repo (see
>> [1] if you want to help there);
>> - Opening of PRs for gsets;
>> - Docs for the changes;
>> - Release notes;
>> - Tagging;
>> - A final round of testing (and fixing?) after all is merged;
>> - Client support. This is crucial, but all the above work only has
>> client support in the Basho erlang clients. I'm hoping the community
>> that uses the Java/Python/Ruby etc clients can step up here. But we can 
>> release Riak before all client work is done.
>> 
>> The optimist in me says 2 weeks.
>> 
>> What do I mean "release"?
>> 
>> From my point of view the release is the tags. After that I'm sincerely
>> hoping ESL will continue to kindly build and host the actual artifacts.
>> 
>> What's in the release?
>> 
>> As a reminder, since it's been a while since StokeCon17, this release
>> contains:
>> 
>> - Some developer clean-up around `make test` and `riak_test` to make
>> them more reliable/trustworthy;
>> - Open source MDC Repl (thanks bet365!);
>> - A fix for a bug in riak core claim that led to unbalanced rings[2];
>> - `node_confirms` a feature like `w` or `pw` but for physical diversity in 
>> durability[3];
>> - `participate in coverage` an admin setting that takes a node out of
>> the coverage plan (for example after adding a node while transfers
>> take place);
>> - Riak repl fix, and change to unsafe default behaviour[4];
>> - Addition of a GSet to riak data types;
>> - Fix to repl stats[5].
>> 
>> Sorry if I missed anything. The release notes will have it all.
>> 
>> Work is already begun on Riak 3.0 with OTP20 support well under way.
>> Some candidates for inclusion that we're working on are a new
>> pure-Erlang backend[6], and a radical overhaul of AAE for both intra and 
>> inter-cluster anti-entropy.
>> 
>> Sorry for the delay, thanks for your patience. We’ll keep you posted.
>> 
>> Cheers
>> 
>> Russell
>> 
>> Titus Systems - ti-sys.co.uk
>> 
>> [1] Coverage:
>> https://github.com/basho/riak_core/pull/917
>> https://github.com/basho/riak_kv/pull/1664
>> https://github.com/basho/riak_test/pull/1300
>> 
>> Repl:
>> https://github.com/basho/riak_repl/pull/777
>> https://github.com/basho/riak_test/pull/1301
>> 
>> Node Confirms:
>> https://github.com/basho/riak_test/pull/1299
>> https://github.com/basho/riak_test/pull/1298
>> https://github.com/basho/riak-erlang-client/pull/371
>> https://github.com/basho/riak_core/pull/915
>> https://github.com/basho/riak-erlang-http-client/pull/69
>> 
>> [2]
>> https://github.com/basho/riak_core/blob/develop-2.2/docs/claim-fixes.md
>> [3]
>> https://github.com/ramensen/riak_kv/blob/rs-physical-promises/docs/Node-Diversity.md
>> [4] https://github.com/basho/riak_repl/issues/774
>> https://github.com/basho/riak_repl/issues/772
>> [5] https://github.com/basho/riak_repl/pull/776
>> [6] https://github.com/martinsumner/leveled/
>> 
>> 
>> 
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> 
> 
> ___
> riak-users mailing list
> 

Re: Riak-2.2.5 progress update

[Russell Brown]

Hi Again,

More progress update. All the PRs are merged (thanks Bryan Hunt (ESL)
and Nick/Martin (bet365)).

I’m planning on tagging Riak with riak-2.2.5RC1 this week.

We haven’t been able to run load tests. I’m not 100% sure that all
basho’s releases had extensive load tests run, though I know that
later releases had a perf team, and MvM always load tested leveldb.

The aim is to have those willing parties that deploy riak, and
therefore have perf testing already set up, to test and report
back. The NHS will run load tests and report results back. I hope that others 
can do the same.
To the end we’ll probably tag RC1-noperftest.

There are a few failing riak tests (was Riak ever released without a
failing riak-test?) If you have the time/capacity to run riak-test,
and you’re interested in helping out, get in touch and I’ll help you
get started.

The failures, should one pique your interest:

datatypes - riak667_mixed-eleveldb
ensemble - ensemble_basic3-eleveldb ensemble_basic4-eleveldb
yoko - yz_crdt-eleveldb yz_solr_upgrade_downgrade-eleveldb

Let me know if you want to look into ensemble or yoko.

Still aiming to have this tagged by end-of-week.

Cheers

Russell

On 2 Mar 2018, at 09:44, Russell Brown  wrote:

> Hi,
> 
> Just an update on the progress of riak-2.2.5 release. I realize we said
> "end of 2017" and then "end of Jan" and then "end of Feb" and here we
> are, 1st March, spring is upon us, and still no 2.2.5. I thought it best
> to at least keep you posted.
> 
> Why no release? Well, we're not quite finished yet. In terms of what is
> left:
> 
> - a few PRs need review and merge against the upstream Basho repo (see
> [1] if you want to help there);
> - Opening of PRs for gsets;
> - Docs for the changes;
> - Release notes;
> - Tagging;
> - A final round of testing (and fixing?) after all is merged;
> - Client support. This is crucial, but all the above work only has
> client support in the Basho erlang clients. I'm hoping the community
> that uses the Java/Python/Ruby etc clients can step up here. But we can 
> release Riak before all client work is done.
> 
> The optimist in me says 2 weeks.
> 
> What do I mean "release"?
> 
> From my point of view the release is the tags. After that I'm sincerely
> hoping ESL will continue to kindly build and host the actual artifacts.
> 
> What's in the release?
> 
> As a reminder, since it's been a while since StokeCon17, this release
> contains:
> 
> - Some developer clean-up around `make test` and `riak_test` to make
> them more reliable/trustworthy;
> - Open source MDC Repl (thanks bet365!);
> - A fix for a bug in riak core claim that led to unbalanced rings[2];
> - `node_confirms` a feature like `w` or `pw` but for physical diversity in 
> durability[3];
> - `participate in coverage` an admin setting that takes a node out of
> the coverage plan (for example after adding a node while transfers
> take place);
> - Riak repl fix, and change to unsafe default behaviour[4];
> - Addition of a GSet to riak data types;
> - Fix to repl stats[5].
> 
> Sorry if I missed anything. The release notes will have it all.
> 
> Work is already begun on Riak 3.0 with OTP20 support well under way.
> Some candidates for inclusion that we're working on are a new
> pure-Erlang backend[6], and a radical overhaul of AAE for both intra and 
> inter-cluster anti-entropy.
> 
> Sorry for the delay, thanks for your patience. We’ll keep you posted.
> 
> Cheers
> 
> Russell
> 
> Titus Systems - ti-sys.co.uk
> 
> [1] Coverage:
> https://github.com/basho/riak_core/pull/917
> https://github.com/basho/riak_kv/pull/1664
> https://github.com/basho/riak_test/pull/1300
> 
> Repl:
> https://github.com/basho/riak_repl/pull/777
> https://github.com/basho/riak_test/pull/1301
> 
> Node Confirms:
> https://github.com/basho/riak_test/pull/1299
> https://github.com/basho/riak_test/pull/1298
> https://github.com/basho/riak-erlang-client/pull/371
> https://github.com/basho/riak_core/pull/915
> https://github.com/basho/riak-erlang-http-client/pull/69
> 
> [2]
> https://github.com/basho/riak_core/blob/develop-2.2/docs/claim-fixes.md
> [3]
> https://github.com/ramensen/riak_kv/blob/rs-physical-promises/docs/Node-Diversity.md
> [4] https://github.com/basho/riak_repl/issues/774
> https://github.com/basho/riak_repl/issues/772
> [5] https://github.com/basho/riak_repl/pull/776
> [6] https://github.com/martinsumner/leveled/
> 
> 
> 
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: OTP20+ and rebar3 for Riak 3.0

[DeadZen]

deadzen on skype, dead...@deadzen.com

On Mon, Feb 5, 2018 at 10:10 AM <martin@bet365.com> wrote:

> Just sorting out the logistics of this meeting with Russell - it's going
> to be done over Skype. We need to grab an email address from those who are
> wanting to join in, so we can send out the invite.
>
> As mentioned before, bet365 are keen to get Riak building on OTP20, as are
> a few others. The more users / orgs that we can get on-board, the better
> for discussion, decisions etc.
>
> Please reply with your address and we'll get it sent out.
>
> Thanks
>
> Martin
> 
> From: riak-users [riak-users-boun...@lists.basho.com] on behalf of
> martin@bet365.com [martin@bet365.com]
> Sent: 01 February 2018 11:57
> To: russell.br...@mac.com; riak-users@lists.basho.com
> Subject: RE: OTP20+ and rebar3 for Riak 3.0
>
> Just to echo what had been said on Slack already, myself and others from
> bet365 will be joining. Definitely on-board with getting the project to a
> current OTP version. Had started to look at some of the work - so it's
> going to be useful to get consensus on things such as the desire to remain
> backwards compatible etc.
>
> Thanks
>
> Martin
>
> 
> From: riak-users [riak-users-boun...@lists.basho.com] on behalf of
> Russell Brown [russell.br...@mac.com]
> Sent: 29 January 2018 21:28
> To: riak-users
> Subject: OTP20+ and rebar3 for Riak 3.0
>
> It _really_ needs doing. And a whole host of people have said they’re keen
> to work on it.
>
> Ted Burghart did a lot of work on this at Basho (talk here:
> https://www.youtube.com/watch?v=TcUTsYjEon4) and of course Heinz Gies has
> made riak_core_ng available for some time.
>
> At least 3 organisations have expressed a willingness to do the work, and
> as much as I hate meetings, I think we need to talk to get organised to get
> this done, rather than each org/individual attacking alone.
>
> I hate Wednesdays almost as much as I hate meetings, so how does 1700 GMT,
> Wednesday 7th February sound?
>
> Cheers
>
> Russell
>
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> This email and any files transmitted with it are confidential and contain
> information which may be privileged or confidential and are intended solely
> to be for the use of the individual(s) or entity to which they are
> addressed. If you are not the intended recipient be aware that any
> disclosure, copying, distribution or use of the contents of this
> information is strictly prohibited and may be illegal. If you have received
> this email in error, please notify us by telephone or email immediately and
> delete it from your system. Activity and use of our email system is
> monitored to secure its effective operation and for other lawful business
> purposes. Communications using this system will also be monitored and may
> be recorded to secure effective operation and for other lawful business
> purposes. Internet emails are not necessarily secure. We do not accept
> responsibility for changes made to this message after it was sent. You are
> advised to scan this message for viruses and we cannot accept liability for
> any loss or damage which may be caused as a result of any computer virus.
>
> This email is sent by a bet365 group entity. The bet365 group includes the
> following entities: Hillside (Shared Services) Limited (registration no.
> 3958393), Hillside (Spain New Media) Plc (registration no. 07833226),
> bet365 Group Limited (registration no. 4241161), Hillside (Technology)
> Limited (registration no. 8273456), Hillside (Media Services) Limited
> (registration no. 9171710), Hillside (Trader Services) Limited
> (registration no. 9171598) each registered in England and Wales with a
> registered office address at bet365 House, Media Way, Stoke-on-Trent, ST1
> 5SZ, United Kingdom; Hillside (Gibraltar) Limited (registration no. 97927),
> Hillside (Sports) GP Limited (registration no. 111829) and Hillside
> (Gaming) GP Limited (registered no. 111830) each registered in Gibraltar
> with a registered office address at Unit 1.1, First Floor, Waterport Place, 2
> Europort Avenue, Gibraltar
> <https://maps.google.com/?q=2+Europort+Avenue,+Gibraltar=gmail=g>;
> Hillside (UK Sports) LP (registration no. 117), Hillside (Sports) LP
> (registration no. 118), Hillside (International Sports) LP (registration
> no. 119), Hillside (Gaming) LP (registration no. 120) and Hillside
> (International Gaming) LP (registration no. 121) each registered in
> Gibraltar with a princi

Re: Meetup at ESL offices riak_core vs partisan

[Bryan Hunt]

It will be - the talks are posted under Erlang Solutions channel 
 on YouTube - I’ll 
post the link afterward. 

Bryan 

> On 16 Jan 2018, at 15:55, Nick Marino  wrote:
> 
> That sounds like a great talk, any chance of it being videoed and posted 
> online?
> 
> Nick
> 
> On Tue, Jan 16, 2018 at 9:37 AM, Bryan Hunt  > wrote:
> Meetup event at ESL London offices tomorrow (Wednesday) evening 
> 
> https://www.meetup.com/riak-london/events/246719322/ 
> 
> 
> Mariano Guerra is in London for his talk on building distributed 
> applications: riak_core vs partisan. 
> 
> Mariano will explore the similarities and differences between riak_core and 
> partisan, and what they have in common.
> 
> In addition, Mariano will be taking us through the process of building a 
> basic app using both plumtree for gossip and hyparview for peer membership.
> 
> Enjoy ! 
> 
> 
> 
> ___
> riak-users mailing list
> riak-users@lists.basho.com 
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com 
> 
> 
> 

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: K8n Riak statefulset ?

[Bryan Hunt]

Excellent. Thank you so much Deepthi. 

> On 15 Dec 2017, at 12:50, Deepthi Devaki  wrote:
> 
> We have done a test deployment of Antidote on AWS with kubernetes. Here is 
> the setup https://github.com/deepthidevaki/antidote-aws 
> . 
> 
> On Fri, Dec 15, 2017 at 12:26 PM, Bryan Hunt  > wrote:
> Thanks Chris. Much appreciated ! 
> 
> 
>> On 14 Dec 2017, at 21:57, Christopher Meiklejohn 
>> > 
>> wrote:
>> 
>> I believe the Antidote folks (Riak Core application) have.  I can forward 
>> this message on to them.
>> 
>> On Thu, Dec 14, 2017, 22:38 Bryan Hunt > > wrote:
>> Anyone done any work with Riak and k8n statefulset ? I see people already 
>> doing stuff with rabbitmq HA 
>> https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/ 
>>  
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com 
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com 
>> 
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com 
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com 
>> 
> 
> 
> 
> 
> -- 
> Regards,
> Deepthi Akkoorath
> http://dd.thekkedam.org/ 

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: K8n Riak statefulset ?

[Bryan Hunt]

Thanks Chris. Much appreciated ! 

> On 14 Dec 2017, at 21:57, Christopher Meiklejohn 
>  wrote:
> 
> I believe the Antidote folks (Riak Core application) have.  I can forward 
> this message on to them.
> 
> On Thu, Dec 14, 2017, 22:38 Bryan Hunt  > wrote:
> Anyone done any work with Riak and k8n statefulset ? I see people already 
> doing stuff with rabbitmq 
> HAhttps://kubernetes.io/docs/concepts/workloads/controllers/statefulset/ 
>  
> ___
> riak-users mailing list
> riak-users@lists.basho.com 
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com 
> 
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: WARNING: Not all replicas will be on distinct nodes

[Martin Sumner]

>> com/2017-August/019488.html) and the links in it for some more details
>> on issues with the core claim algorithm.  The fix is in the pending release
>> 2.2.5 which Russell is adding the finishing touches to at the moment.
>>
>> However, the fix may not immediately resolve your problem - the fix is
>> about preventing this situation, not necessarily about resolving it once it
>> has been created.  Also the issue we saw that would lead to this, would not
>> (I think) be triggered by adding a single node - unless the cluster already
>> had the problem.  So it is possible, although you are seeing the warning
>> now, you had the issue when your originally created the cluster, and the
>> change is just persisting the issue.  For instance going from nothing
>> straight to a 6-node with a ring-size of 128 would create this problem.
>>
>> As a workaround there is the core claim v3 algorithm which can be turned
>> on, and you can see if this offers a better cluster plan without
>> violations.  I can't right now remember how to trigger v3 claim algorithm
>> though - google letting me down.
>>
>> Ultimately, this may not be such a crisis.  The error is through whenever
>> the cluster cannot guarantee a "target_n_val" of 4.  So if you have an
>> n_val of 3 - you're not necessarily at risk of data loss.To know you
>> will have to look at your ring via riak attach (see bullet point 2 in
>> http://docs.basho.com/riak/kv/2.2.3/using/running-a-clust
>> er/#add-a-second-node-to-your-cluster).
>>
>> If you can figure out the violations from your ring, you may be able to
>> resolve by leaving the node that has the violations, and then re-adding it.
>>
>> Sorry, I'm a bit rushed - but I hope this helps get you started.
>>
>> Martin
>>
>>
>>
>>
>>
>> On 14 December 2017 at 19:49, Daniel Miller <dmil...@dimagi.com> wrote:
>>
>>> I have a 6 node cluster (now 7) with ring size 128. On adding the most
>>> recent node I got the WARNING: Not all replicas will be on distinct nodes.
>>> After the initial plan I ran the following sequence many times, but always
>>> got the same plan output:
>>>
>>> sudo riak-admin cluster clear && \
>>> sleep 10 && \
>>> sudo service riak start && \
>>> sudo riak-admin wait-for-service riak_kv && \
>>> sudo riak-admin cluster join riak@hqriak20.internal && \
>>> sudo riak-admin cluster plan
>>>
>>>
>>> The plan looked the same every time, and I eventually committed it
>>> because the cluster capacity is running low:
>>>
>>>
>>> Success: staged join request for 'riak@riak29.internal' to
>>> 'riak@riak20.internal'
>>> === Staged Changes
>>> 
>>> Action Details(s)
>>> 
>>> ---
>>> join   'riak@riak29.internal'
>>> 
>>> ---
>>>
>>>
>>> NOTE: Applying these changes will result in 1 cluster transition
>>>
>>> 
>>> ###
>>>  After cluster transition 1/1
>>> 
>>> ###
>>>
>>> = Membership
>>> ==
>>> Status RingPendingNode
>>> 
>>> ---
>>> valid  17.2% 14.1%'riak@riak20.internal'
>>> valid  17.2% 14.8%'riak@riak21.internal'
>>> valid  16.4% 14.1%'riak@riak22.internal'
>>> valid  16.4% 14.1%'riak@riak23.internal'
>>> valid  16.4% 14.1%'riak@riak24.internal'
>>> valid  16.4% 14.8%'riak@riak28.internal'
>>> valid   0.0% 14.1%'riak@riak29.internal'
>>> 
>>> ---
>>> Valid:7 / Leaving:0 / Exiting:0 / Joining:0 / Down:0
>>>
>>> WARNING: Not all replicas will be on distinct nodes
>>>
>>> Transfers resulting from cluster changes: 18
>>>   2 transfers from 'riak@riak28.internal' to 'riak@riak29.internal'
>>>   3 transfer

Re: WARNING: Not all replicas will be on distinct nodes

[Daniel Miller]

tter cluster plan without
> violations.  I can't right now remember how to trigger v3 claim algorithm
> though - google letting me down.
>
> Ultimately, this may not be such a crisis.  The error is through whenever
> the cluster cannot guarantee a "target_n_val" of 4.  So if you have an
> n_val of 3 - you're not necessarily at risk of data loss.To know you
> will have to look at your ring via riak attach (see bullet point 2 in
> http://docs.basho.com/riak/kv/2.2.3/using/running-a-clust
> er/#add-a-second-node-to-your-cluster).
>
> If you can figure out the violations from your ring, you may be able to
> resolve by leaving the node that has the violations, and then re-adding it.
>
> Sorry, I'm a bit rushed - but I hope this helps get you started.
>
> Martin
>
>
>
>
>
> On 14 December 2017 at 19:49, Daniel Miller <dmil...@dimagi.com> wrote:
>
>> I have a 6 node cluster (now 7) with ring size 128. On adding the most
>> recent node I got the WARNING: Not all replicas will be on distinct nodes.
>> After the initial plan I ran the following sequence many times, but always
>> got the same plan output:
>>
>> sudo riak-admin cluster clear && \
>> sleep 10 && \
>> sudo service riak start && \
>> sudo riak-admin wait-for-service riak_kv && \
>> sudo riak-admin cluster join riak@hqriak20.internal && \
>> sudo riak-admin cluster plan
>>
>>
>> The plan looked the same every time, and I eventually committed it
>> because the cluster capacity is running low:
>>
>>
>> Success: staged join request for 'riak@riak29.internal' to
>> 'riak@riak20.internal'
>> === Staged Changes
>> 
>> Action Details(s)
>> 
>> ---
>> join   'riak@riak29.internal'
>> 
>> ---
>>
>>
>> NOTE: Applying these changes will result in 1 cluster transition
>>
>> 
>> ###
>>  After cluster transition 1/1
>> 
>> ###
>>
>> = Membership
>> ==
>> Status RingPendingNode
>> 
>> ---
>> valid  17.2% 14.1%'riak@riak20.internal'
>> valid  17.2% 14.8%'riak@riak21.internal'
>> valid  16.4% 14.1%'riak@riak22.internal'
>> valid  16.4% 14.1%'riak@riak23.internal'
>> valid  16.4% 14.1%'riak@riak24.internal'
>> valid  16.4% 14.8%'riak@riak28.internal'
>> valid   0.0% 14.1%'riak@riak29.internal'
>> 
>> ---
>> Valid:7 / Leaving:0 / Exiting:0 / Joining:0 / Down:0
>>
>> WARNING: Not all replicas will be on distinct nodes
>>
>> Transfers resulting from cluster changes: 18
>>   2 transfers from 'riak@riak28.internal' to 'riak@riak29.internal'
>>   3 transfers from 'riak@riak21.internal' to 'riak@riak29.internal'
>>   3 transfers from 'riak@riak23.internal' to 'riak@riak29.internal'
>>   3 transfers from 'riak@riak24.internal' to 'riak@riak29.internal'
>>   4 transfers from 'riak@riak20.internal' to 'riak@riak29.internal'
>>   3 transfers from 'riak@riak22.internal' to 'riak@riak29.internal'
>>
>>
>> My understanding is that if some replicas are not on distinct nodes then
>> I may have permanent data loss if a single physical node is lost (please
>> let me know if that is not correct). Questions:
>>
>> How do I diagnose which node(s) have duplicate replicas?
>> What can I do to fix this situation?
>>
>> Thanks!
>> Daniel
>>
>>
>> P.S. I am unable to get anything useful out of `riak-admin diag`. It
>> appears to be broken on the version of Riak I'm using (2.2.1). Here's the
>> output I get:
>>
>> $ sudo riak-admin diag
>> RPC to 'riak@hqriak20.internal' failed: {'EXIT',
>>{undef,
>> [{lager,
>>
>> get_loglevels,
>>   [],[]},
>>
>> {riaknostic,run,
>>

Re: WARNING: Not all replicas will be on distinct nodes

[Martin Sumner]

Daniel,

See this post (
http://lists.basho.com/pipermail/riak-users_lists.basho.com/2017-August/019488.html)
and the links in it for some more details on issues with the core claim
algorithm.  The fix is in the pending release 2.2.5 which Russell is adding
the finishing touches to at the moment.

However, the fix may not immediately resolve your problem - the fix is
about preventing this situation, not necessarily about resolving it once it
has been created.  Also the issue we saw that would lead to this, would not
(I think) be triggered by adding a single node - unless the cluster already
had the problem.  So it is possible, although you are seeing the warning
now, you had the issue when your originally created the cluster, and the
change is just persisting the issue.  For instance going from nothing
straight to a 6-node with a ring-size of 128 would create this problem.

As a workaround there is the core claim v3 algorithm which can be turned
on, and you can see if this offers a better cluster plan without
violations.  I can't right now remember how to trigger v3 claim algorithm
though - google letting me down.

Ultimately, this may not be such a crisis.  The error is through whenever
the cluster cannot guarantee a "target_n_val" of 4.  So if you have an
n_val of 3 - you're not necessarily at risk of data loss.To know you
will have to look at your ring via riak attach (see bullet point 2 in
http://docs.basho.com/riak/kv/2.2.3/using/running-a-
cluster/#add-a-second-node-to-your-cluster).

If you can figure out the violations from your ring, you may be able to
resolve by leaving the node that has the violations, and then re-adding it.

Sorry, I'm a bit rushed - but I hope this helps get you started.

Martin





On 14 December 2017 at 19:49, Daniel Miller <dmil...@dimagi.com> wrote:

> I have a 6 node cluster (now 7) with ring size 128. On adding the most
> recent node I got the WARNING: Not all replicas will be on distinct nodes.
> After the initial plan I ran the following sequence many times, but always
> got the same plan output:
>
> sudo riak-admin cluster clear && \
> sleep 10 && \
> sudo service riak start && \
> sudo riak-admin wait-for-service riak_kv && \
> sudo riak-admin cluster join riak@hqriak20.internal && \
> sudo riak-admin cluster plan
>
>
> The plan looked the same every time, and I eventually committed it because
> the cluster capacity is running low:
>
>
> Success: staged join request for 'riak@riak29.internal' to
> 'riak@riak20.internal'
> === Staged Changes
> 
> Action Details(s)
> 
> ---
> join   'riak@riak29.internal'
> 
> ---
>
>
> NOTE: Applying these changes will result in 1 cluster transition
>
> 
> ###
>  After cluster transition 1/1
> 
> ###
>
> = Membership
> ==
> Status RingPendingNode
> 
> ---
> valid  17.2% 14.1%'riak@riak20.internal'
> valid  17.2% 14.8%'riak@riak21.internal'
> valid  16.4% 14.1%'riak@riak22.internal'
> valid  16.4% 14.1%'riak@riak23.internal'
> valid  16.4% 14.1%'riak@riak24.internal'
> valid  16.4% 14.8%'riak@riak28.internal'
> valid   0.0% 14.1%'riak@riak29.internal'
> 
> ---
> Valid:7 / Leaving:0 / Exiting:0 / Joining:0 / Down:0
>
> WARNING: Not all replicas will be on distinct nodes
>
> Transfers resulting from cluster changes: 18
>   2 transfers from 'riak@riak28.internal' to 'riak@riak29.internal'
>   3 transfers from 'riak@riak21.internal' to 'riak@riak29.internal'
>   3 transfers from 'riak@riak23.internal' to 'riak@riak29.internal'
>   3 transfers from 'riak@riak24.internal' to 'riak@riak29.internal'
>   4 transfers from 'riak@riak20.internal' to 'riak@riak29.internal'
>   3 transfers from 'riak@riak22.internal' to 'riak@riak29.internal'
>
>
> My understanding is that if some replicas are not on distinct nodes then I
> may have permanent data loss if a single physical node is lost (please let
> me know if that is not correct). Questions:
>
> How do I diagnose which node(s) have duplicate replicas?
> What can I do to fix this situatio

Re: riak_repl integration

[Bryan Hunt]

Raghu,

We’re mid way through building packages, in the meantime, if you’re feeling 
impatient you can try the following : 

https://gist.github.com/bryanhuntesl/cc0eefa6939b757ca7b86ebb1d7b29b2 
<https://gist.github.com/bryanhuntesl/cc0eefa6939b757ca7b86ebb1d7b29b2>

The above recipe requires a reasonably recent Docker daemon and cli tools 
installed on the host. 

Best Regards,

Bryan Hunt



> On 5 Dec 2017, at 19:17, Raghavendra Sayana 
> <raghavendra.sayana.t...@statefarm.com> wrote:
> 
> Hey Bryan,
> 
> We are running riak 2.1.4 on RHEL 6
> 
> Thanks
> Raghu
> 
> -Original Message-
> From: Bryan Hunt [mailto:bryan.h...@erlang-solutions.com] 
> Sent: Tuesday, December 05, 2017 12:59 PM
> To: Raghavendra Sayana <raghavendra.sayana.t...@statefarm.com>
> Cc: Dan Sweeney <dan.sweeney.n...@statefarm.com>; riak-users 
> <riak-users@lists.basho.com>
> Subject: Re: riak_repl integration
> 
> Raghu, 
> 
> What bistro are you running? 
> 
> Bryan 
> 
>> On 5 Dec 2017, at 17:10, Russell Brown <russell.br...@icloud.com> wrote:
>> 
>> Hi Raghu,
>> At present Riak is still stuck on the r16 basho OTP. This will change next 
>> year. For now, and the next release of riak, r16 is the compiler you need.
>> 
>> If you want to use riak_repl with open source riak, there are a couple of 
>> options. There is the 2.2.4 tag in the basho riak repo, which bet365 kindly 
>> put together when riak_repl was open sourced. Cloning 
>> https://github.com/basho/riak and checking out tag 2.2.4, then running `make 
>> rel` will get you a local release of OS riak as it was at 2.2.3 + riak_repl. 
>> Someone did mention maybe building packages of 2.2.4, if you’d rather 
>> download than build your own, but I don’t know if that happened yet.
>> 
>> In order to _use_ MDC you can follow the docs on basho’s site 
>> http://docs.basho.com/riak/kv/2.2.3/configuring/v3-multi-datacenter/.
>> 
>> There is a branch develop-2.2.5 which is the active work for the next 
>> release of riak, which will contain riak_repl.  We plan to release 
>> riak-2.2.5 (OS riak+repl, and some small features+bug fixes) in early 2018.
>> 
>> Hope that helps
>> 
>> Cheers
>> 
>> Russell
>> 
>> 
>> On 5 Dec 2017, at 16:11, Raghavendra Sayana 
>> <raghavendra.sayana.t...@statefarm.com> wrote:
>> 
>>> Hi All,
>>> 
>>> I want to use riak MDC replication using riak_repl module. Is there any 
>>> documentation on how I can perform this integration? Do you know what 
>>> erlang compiler version I should be using to compile the code? Any help on 
>>> this is appreciated.
>>> 
>>> Thanks
>>> Raghu
>>> 
>>> ___
>>> riak-users mailing list
>>> riak-users@lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> 
>> 
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> 
> 

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: riak_repl integration

[Bryan Hunt]

Raghu, 

What bistro are you running? 

Bryan 

> On 5 Dec 2017, at 17:10, Russell Brown  wrote:
> 
> Hi Raghu,
> At present Riak is still stuck on the r16 basho OTP. This will change next 
> year. For now, and the next release of riak, r16 is the compiler you need.
> 
> If you want to use riak_repl with open source riak, there are a couple of 
> options. There is the 2.2.4 tag in the basho riak repo, which bet365 kindly 
> put together when riak_repl was open sourced. Cloning 
> https://github.com/basho/riak and checking out tag 2.2.4, then running `make 
> rel` will get you a local release of OS riak as it was at 2.2.3 + riak_repl. 
> Someone did mention maybe building packages of 2.2.4, if you’d rather 
> download than build your own, but I don’t know if that happened yet.
> 
> In order to _use_ MDC you can follow the docs on basho’s site 
> http://docs.basho.com/riak/kv/2.2.3/configuring/v3-multi-datacenter/.
> 
> There is a branch develop-2.2.5 which is the active work for the next release 
> of riak, which will contain riak_repl.  We plan to release riak-2.2.5 (OS 
> riak+repl, and some small features+bug fixes) in early 2018.
> 
> Hope that helps
> 
> Cheers
> 
> Russell
> 
> 
> On 5 Dec 2017, at 16:11, Raghavendra Sayana 
>  wrote:
> 
>> Hi All,
>> 
>> I want to use riak MDC replication using riak_repl module. Is there any 
>> documentation on how I can perform this integration? Do you know what erlang 
>> compiler version I should be using to compile the code? Any help on this is 
>> appreciated.
>> 
>> Thanks
>> Raghu
>> 
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> 
> 
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: riak_repl integration

[Russell Brown]

Hi Raghu,
At present Riak is still stuck on the r16 basho OTP. This will change next 
year. For now, and the next release of riak, r16 is the compiler you need.

If you want to use riak_repl with open source riak, there are a couple of 
options. There is the 2.2.4 tag in the basho riak repo, which bet365 kindly put 
together when riak_repl was open sourced. Cloning https://github.com/basho/riak 
and checking out tag 2.2.4, then running `make rel` will get you a local 
release of OS riak as it was at 2.2.3 + riak_repl. Someone did mention maybe 
building packages of 2.2.4, if you’d rather download than build your own, but I 
don’t know if that happened yet.

In order to _use_ MDC you can follow the docs on basho’s site 
http://docs.basho.com/riak/kv/2.2.3/configuring/v3-multi-datacenter/.

There is a branch develop-2.2.5 which is the active work for the next release 
of riak, which will contain riak_repl.  We plan to release riak-2.2.5 (OS 
riak+repl, and some small features+bug fixes) in early 2018.

Hope that helps

Cheers

Russell


On 5 Dec 2017, at 16:11, Raghavendra Sayana 
 wrote:

> Hi All,
>  
> I want to use riak MDC replication using riak_repl module. Is there any 
> documentation on how I can perform this integration? Do you know what erlang 
> compiler version I should be using to compile the code? Any help on this is 
> appreciated.
>  
> Thanks
> Raghu
>  
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Deprecation of Riak SNMP and Riak JMX?

[Christopher Meiklejohn]

Yes, I'm in agreement that they should be kept around, but if you require
that support, you should build Riak from source for it.

Christopher

On Mon, Nov 13, 2017, 16:51 Bryan Hunt 
wrote:

> Already discussed on Slack but just to be public:
>
> I’m in favour of keeping the repositories around but not including them as
> Riak dependencies or supporting them going forward.
>
> Any others ?
>
> > On 13 Nov 2017, at 10:48, Russell Brown  wrote:
> >
> > Hi all,
> > It looks like we’re moving toward shipping the open source repl code
> with the next release of Riak.
> >
> > I’m canvassing for opinions about the riak_snmp and riak_jmx portions of
> the enterprise code. Is there anyone out there that depends on these
> features? I’d like to deprecate them in the next release, and remove them
> in the release following that.
> >
> > Cheers
> >
> > Russell
> > ___
> > riak-users mailing list
> > riak-users@lists.basho.com
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Slack (was: Re: Deprecation of Riak SNMP and Riak JMX?)

[Bryan Hunt]

To not include them in the open-source release. 

> On 13 Nov 2017, at 16:00, martin@bet365.com wrote:
> 
> I'm in support of deprecating. Is the intention to include them in the 
> open-source release, only to remove them in the following release? May as 
> well not include them to start with, I'd have thought?
> 
> Martin Cox
> Software Developer


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com