Query on Riak Search in a cluster of 3 nodes behind ELB is giving different result everytime
Hello everyone,
I've just joined the list, and am a bit late to the party. Sorry about
that. Thought I would contribute an answer anyway.
Santi, what is you n_val?
I observed the behavior you are describing on Riak 2.0.0 with n_val=3 in
two cases:
1. One of the nodes was not part of the cluster although the cluster was
thinking it was. Don't ask me how that happened. Obviously, when a request
hit that node part of the entries could not be found there.
2. Look for indexing errors in Solr console and Riak logs. Each Riak node
has its own Solr repository, so if an entry fails to be indexed on any of
them, search results will be inconsistent depending on which set of nodes
returns it. Let's say you have replicas on nodes A, B, and C. Entry X
failed to be indexed on A, entry Y failed to be indexed on A and B, and
entry Z was indexed OK on all nodes. Then you may get {X,Y,Z}, {X,Z}, or
{Z} as your search results.
In our case the indexing failures were caused by disk/filesystem errors.
Regards,
Vitaly
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Query on Riak Search in a cluster of 3 nodes behind ELB is giving different result everytime
Oh, ok. Cool to know :) thanks
On Wed, Apr 1, 2015 at 10:06 AM, Vitaliy E 13vitam...@gmail.com wrote:
Hi Alex,
riak-admin member-status is the command
(N - 1) nodes were showing all N nodes as part of the cluster, but Ring
Ready: false at the same time. One node was showing just itself in
risk-admin member-status. Strangely, when shut down its status was
reflected in the admin console of the cluster, but no replication was done
so I'm sure it wasn't working together with the rest of the nodes.
Repaired by joining the cluster as if it was never attempted before:
riak-admin cluster join node
riak-admin cluster plan
riak-admin cluster commit
Regards,
Vitaliy
On Wed, Apr 1, 2015 at 10:54 AM, Alex De la rosa alex.rosa@gmail.com
wrote:
Hi Vitaliy,
How did you find out a node in the cluster was not part in the cluster?
any commands to check that? And then, how did you fix that? (Just curious
and for future references)
Thanks,
Alex
On Wed, Apr 1, 2015 at 9:50 AM, Vitaliy E 13vitam...@gmail.com wrote:
Hello everyone,
I've just joined the list, and am a bit late to the party. Sorry about
that. Thought I would contribute an answer anyway.
Santi, what is you n_val?
I observed the behavior you are describing on Riak 2.0.0 with n_val=3 in
two cases:
1. One of the nodes was not part of the cluster although the cluster was
thinking it was. Don't ask me how that happened. Obviously, when a request
hit that node part of the entries could not be found there.
2. Look for indexing errors in Solr console and Riak logs. Each Riak
node has its own Solr repository, so if an entry fails to be indexed on
any of them, search results will be inconsistent depending on which set of
nodes returns it. Let's say you have replicas on nodes A, B, and C. Entry X
failed to be indexed on A, entry Y failed to be indexed on A and B, and
entry Z was indexed OK on all nodes. Then you may get {X,Y,Z}, {X,Z}, or
{Z} as your search results.
In our case the indexing failures were caused by disk/filesystem errors.
Regards,
Vitaly
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Query on Riak Search in a cluster of 3 nodes behind ELB is giving different result everytime
Hi Vitaliy,
How did you find out a node in the cluster was not part in the cluster? any
commands to check that? And then, how did you fix that? (Just curious and
for future references)
Thanks,
Alex
On Wed, Apr 1, 2015 at 9:50 AM, Vitaliy E 13vitam...@gmail.com wrote:
Hello everyone,
I've just joined the list, and am a bit late to the party. Sorry about
that. Thought I would contribute an answer anyway.
Santi, what is you n_val?
I observed the behavior you are describing on Riak 2.0.0 with n_val=3 in
two cases:
1. One of the nodes was not part of the cluster although the cluster was
thinking it was. Don't ask me how that happened. Obviously, when a request
hit that node part of the entries could not be found there.
2. Look for indexing errors in Solr console and Riak logs. Each Riak node
has its own Solr repository, so if an entry fails to be indexed on any of
them, search results will be inconsistent depending on which set of nodes
returns it. Let's say you have replicas on nodes A, B, and C. Entry X
failed to be indexed on A, entry Y failed to be indexed on A and B, and
entry Z was indexed OK on all nodes. Then you may get {X,Y,Z}, {X,Z}, or
{Z} as your search results.
In our case the indexing failures were caused by disk/filesystem errors.
Regards,
Vitaly
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Query on Riak Search in a cluster of 3 nodes behind ELB is giving different result everytime
Hi Alex,
riak-admin member-status is the command
(N - 1) nodes were showing all N nodes as part of the cluster, but Ring
Ready: false at the same time. One node was showing just itself in
risk-admin member-status. Strangely, when shut down its status was
reflected in the admin console of the cluster, but no replication was done
so I'm sure it wasn't working together with the rest of the nodes.
Repaired by joining the cluster as if it was never attempted before:
riak-admin cluster join node
riak-admin cluster plan
riak-admin cluster commit
Regards,
Vitaliy
On Wed, Apr 1, 2015 at 10:54 AM, Alex De la rosa alex.rosa@gmail.com
wrote:
Hi Vitaliy,
How did you find out a node in the cluster was not part in the cluster?
any commands to check that? And then, how did you fix that? (Just curious
and for future references)
Thanks,
Alex
On Wed, Apr 1, 2015 at 9:50 AM, Vitaliy E 13vitam...@gmail.com wrote:
Hello everyone,
I've just joined the list, and am a bit late to the party. Sorry about
that. Thought I would contribute an answer anyway.
Santi, what is you n_val?
I observed the behavior you are describing on Riak 2.0.0 with n_val=3 in
two cases:
1. One of the nodes was not part of the cluster although the cluster was
thinking it was. Don't ask me how that happened. Obviously, when a request
hit that node part of the entries could not be found there.
2. Look for indexing errors in Solr console and Riak logs. Each Riak node
has its own Solr repository, so if an entry fails to be indexed on any of
them, search results will be inconsistent depending on which set of nodes
returns it. Let's say you have replicas on nodes A, B, and C. Entry X
failed to be indexed on A, entry Y failed to be indexed on A and B, and
entry Z was indexed OK on all nodes. Then you may get {X,Y,Z}, {X,Z}, or
{Z} as your search results.
In our case the indexing failures were caused by disk/filesystem errors.
Regards,
Vitaly
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Query on Riak Search in a cluster of 3 nodes behind ELB is giving different result everytime
n_val is 3 for us and we have 3 nodes
On Apr 1, 2015 1:21 PM, Vitaliy E 13vitam...@gmail.com wrote:
Hello everyone,
I've just joined the list, and am a bit late to the party. Sorry about
that. Thought I would contribute an answer anyway.
Santi, what is you n_val?
I observed the behavior you are describing on Riak 2.0.0 with n_val=3 in
two cases:
1. One of the nodes was not part of the cluster although the cluster was
thinking it was. Don't ask me how that happened. Obviously, when a request
hit that node part of the entries could not be found there.
2. Look for indexing errors in Solr console and Riak logs. Each Riak node
has its own Solr repository, so if an entry fails to be indexed on any of
them, search results will be inconsistent depending on which set of nodes
returns it. Let's say you have replicas on nodes A, B, and C. Entry X
failed to be indexed on A, entry Y failed to be indexed on A and B, and
entry Z was indexed OK on all nodes. Then you may get {X,Y,Z}, {X,Z}, or
{Z} as your search results.
In our case the indexing failures were caused by disk/filesystem errors.
Regards,
Vitaly
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Query on Riak Search in a cluster of 3 nodes behind ELB is giving different result everytime
Hey Vitaly,
experienced the same issue recently (see
http://lists.basho.com/pipermail/riak-users_lists.basho.com/2015-March/016972.html).
When you have a look at your now working solr instances admin pages, do you
have everywhere the index counts the same? (i.e. the numDocs, maxDocs,
deletedDocs)
Thanks,
Michael
2015-04-01 9:50 GMT+02:00 Vitaliy E 13vitam...@gmail.com:
Hello everyone,
I've just joined the list, and am a bit late to the party. Sorry about
that. Thought I would contribute an answer anyway.
Santi, what is you n_val?
I observed the behavior you are describing on Riak 2.0.0 with n_val=3 in
two cases:
1. One of the nodes was not part of the cluster although the cluster was
thinking it was. Don't ask me how that happened. Obviously, when a request
hit that node part of the entries could not be found there.
2. Look for indexing errors in Solr console and Riak logs. Each Riak node
has its own Solr repository, so if an entry fails to be indexed on any of
them, search results will be inconsistent depending on which set of nodes
returns it. Let's say you have replicas on nodes A, B, and C. Entry X
failed to be indexed on A, entry Y failed to be indexed on A and B, and
entry Z was indexed OK on all nodes. Then you may get {X,Y,Z}, {X,Z}, or
{Z} as your search results.
In our case the indexing failures were caused by disk/filesystem errors.
Regards,
Vitaly
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Query on Riak Search in a cluster of 3 nodes behind ELB is giving different result everytime
So the problem I've described may be what you are experiencing (3
replicas). Do you have any indexing errors in the logs? What is riak-admin
member-status showing on each node?
On Wed, Apr 1, 2015 at 11:02 AM, Santi Kumar sa...@veradocs.com wrote:
n_val is 3 for us and we have 3 nodes
On Apr 1, 2015 1:21 PM, Vitaliy E 13vitam...@gmail.com wrote:
Hello everyone,
I've just joined the list, and am a bit late to the party. Sorry about
that. Thought I would contribute an answer anyway.
Santi, what is you n_val?
I observed the behavior you are describing on Riak 2.0.0 with n_val=3 in
two cases:
1. One of the nodes was not part of the cluster although the cluster was
thinking it was. Don't ask me how that happened. Obviously, when a request
hit that node part of the entries could not be found there.
2. Look for indexing errors in Solr console and Riak logs. Each Riak node
has its own Solr repository, so if an entry fails to be indexed on any of
them, search results will be inconsistent depending on which set of nodes
returns it. Let's say you have replicas on nodes A, B, and C. Entry X
failed to be indexed on A, entry Y failed to be indexed on A and B, and
entry Z was indexed OK on all nodes. Then you may get {X,Y,Z}, {X,Z}, or
{Z} as your search results.
In our case the indexing failures were caused by disk/filesystem errors.
Regards,
Vitaly
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
bucket-types: renaming and deleting
Hi, I am evaluating Riak for my employer and need to rename a bucket-type. I think that it would also be useful to be able to delete bucket-types. In order to delete a bucket-type, do I need to stop Riak and delete the data for the bucket-type? If so, what exactly would I delete (under /var/lib/riak)? Or is there another way to remove a bucket-type? Thanks, Jonathan ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Packagecloud.io issue
Alex- We’ve managed to reproduce that issue and are researching it now. It does appear to be something in the package itself; not sure how that didn’t turn up until now. We’ll keep you updated. Thanks! —Greg On Mar 30, 2015, at 9:00 AM, riak-users-requ...@lists.basho.com wrote: Date: Mon, 30 Mar 2015 11:55:41 +0200 From: Alex De la rosa alex.rosa@gmail.com To: riak-users riak-users@lists.basho.com Subject: Packagecloud.io issue Message-ID: CAPphDGoinadFe+qBSaLPzf5z_8XJx1kAfXcGfc=o4t1aea6...@mail.gmail.com Content-Type: text/plain; charset=utf-8 Hi there, I want to report again a problem I have with packagecloud.io; everytime I do aptitude update I get a hit as if a new version of Riak is available (that is not)... and I have to re-install Riak to a version I already have. I just did an aptitude safe-upgrade and upgraded Riak to version 2.0.5 (latest): # aptitude safe-upgrade The following packages will be upgraded: ... ... ... riak ... ... ... ... ... ... [...] Unpacking riak (2.0.5-1) over (2.0.1) ... [...] # riak version 2.0.5 # aptitude update Hit https://packagecloud.io trusty InRelease Hit https://packagecloud.io trusty/main Sources Hit https://packagecloud.io trusty/main amd64 Packages Hit https://packagecloud.io trusty/main i386 Packages # aptitude safe-upgrade The following packages will be upgraded: riak 1 packages upgraded, 0 newly installed, 0 to remove and 0 not upgraded. Need to get 0 B/55.4 MB of archives. After unpacking 0 B will be used. Do you want to continue? [Y/n/?] if i select Y i get the following: Unpacking riak (2.0.5-1) over (2.0.5-1) ... and everytime I check for updates, riak comes as requiring an update although is not true. How to fix that? makes no sense it continues coming as needing to be upgraded when is the same exact version. Thanks, Alex -- next part -- An HTML attachment was scrubbed... URL: http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20150330/b2e2620c/attachment-0001.html ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Packagecloud.io issue
Good to know it will finally get fixed : ) [not sure how that didn’t turn up until now] --- I actually reported it several times in the past: August 2014: http://lists.basho.com/pipermail/riak-users_lists.basho.com/2014-August/015702.html September 2014: http://lists.basho.com/pipermail/riak-users_lists.basho.com/2014-September/015884.html and now again in March 2015 : P Thanks, Alex On Wed, Apr 1, 2015 at 8:38 PM, Greg Cymbalski gcymbal...@basho.com wrote: Alex- We’ve managed to reproduce that issue and are researching it now. It does appear to be something in the package itself; not sure how that didn’t turn up until now. We’ll keep you updated. Thanks! —Greg On Mar 30, 2015, at 9:00 AM, riak-users-requ...@lists.basho.com wrote: Date: Mon, 30 Mar 2015 11:55:41 +0200 From: Alex De la rosa alex.rosa@gmail.com To: riak-users riak-users@lists.basho.com Subject: Packagecloud.io issue Message-ID: CAPphDGoinadFe+qBSaLPzf5z_8XJx1kAfXcGfc=o4t1aea6...@mail.gmail.com Content-Type: text/plain; charset=utf-8 Hi there, I want to report again a problem I have with packagecloud.io; everytime I do aptitude update I get a hit as if a new version of Riak is available (that is not)... and I have to re-install Riak to a version I already have. I just did an aptitude safe-upgrade and upgraded Riak to version 2.0.5 (latest): # aptitude safe-upgrade The following packages will be upgraded: ... ... ... riak ... ... ... ... ... ... [...] Unpacking riak (2.0.5-1) over (2.0.1) ... [...] # riak version 2.0.5 # aptitude update Hit https://packagecloud.io trusty InRelease Hit https://packagecloud.io trusty/main Sources Hit https://packagecloud.io trusty/main amd64 Packages Hit https://packagecloud.io trusty/main i386 Packages # aptitude safe-upgrade The following packages will be upgraded: riak 1 packages upgraded, 0 newly installed, 0 to remove and 0 not upgraded. Need to get 0 B/55.4 MB of archives. After unpacking 0 B will be used. Do you want to continue? [Y/n/?] if i select Y i get the following: Unpacking riak (2.0.5-1) over (2.0.5-1) ... and everytime I check for updates, riak comes as requiring an update although is not true. How to fix that? makes no sense it continues coming as needing to be upgraded when is the same exact version. Thanks, Alex -- next part -- An HTML attachment was scrubbed... URL: http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20150330/b2e2620c/attachment-0001.html ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
[Announcement] Riak CS 2.0.0 Release
We shipped Riak CS 2.0.0 last night. A huge thanks goes out to all who made it happen! Riak CS 2.0.0 includes many improvements you can review here [3]. To list a few: * Integration with cuttlefish, which means new conveniences like riak.conf for configuration * Enhanced conflict resolution (thanks to dotted version vectors) that resolves situations like sibling explosion * The ability to tier storage for data files using LevelDB Want to upgrade? Upgrading a Riak CS system involves upgrading the underlying Riak, Riak CS and Stanchion installations - there is much more on that here [1]. You can download the latest code here [2]. The full release notes are available here [3] and will continue to be updated. Enjoy! [1] http://docs.basho.com/riakcs/latest/cookbooks/Riak-CS-Release-Notes/#Riak-CS-2-0-0 [2] http://docs.basho.com/riakcs/latest/riakcs-downloads/ [3] https://github.com/basho/riak_cs/blob/develop/RELEASE-NOTES.md Matt Brender Developer Advocacy Lead @mjbrender ᐧ ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
