Re: Pending handoff when node offline
Magnus Thanks for confirming. We've had issues with 2i (coverage queries) during node startup where some keys potentially might not appear in results. More details on the here: http://riak-users.197444.n3.nabble.com/Keys-not-listed-during-vnode-transfers-td4027133.html#a4027139 We've been using wait-for-service in our script but that was not sufficient to workaround issue with 2i. I will try to workaround it or check if newer version of Riak solved to 2i problem. Thanks -- View this message in context: http://riak-users.197444.n3.nabble.com/Pending-handoff-when-node-offline-tp4033844p4033846.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Pending handoff when node offline
Hi all Am I right thinking that when node goes offline *riak-admin transfers* will always show transfers to be done? E.g. riak-admin transfers Attempting to restart script through sudo -H -u riak [sudo] password for myuser: Nodes ['riak@10.173.240.12'] are currently down. 'riak@10.173.240.9' waiting to handoff 18 partitions 'riak@10.173.240.11' waiting to handoff 13 partitions 'riak@10.173.240.10' waiting to handoff 13 partitions Active Transfers: Node 'riak@10.173.240.12' could not be contacted Even when I mark node as down with *riak-admin down* transfers are still waiting. Is that normal? The reason I ask is because our services before they start are checking if all transfers are complete (normal process during riak startup). This is because in the past we've had issues with using 2i queries when Riak. Unfortunately this means that after e.g. reboot our services won't start until timeout expires or missing node comes back and handoff finishes. Maybe there is a better way to check if Riak cluster is ready for 2i queries? We are still on Riak 1.3.1 Regards Daniel -- View this message in context: http://riak-users.197444.n3.nabble.com/Pending-handoff-when-node-offline-tp4033844.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Node join not committed
Hi Jon Thanks for confirming this. We did do plan/commit and everything worked as expected, no issues whatsoever Thanks a bunch Daniel -- View this message in context: http://riak-users.197444.n3.nabble.com/Node-join-not-committed-tp4033571p4033573.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Node join not committed
Hi Our attach script failed and it only issued *cluster join* but not *cluster plan* and *cluster commit*. so the node was visible as joining in member_status Since then both part (original cluster and new node) were taking writes but in our configuration each node takes writes from it local process (no load balancing etc) I've checked content of both parts and it looks like riak db is in sync i.e. new node can see data that were stored on original cluster. Am I right assuming join is enough and cluster is seen as a whole? What are the consequences on not running plan & commit? What would cancel do in this scenario? Also is it safe to issue those to and finish process? Regards Daniel -- View this message in context: http://riak-users.197444.n3.nabble.com/Node-join-not-committed-tp4033571.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Clarifying withoutFetch() with LevelDB and
Alex, Thanks for answering this one and pointing me into right direction. I did an experiment and wrote 0 bytes instead of a JSON and got the same effect - level db folder is 80-220MB in size and activity around 20MB/s write to disk, no read from disk. Java client reports speed 45 secs for 1000 entries, so avg. 45ms per entry Then I change the code so it writes to unique keys. Astonishing difference. Very little write activity to disk ~ 600kB/s per node and db is only 6MB big! Java client reports 2.5 secs for 1000 entries! The difference is huge both in speed and storage! Now, I was always under the impression that writing with stale clock using withoutFetch() would be the quickest way to put data into Riak. Looks like I was wrong. Would the all overhead be basically vclocks? I did not know that even if I'm using withoutFetch() data is still read in the background (?) Regarding data model. I'm trying to solve particular problem. I'm modeling timeline in Riak and I wanted to group events into batches of 1 hour windows. So basically timeboxing. Data has to go to disk so there's no option for me to delay write. Once 1000 events per key is reached next key is selected. Keys are predictable so I can calculate them when read operation happens. I want to grab as much events in one read operation as possible hence the idea of writing in a controlled way to the same key with stale clock. Is there any better way to model that? Obviously next thing I will try is to resolve sibling during write but I hoped I can avoid/delay that until read happens. This vclock/storage/bandwidth explosion really surprised me. Regards Daniel -- View this message in context: http://riak-users.197444.n3.nabble.com/Clarifying-withoutFetch-with-LevelDB-and-tp4033051p4033057.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Java client 1.1.4 and headOnly() in domain buckets
Hi Alex >> It appears that the domain buckets api does not support headOnly(). That >> api was written to be a higher-level abstraction around a common usage, >> so >> it abstracted that idea of head vs object data away. I think it may be quite useful functionality anyway, to check the existence of a key. This is what I'm using it for to avoid loading big objects from leveldb. Can you confirm if that operation does not read data from the backend or is there any other better way to check existence of the key? >> If you need that support, I would use the regular bucket methods instead. That's what I ended up doing although it's not convenient to have two bucket instances configured differently to access data in one way or another i.e. writing through DomainBucket, checking keys through regular bucket. D. -- View this message in context: http://riak-users.197444.n3.nabble.com/Java-client-1-1-4-and-headOnly-in-domain-buckets-tp4033042p4033056.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Clarifying withoutFetch() with LevelDB and
We are using Java client 1.1.4. We haven't moved to newer version of Riak as as for the moment we don't need any new features. Also roll out of the new version may be complicated since we have multiple clusters. As with regards to object size its ~250-300 bytes per write. We store simple JSON structures. Is there anything in new versions that would limit size of data going to the disk? And more importantly is there a way of determining why levelDB grows so big? I'm using ring size 128 which is probably too high at the moment, but after switching to 64 not much has changed. I also disabled 2i indexes that I thought may matter (four 16 bytes fields) and that did not made any difference, still 25-38MB/s write to level db per node. D. -- View this message in context: http://riak-users.197444.n3.nabble.com/Clarifying-withoutFetch-with-LevelDB-and-tp4033051p4033053.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Clarifying withoutFetch() with LevelDB and
Hi I'm using 4 node Riak cluster v1.3.1 I wanted to know a little bit more about using withoutFetch() option when used with levelDB. I'm trying to write to a single key as fast as I can with n=3. I deliberately create siblings by writing with stale vclock. I'm limiting number of writes to 1000 per key to keep size of Riak object under control and then I switch to another key. Siblings will probably never be resolved (or resolved in realtime during sporadic reads) Single write operation is about 250 bytes, rate 10-80 events per sec which gives 3-20kB per second per node. So roughly 100kB / s for the cluster. During the test I see activity on the on disk via iostat and it's between 20-30 MB/s on each node. Even taking into account multiple copies and overhead of Riak (vclocks etc) this seems to be pretty high rate. I don't see any read activity which suggest withoutFetch() works as expected. After 2 mins of tests leveldb on each node is 250MB is size, before test (11MB) Am I using it incorrectly? Is writing in this way to a single key a good idea or will I be bitten by something? How to explain high number of MB written to disks? Regards Daniel -- View this message in context: http://riak-users.197444.n3.nabble.com/Clarifying-withoutFetch-with-LevelDB-and-tp4033051.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Java client 1.1.4 and headOnly() in domain buckets
We are using official 1.1.4 which is the latest recommended with Riak 1.3 we have installed. Upgrade to Riak 1.4 is not possible at the moment. D. -- View this message in context: http://riak-users.197444.n3.nabble.com/Java-client-1-1-4-and-headOnly-in-domain-buckets-tp4033042p4033048.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Java client 1.1.4 and headOnly() in domain buckets
Hi all
Am I right thinking that v1.1.4 does not support headOnly() on domain
buckets?
During domain.fetch() line 237 in
https://github.com/basho/riak-java-client/blob/1.1.4/src/main/java/com/basho/riak/client/bucket/DomainBucket.java
there is no check/call headOnly() on FetchMeta object.
Code snippet
public T fetch(String key) throws RiakException {
final FetchObject fo = bucket.fetch(key, clazz)
.withConverter(converter)
.withResolver(resolver)
.withRetrier(retrier);
if (fetchMeta.hasR()) {
fo.r(fetchMeta.getR());
}
if (fetchMeta.hasPr()) {
fo.pr(fetchMeta.getPr());
}
if (fetchMeta.hasBasicQuorum()) {
fo.basicQuorum(fetchMeta.getBasicQuorum());
}
if (fetchMeta.hasNotFoundOk()) {
fo.notFoundOK(fetchMeta.getNotFoundOK());
}
if (fetchMeta.hasReturnDeletedVClock()) {
fo.returnDeletedVClock(fetchMeta.getReturnDeletedVClock());
}
return fo.execute();
}
--
View this message in context:
http://riak-users.197444.n3.nabble.com/Java-client-1-1-4-and-headOnly-in-domain-buckets-tp4033042.html
Sent from the Riak Users mailing list archive at Nabble.com.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak 1.3.1 crashing with segfault
Thanks Magnus I'm using memtest86 on set of another 3 servers with identical configuration to see if I can trigger that as well. I cannot do it on the failing node at the moment since it's a remote site but I agree it's a strong indication of RAM module problem. As a test I moved var/lib/riak to RAID6 array on that node and Riak crashed after few hours with segfault. Also I had a random Segmentation fault when launched "top". Thanks -- View this message in context: http://riak-users.197444.n3.nabble.com/Riak-1-3-1-crashing-with-segfault-tp4032638p4032702.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak 1.3.1 crashing with segfault
I moved /var/lib/riak folder to RAID array Another crash happened 20 minutes after Riak start. Another crash 20 mins after start = = LOGGING STARTED Wed Feb 25 12:40:07 UTC 2015 = Exec: /usr/lib/riak/erts-5.9.1/bin/erlexec -boot /usr/lib/riak/releases/1.3.1/riak -embedded -config /etc/riak/app.config -pa /usr/lib/riak/lib/basho-patches -args_file /etc/riak/vm.args -- console Root: /usr/lib/riak Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:4:4] [async-threads:64] [kernel-poll:true] Eshell V5.9.1 (abort with ^G) (riak@10.173.240.2)1> = ALIVE Wed Feb 25 12:55:09 UTC 2015 = Wed Feb 25 13:02:48 UTC 2015 Erlang has closed /usr/lib/riak/lib/os_mon-2.2.9/priv/bin/memsup: Erlang has closed. in kern.log Feb 25 13:02:47 node2 kernel: [710669.067800] beam.smp[24170]: segfault at 8523111 ip 08523111 sp 7ff100e2fbe8 error 14 Is there a way to diagnose that? Is there a core dump somewhere? Any help appreciated Daniel -- View this message in context: http://riak-users.197444.n3.nabble.com/Riak-1-3-1-crashing-with-segfault-tp4032638p4032700.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak 1.3.1 crashing with segfault
Hi I've checked all logs and there is nothing regarding memory issues. Since then I've had several Riak crashes but looks like other processes are failing as well Feb 2 22:05:28 node2 kernel: [20052.901884] beam.smp[1830]: segfault at 8523111 ip 08523111 sp 7f03ba821be8 error 14 in 13.log[7f01b689+140] Feb 4 10:16:01 node2 kernel: [150082.149742] MegaCli[29635]: segfault at 8549b14 ip 08549b14 sp 7fff3c5f7840 error 14 in libtinfo.so.5.9[7f3ee87ba000+22000] Feb 4 16:35:27 node2 kernel: [172812.410113] MegaCli[23019]: segfault at 85473d6 ip 085473d6 sp 7fffc8f49f20 error 14 in libtinfo.so.5.9[7fa9e90c3000+22000] Feb 6 14:50:38 node2 kernel: [339062.483587] sh[24637]: segfault at 840460d ip 0840460d sp 7fff35251160 error 14 in libc-2.15.so[7fd4cc395000+1b5000] Feb 7 11:59:32 node2 kernel: [415077.342034] df[6393]: segfault at 84029a0 ip 084029a0 sp 7fff758406d0 error 14 in libc-2.15.so[7f5433243000+1b5000] Feb 8 10:04:31 node2 kernel: [494451.877635] df[22107]: segfault at 8404b00 ip 08404b00 sp 7eee1b08 error 14 in libc-2.15.so[7fbd82ce2000+1b5000] Feb 9 16:30:20 node2 kernel: [603829.476142] ls[2873]: segfault at 840d04e ip 0840d04e sp 7fffaff38c60 error 14 in libnss_files-2.15.so[7f257c9c4000+c000] Feb 9 18:26:13 node2 kernel: [ 6503.710549] beam.smp[2140]: segfault at 8523a00 ip 08523a00 sp 7f3955ff2d80 error 14 in 06.log[7f377f27b000+140] Feb 10 17:34:46 node2 kernel: [36949.199740] beam.smp[1877]: segfault at 85650b2 ip 085650b2 sp 7faba120fa70 error 14 in 09.log[7fa99827c000+140] Feb 11 20:37:15 node2 kernel: [134145.969112] beam.smp[7276]: segfault at 852287e ip 0852287e sp 7ff8625c9be0 error 14 in 12.log[7ff66f703000+140] Feb 13 08:58:57 node2 kernel: [ 6414.659327] beam.smp[1877]: segfault at 8569cfc ip 08569cfc sp 7f55aa48bab0 error 14 in 12.log[7f537dc0e000+140] Feb 15 03:20:30 node2 kernel: [133707.360153] MegaCli[7442]: segfault at 85473d6 ip 085473d6 sp 7fff39ba1570 error 14 in libtinfo.so.5.9[7f2728f79000+22000] Feb 15 10:02:23 node2 kernel: [157782.787481] beam.smp[2023]: segfault at 85239d0 ip 085239d0 sp 7f47e3fe6d68 error 14 in 61.log[7f463e32f000+140] Feb 16 17:30:18 node2 kernel: [270880.717532] console-kit-dae[1548]: segfault at84123e8 ip 084123e8 sp 7fffc6d8c0c0 error 14 Feb 16 19:31:45 node2 kernel: [278156.348900] beam.smp[16617]: segfault at 85650b2 ip 085650b2 sp 7f3dbe74ba70 error 14 in 19.log[7f3b89f65000+140] Feb 16 21:45:34 node2 kernel: [286172.695110] sh[12432]: segfault at 840460d ip 0840460d sp 7fffe6e2b3b0 error 14 in libc-2.15.so[7f9b57c77000+1b5000] Feb 17 07:27:23 node2 kernel: [ 457.418215] beam.smp[1824]: segfault at 8523111 ip 08523111 sp 7f36e9574be8 error 14 in 21.log[7f34c24f3000+140] Feb 25 10:46:04 node2 kernel: [702478.037041] beam.smp[8832]: segfault at 8522980 ip 08522980 sp 7fe8bbffede8 error 14 in 06.log[7fe713e2c000+140] Riak is always touching anti-entropy files, like in the last example: /var/lib/riak/anti_entropy/22835963083295358096932575511191922182123945984/06.log Could it be an SSD failing? Daniel -- View this message in context: http://riak-users.197444.n3.nabble.com/Riak-1-3-1-crashing-with-segfault-tp4032638p4032699.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak 1.3.1 crashing with segfault
Ciprian Thanks for reply. I will check that as soon as I will get access to the servers again D. -- View this message in context: http://riak-users.197444.n3.nabble.com/Riak-1-3-1-crashing-with-segfault-tp4032638p4032673.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak 1.3.1 high swap usage
I absolutely agree. That is why we've change the setting vm.swappiness to 1 so it swaps only when absolutely necessary. I think we underestimated how much swap may be needed, but I also don't understand why so much hungry on memory. Is there a particular activity, like 2i queries, AAE or levelDB compaction that would require significant amount of memory? What's the best way to go forward? Thanks Daniel -- View this message in context: http://riak-users.197444.n3.nabble.com/Riak-1-3-1-high-swap-usage-tp4032649p4032664.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak 1.3.1 high swap usage
We are using levelDB as backend without any tuning. Also we are aware that performance may suffer due to potentially storing some of the copies (n=3) twice on the server. We are not so much concerned about latencies caused by that. What is worrying though is almost unbounded growth of swap used, which essentially filled up our system disk. Probably our Riak instances require some tuning but I would like to know first why we need to tune it e.g. AAE consumes large swap. Regarding failure scenarios. We've seen OS' oom-killer nuking beam.smp and also our process due to the failure of memory allocation. So far we haven't experienced situation where killing one node would cause cascading failures of the nodes. We were slightly reluctant to upgrade to 1.4 due to some AAE issues reported on the mailing list. I was under impression 1.4 may consume even more memory with AAE but I may be totally wrong here. Daniel -- View this message in context: http://riak-users.197444.n3.nabble.com/Riak-1-3-1-high-swap-usage-tp4032649p4032662.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Secondary index in riak
My ideas: 1. Rewrite (read-write) object with new values for all indexes 2. Enable siblings on a bucket, write empty object with update for your index, that will create sibling. Then whenever you read object do merge of object+indexes. This may be more appropriate if you have big objects and want to "save" bandwidth Option 1 is definitely simpler though. -- View this message in context: http://riak-users.197444.n3.nabble.com/Secondary-index-in-riak-tp4032650p4032652.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak 1.3.1 high swap usage
Hi On 3 node cluster Ubuntu 12.04, nodes 8GB RAM all nodes show 6GB taken beam.smp, 2GB by our process. beam started swapping and currently is using 23GB of swap space. vm.swappiness is set to 1 We are using ring 128. /var/lib/riak is 37GB in size 11GB of which is used by anti-entropy Is there a way to find out why there is so much memory required by Riak process? Thanks Daniel -- View this message in context: http://riak-users.197444.n3.nabble.com/Riak-1-3-1-high-swap-usage-tp4032649.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak 1.3.1 crashing with segfault
We are experiencing crash of beam.smp on one of nodes in 3-node cluster (ring
128)
Distro is Ubuntu 12.04 with 16GB of memory (almost exclusive for Riak)
= Sun Feb 15 10:02:23 UTC 2015
Erlang has closed/usr/lib/riak/lib/os_mon-2.2.9/priv/bin/memsup:
Erlang has closed.
Hi I've got following error in syslog
Feb 15 10:02:23 node2 kernel: [157782.787481] beam.smp[2023]: segfault at
85239d0 ip 085239d0 sp 7f47e3fe6d68 error 14 in
61.log[7f463e32f000+140]
I'm not sure which causes the other.
61.log was from leveldb folder but it ahs been deleted after restart of
Riak I believe
Last thing I could find in console log is AAE activity
2015-02-14 15:24:18.329 [info]
<0.7258.26>@riak_kv_exchange_fsm:key_exchange:204 Repaired 3 keys during
active anti-entropy exchange of
{25119559391624893906625833062344003363405824,3} between
{25119559391624893906625833062344003363405824,'riak@10.173.240.2'} and
{262613575457896618114724618378707105094425378816,'riak@10.173.240.3'}
2015-02-14 15:31:03.594 [info]
<0.10920.26>@riak_kv_exchange_fsm:key_exchange:204 Repaired 58 keys during
active anti-entropy exchange of
{376793390874373408599387495934666716005045108736,3} between
{376793390874373408599387495934666716005045108736,'riak@10.173.240.2'} and
{388211372416021087647853783690262677096107081728,'riak@10.173.240.2'}
2015-02-14 15:33:48.637 [info]
<0.12367.26>@riak_kv_exchange_fsm:key_exchange:204 Repaired 37 keys during
active anti-entropy exchange of
{422465317040964124793252646957050560369293000704,3} between
{422465317040964124793252646957050560369293000704,'riak@10.173.240.2'} and
{445301280124259482890185222468242482551416946688,'riak@10.173.240.3'}
2015-02-14 15:34:03.454 [info]
<0.12546.26>@riak_kv_exchange_fsm:key_exchange:204 Repaired 37 keys during
active anti-entropy exchange of
{422465317040964124793252646957050560369293000704,3} between
{433883298582611803841718934712646521460354973696,'riak@10.173.240.2'} and
{445301280124259482890185222468242482551416946688,'riak@10.173.240.3'}
2015-02-14 15:55:18.518 [info]
<0.23498.26>@riak_kv_exchange_fsm:key_exchange:204 Repaired 1 keys during
active anti-entropy exchange of
{1061872283373234151507364761270424381468763488256,3} between
{1061872283373234151507364761270424381468763488256,'riak@10.173.240.2'} and
{1073290264914881830555831049026020342559825461248,'riak@10.173.240.3'}
2015-02-14 15:59:33.522 [info]
<0.25935.26>@riak_kv_exchange_fsm:key_exchange:204 Repaired 1 keys during
active anti-entropy exchange of
{1187470080331358621040493926581979953470445191168,3} between
{119061873006300088960214337575914561507164160,'riak@10.173.240.2'} and
{1210306043414653979137426502093171875652569137152,'riak@10.173.240.3'}
2015-02-14 15:59:48.513 [info]
<0.26044.26>@riak_kv_exchange_fsm:key_exchange:204 Repaired 1 keys during
active anti-entropy exchange of
{119061873006300088960214337575914561507164160,3} between
{119061873006300088960214337575914561507164160,'riak@10.173.240.2'} and
{1210306043414653979137426502093171875652569137152,'riak@10.173.240.3'}
2015-02-14 20:08:49.674 [info]
<0.29386.30>@riak_kv_exchange_fsm:key_exchange:204 Repaired 5 keys during
active anti-entropy exchange of
{148433760041419827630061740822747494183805648896,3} between
{148433760041419827630061740822747494183805648896,'riak@10.173.240.2'} and
{171269723124715185726994316333939416365929594880,'riak@10.173.240.3'}
2015-02-14 20:09:04.516 [info]
<0.29501.30>@riak_kv_exchange_fsm:key_exchange:204 Repaired 5 keys during
active anti-entropy exchange of
{148433760041419827630061740822747494183805648896,3} between
{159851741583067506678528028578343455274867621888,'riak@10.173.240.2'} and
{171269723124715185726994316333939416365929594880,'riak@10.173.240.3'}
Is it possible that AAE is causing the problems here.
Regards
Daniel
--
View this message in context:
http://riak-users.197444.n3.nabble.com/Riak-1-3-1-crashing-with-segfault-tp4032638.html
Sent from the Riak Users mailing list archive at Nabble.com.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak Corruption
Also it may be worth checking if there is any 0-byte file in AAE folder. I've seen corruptions like that in the past (although not on AAE but on ring files). If you find and remove corrupted file, rebuilding AAE will be faster/cheaper. It would be good if that error showed which file could not be read. Daniel -- View this message in context: http://riak-users.197444.n3.nabble.com/Riak-Corruption-tp4032599p4032612.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak client returnTerm and regexp
By the look of it it seems returnTerm is available in 1.3+ and regexp matching got merged into 2.0? Also is there any documentation what subset of Perl regexp is supported? Thanks Daniel -- View this message in context: http://riak-users.197444.n3.nabble.com/Riak-client-returnTerm-and-regexp-tp4032539p4032556.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak client returnTerm and regexp
I watched Ricon2014 video from Martin @NHS. Before the end of his talk he briefly mentions returnTerm option and also something about regular expression matching (2i ?) https://www.youtube.com/watch?v=5Plsj6Zl-kM http://basho.github.io/riak-java-client/1.4.4/com/basho/riak/pbc/IndexRequest.html#returnTerms() Does anyone know where can I find more information about those two? Regards Daniel -- View this message in context: http://riak-users.197444.n3.nabble.com/Riak-client-returnTerm-and-regexp-tp4032539.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re:
That's what I was thinking after looking into details of https://github.com/basho/riak-java-client/pull/351 When I saw release of 1.1.4 a comment in changelog did not seem like of a huge importance This is a minor bugfix release Fix RiakConnection.get_code(int) We will upgrade client library. BTW. This bug and its description: "This leads to a connection being returned to the pool with data still in the input stream, and a subsequent operation will fail due to reading it (potentially causing a huge memory allocation and possible OOM error)." brings to my mind recent infamous Heartbleed bug where tiny problem can have unforeseen results. Butterfly effect I suppose... Cheers D. On 8 May 2014 17:55, Brian Roach wrote: > Hi Daniel, > > I suspect this is the bug fixed in PR#352 ( > https://github.com/basho/riak-java-client/pull/352 ) > > Try upgrading to the 1.1.4 client release and see if the problem persists. > > Thanks, > - Roach > > On Thu, May 8, 2014 at 8:02 AM, Daniel Iwan wrote: > > Hi > > > > I got following exception with riak Java client 1.1.3, Riak cluster 1.3.1 > > I don't see any error messages in Riak's console log. Any idea what may > be > > causing this? > > > > Caused by: com.basho.riak.client.RiakRetryFailedException: > > java.io.IOException: bad message code. Expected: 14 actual: 1 > > at > com.basho.riak.client.cap.DefaultRetrier.attempt(DefaultRetrier.java:79) > > at > com.basho.riak.client.cap.DefaultRetrier.attempt(DefaultRetrier.java:81) > > at > com.basho.riak.client.cap.DefaultRetrier.attempt(DefaultRetrier.java:81) > > at > com.basho.riak.client.cap.DefaultRetrier.attempt(DefaultRetrier.java:81) > > at > com.basho.riak.client.cap.DefaultRetrier.attempt(DefaultRetrier.java:53) > > at > > > com.basho.riak.client.operations.DeleteObject.execute(DeleteObject.java:111) > > at > com.basho.riak.client.bucket.DomainBucket.delete(DomainBucket.java:484) > > at > com.basho.riak.client.bucket.DomainBucket.delete(DomainBucket.java:418) > > at server.riak.k.b(SourceFile:104) > > ... 19 more > > Caused by: java.io.IOException: bad message code. Expected: 14 actual: 1 > > at > com.basho.riak.pbc.RiakConnection.receive_code(RiakConnection.java:153) > > at com.basho.riak.pbc.RiakClient.delete(RiakClient.java:622) > > at com.basho.riak.pbc.RiakClient.delete(RiakClient.java:609) > > at > > > com.basho.riak.client.raw.pbc.PBClientAdapter.delete(PBClientAdapter.java:222) > > at > > > com.basho.riak.client.operations.DeleteObject$2.call(DeleteObject.java:106) > > at > > > com.basho.riak.client.operations.DeleteObject$2.call(DeleteObject.java:104) > > at > com.basho.riak.client.cap.DefaultRetrier.attempt(DefaultRetrier.java:72) > > ... 27 more > > > > Regards > > Daniel > > > > ___ > > riak-users mailing list > > riak-users@lists.basho.com > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: bad message code. Expected: 14 actual: 1
Thanks Bryan but according to Java github project : riak-client-2.0-technical-preview - Core preview for Riak 2.0 riak-client-1.4.4 - For use with Riak 1.4+ riak-client-1.1.4 - For use with < Riak 1.4.0 Is v1.1.4 https://github.com/basho/riak-java-client/commit/875bfbe66ac9c24a4ddc1b5cca819b8f2aee8381 fixing exactly that problem? Daniel On 8 May 2014 17:30, bryan hunt wrote: > > Daniel > > http://docs.basho.com/riak/latest/dev/references/protocol-buffers/ > > Sounds it’s trying to process the response for a ping instead of the > response for a delete. > > I’d upgrade to Java client 1.1.4 and see if the behavior continues. > > Best Regards, > > Bryan Hunt > > > > On 8 May 2014, at 15:02, Daniel Iwan wrote: > > > Hi > > > > I got following exception with riak Java client 1.1.3, Riak cluster 1.3.1 > > I don't see any error messages in Riak's console log. Any idea what may > be causing this? > > > > Caused by: com.basho.riak.client.RiakRetryFailedException: > java.io.IOException: bad message code. Expected: 14 actual: 1 > > at > com.basho.riak.client.cap.DefaultRetrier.attempt(DefaultRetrier.java:79) > > at > com.basho.riak.client.cap.DefaultRetrier.attempt(DefaultRetrier.java:81) > > at > com.basho.riak.client.cap.DefaultRetrier.attempt(DefaultRetrier.java:81) > > at > com.basho.riak.client.cap.DefaultRetrier.attempt(DefaultRetrier.java:81) > > at > com.basho.riak.client.cap.DefaultRetrier.attempt(DefaultRetrier.java:53) > > at > com.basho.riak.client.operations.DeleteObject.execute(DeleteObject.java:111) > > at > com.basho.riak.client.bucket.DomainBucket.delete(DomainBucket.java:484) > > at > com.basho.riak.client.bucket.DomainBucket.delete(DomainBucket.java:418) > > at server.riak.k.b(SourceFile:104) > > ... 19 more > > Caused by: java.io.IOException: bad message code. Expected: 14 actual: 1 > > at > com.basho.riak.pbc.RiakConnection.receive_code(RiakConnection.java:153) > > at com.basho.riak.pbc.RiakClient.delete(RiakClient.java:622) > > at com.basho.riak.pbc.RiakClient.delete(RiakClient.java:609) > > at > com.basho.riak.client.raw.pbc.PBClientAdapter.delete(PBClientAdapter.java:222) > > at > com.basho.riak.client.operations.DeleteObject$2.call(DeleteObject.java:106) > > at > com.basho.riak.client.operations.DeleteObject$2.call(DeleteObject.java:104) > > at > com.basho.riak.client.cap.DefaultRetrier.attempt(DefaultRetrier.java:72) > > ... 27 more > > > > Regards > > Daniel > > ___ > > riak-users mailing list > > riak-users@lists.basho.com > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
[no subject]
Hi I got following exception with riak Java client 1.1.3, Riak cluster 1.3.1 I don't see any error messages in Riak's console log. Any idea what may be causing this? Caused by: com.basho.riak.client.RiakRetryFailedException: java.io.IOException: bad message code. Expected: 14 actual: 1 at com.basho.riak.client.cap.DefaultRetrier.attempt(DefaultRetrier.java:79) at com.basho.riak.client.cap.DefaultRetrier.attempt(DefaultRetrier.java:81) at com.basho.riak.client.cap.DefaultRetrier.attempt(DefaultRetrier.java:81) at com.basho.riak.client.cap.DefaultRetrier.attempt(DefaultRetrier.java:81) at com.basho.riak.client.cap.DefaultRetrier.attempt(DefaultRetrier.java:53) at com.basho.riak.client.operations.DeleteObject.execute(DeleteObject.java:111) at com.basho.riak.client.bucket.DomainBucket.delete(DomainBucket.java:484) at com.basho.riak.client.bucket.DomainBucket.delete(DomainBucket.java:418) at server.riak.k.b(SourceFile:104) ... 19 more Caused by: java.io.IOException: bad message code. Expected: 14 actual: 1 at com.basho.riak.pbc.RiakConnection.receive_code(RiakConnection.java:153) at com.basho.riak.pbc.RiakClient.delete(RiakClient.java:622) at com.basho.riak.pbc.RiakClient.delete(RiakClient.java:609) at com.basho.riak.client.raw.pbc.PBClientAdapter.delete(PBClientAdapter.java:222) at com.basho.riak.client.operations.DeleteObject$2.call(DeleteObject.java:106) at com.basho.riak.client.operations.DeleteObject$2.call(DeleteObject.java:104) at com.basho.riak.client.cap.DefaultRetrier.attempt(DefaultRetrier.java:72) ... 27 more Regards Daniel ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Partitions placement
Hi Ciprian Thanks for reply I'm assuming 'overlay' you are talking about are vnodes? When creating cluster and joining 2 nodes to first node (3-node cluster) there should be possible distributing partitions to guarantee 3 copies are on distinct machines. Simple sequential vnode assignment would do. Then I guess it's a matter of calculating distances between indexes within each of nodes. Partitions that do not meet that criteria should be moved when scaling up. Scaling up in my opinion should be easier as more nodes in the cluster, sequential partitions are more spread horizontally so probability that a server will hold sequential partitions decreases. With 3 servers it should be guaranteed that at least 2 copies are on distinct servers. On 5 servers should be guaranteed 3 copies are on distinct servers. etc. Does Riak give such guarantees? Can target_n_val be changed later. What are the implications? Is there a description what algorithm will be used for partition placement? Cheers Daniel -- View this message in context: http://riak-users.197444.n3.nabble.com/Partitions-placement-tp4030664p4030679.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Partitions placement
Below is an output of my Riak cluster. 3 physical nodes. Ring size 128.
As far as I can tell when Riak installed fresh it is always place partitions
in the same way on a ring as long as number of vnodes and servers is the
same.
All presentations including "A Little Riak Book' show pretty picture of ring
and nodes claiming partitions in a sequential fashion. That's clearly not a
case.
Output below shows that node2 is picked as favourite, which means replicas
of certain keys will definitely be on the same hardware. Partitions are
split 44 + 42 + 42. Why not 43+43+42?
Another thing, why the algorithm selects nodes in 'random' non-sequential
fashion? When the cluster gets created and nodes 2 & 3 are joined to node 1,
it's a clear situation. Partitions are empty so vnodes could be assigned in
a way so there's no consecutive partitions on the same hw.
My issue is that in my case if node2 goes down and I'm storing some data
with N=2 I will definitely not be able access certain keys and more
surprisingly all 2i will no longer work for the buckets with N=2 due to
{error,insufficient_vnodes_available}. That is all 2i's for those buckets.
I understand that when new nodes are attached Riak tries to avoid
reshuffling everything and just moves certain partitions, and at that point
you may end up with copies on the same physical nodes. But even then Riak
should make best effort and try not to put consecutive partitions on the
same server. If it has to move it anyway it could as well put it on any
other machine but the one that holds partition with preceding and following
index.
I also understand Riak does not guarantee that replicas are on distinct
servers (why? it should, at least for N=2 and N=3 if possible)
I appreciate minimum recommended setup is 5 nodes and I should be storing
with N=3 minimum.
But I just find it confusing when presentations show something that is not
even remotely close to reality.
Just to be clear I have nothing against Riak, I think it's great though bit
disappointing that there are no stronger conditions about replica placement
here.
I'm probably missing something and simplifying too much. Any clarification
appreciated.
Daniel
riak@10.173.240.1)2>
(riak@10.173.240.1)2> {ok, Ring} = riak_core_ring_manager:get_my_ring().
{ok,
{chstate_v2,'riak@10.173.240.1',
[{'riak@10.173.240.1',{303,63561952927}},
{'riak@10.173.240.2',{31,63561952907}},
{'riak@10.173.240.3',{25,63561952907}}],
{128,
[{0,'riak@10.173.240.1'},
{11417981541647679048466287755595961091061972992,
'riak@10.173.240.2'},
{22835963083295358096932575511191922182123945984,
'riak@10.173.240.2'},
{34253944624943037145398863266787883273185918976,
'riak@10.173.240.3'},
{45671926166590716193865151022383844364247891968,
'riak@10.173.240.1'},
{57089907708238395242331438777979805455309864960,
'riak@10.173.240.2'},
{68507889249886074290797726533575766546371837952,
'riak@10.173.240.2'},
{79925870791533753339264014289171727637433810944,
'riak@10.173.240.3'},
{91343852333181432387730302044767688728495783936,
'riak@10.173.240.1'},
{102761833874829111436196589800363649819557756928,
'riak@10.173.240.2'},
{114179815416476790484662877555959610910619729920,
'riak@10.173.240.2'},
{12559779695812446953312916531172001681702912,
'riak@10.173.240.3'},
{137015778499772148581595453067151533092743675904,
'riak@10.173.240.1'},
{148433760041419827630061740822747494183805648896,
'riak@10.173.240.2'},
{159851741583067506678528028578343455274867621888,
'riak@10.173.240.2'},
{171269723124715185726994316333939416365929594880,
'riak@10.173.240.3'},
{182687704666362864775460604089535377456991567872,
'riak@10.173.240.1'},
{194105686208010543823926891845131338548053540864,
'riak@10.173.240.2'},
{205523667749658222872393179600727299639115513856,
'riak@10.173.240.2'},
{216941649291305901920859467356323260730177486848,
and so on
--
View this message in context:
http://riak-users.197444.n3.nabble.com/Partitions-placement-tp4030664.html
Sent from the Riak Users mailing list archive at Nabble.com.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Cluster start and 2i query
Hi Ciprian
Thanks for that info, helps a lot. That was my suspicion about 2i and
primary partitions.
I reproduced it with 4 riak nodes, stopping 2 of them and executing 2i
query.
Since placement of partitions is "random", does it mean I can have max one
node offline to be able to run 2i successfully?
In my original case nodes 4 and 5 were down.
Daniel
On 7 March 2014 15:08, Ciprian Manea [via Riak Users] <
ml-node+s197444n4030631...@n3.nabble.com> wrote:
> Hi Daniel,
>
> Secondary index queries need at least 1/n_val primary partitions to be
> available before it could run successfully and Riak would return
> {error,insufficient_vnodes_available} while the required primary
> partitions are coming up.
>
> I would suggest defensive programming (retrying the 2i queries on error)
> as a way to mitigate this.
>
>
> Thanks,
> Ciprian
>
>
> On Wed, Mar 5, 2014 at 11:06 PM, Daniel Iwan <[hidden
> email]<http://user/SendEmail.jtp?type=node&node=4030631&i=0>
> > wrote:
>
>> Thanks Ciprian
>>
>> We already have wait-for-service in our script and it looks like it's not
>> a
>> sufficient condition to satisfy secondary index query.
>> How long application should wait before starting querying Riak using 2i?
>> Should we do riak-admin transfers to make sure there are no vnode
>> transfers
>> happening?
>>
>> I'm trying to figure it out what {error,insufficient_vnodes_available}
>> means
>> in terms of 2i query.
>> Does it mean not all primary partitions are up?
>>
>> Regards
>> Daniel
>>
>>
>>
>> --
>> View this message in context:
>> http://riak-users.197444.n3.nabble.com/Cluster-start-and-2i-query-tp4030557p4030614.html
>> Sent from the Riak Users mailing list archive at Nabble.com.
>>
>> ___
>> riak-users mailing list
>> [hidden email] <http://user/SendEmail.jtp?type=node&node=4030631&i=1>
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>
>
> ___
> riak-users mailing list
> [hidden email] <http://user/SendEmail.jtp?type=node&node=4030631&i=2>
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
> --
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://riak-users.197444.n3.nabble.com/Cluster-start-and-2i-query-tp4030557p4030631.html
> To unsubscribe from Cluster start and 2i query, click
> here<http://riak-users.197444.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4030557&code=aXdhbi5kYW5pZWxAZ21haWwuY29tfDQwMzA1NTd8LTYxNzE4NjQwNw==>
> .
> NAML<http://riak-users.197444.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context:
http://riak-users.197444.n3.nabble.com/Cluster-start-and-2i-query-tp4030557p4030639.html
Sent from the Riak Users mailing list archive at Nabble.com.___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Cluster start and 2i query
Thanks Ciprian
We already have wait-for-service in our script and it looks like it's not a
sufficient condition to satisfy secondary index query.
How long application should wait before starting querying Riak using 2i?
Should we do riak-admin transfers to make sure there are no vnode transfers
happening?
I'm trying to figure it out what {error,insufficient_vnodes_available} means
in terms of 2i query.
Does it mean not all primary partitions are up?
Regards
Daniel
--
View this message in context:
http://riak-users.197444.n3.nabble.com/Cluster-start-and-2i-query-tp4030557p4030614.html
Sent from the Riak Users mailing list archive at Nabble.com.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Cluster start and 2i query
Any ideas regarding that? Thanks Daniel -- View this message in context: http://riak-users.197444.n3.nabble.com/Cluster-start-and-2i-query-tp4030557p4030610.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Cluster start and 2i query
On 5 node cluster when our servers boot our application (which runs on the
same nodes as riak and queries localhost) I got
Caused by: com.basho.riak.client.RiakRetryFailedException:
com.basho.riak.pbc.RiakError: {error,insufficient_vnodes_available}
at com.basho.riak.client.cap.DefaultRetrier.attempt(DefaultRetrier.java:79)
at com.basho.riak.client.cap.DefaultRetrier.attempt(DefaultRetrier.java:81)
at com.basho.riak.client.cap.DefaultRetrier.attempt(DefaultRetrier.java:81)
at com.basho.riak.client.cap.DefaultRetrier.attempt(DefaultRetrier.java:81)
at com.basho.riak.client.cap.DefaultRetrier.attempt(DefaultRetrier.java:53)
at
com.basho.riak.client.query.indexes.FetchIndex.execute(FetchIndex.java:66)
at server.distributeddb.riak.d.c(SourceFile:461)
... 15 more
Caused by: com.basho.riak.pbc.RiakError:
{error,insufficient_vnodes_available}
at com.basho.riak.pbc.RiakConnection.receive(RiakConnection.java:125)
at com.basho.riak.pbc.RiakClient.processIndexReply(RiakClient.java:446)
at com.basho.riak.pbc.RiakClient.index(RiakClient.java:374)
at
com.basho.riak.client.raw.pbc.PBClientAdapter$3.write(PBClientAdapter.java:445)
at
com.basho.riak.client.raw.query.indexes.BinRangeQuery.write(BinRangeQuery.java:52)
at
com.basho.riak.client.raw.pbc.PBClientAdapter.fetchIndex(PBClientAdapter.java:460)
at
com.basho.riak.client.query.indexes.FetchIndex$1.call(FetchIndex.java:68)
at com.basho.riak.client.query.indexes.FetchIndex$1.call(FetchIndex.java:66)
at com.basho.riak.client.cap.DefaultRetrier.attempt(DefaultRetrier.java:72)
... 21 more
This happened during 2i query on node boot when Riak was not yet ready and
our app started to query Riak. In my case 2 nodes were slower with booting
I know about wait-for-service which probably should be used but I'm not
100% sure it would solve the problem.
I suppose my question is how can I make sure Riak is stable enough to start
querying via 2i
and avoid insufficient_vnodes_available?
Should I wait until at least allNodes-1 is up and running (wait-for-service
reorts Riak node ready?)
On a related note. Does 2i query uses primary vnodes to select covering set?
Regards
Daniel
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Java client, querying using domain bucket and 2i
Hi all Is there a reason there's no 2i querying methods in DomainBucket? That requires to keep both Bucket and DomainBucket references which makes it a bit awkward when passing those around. Thanks Daniel -- View this message in context: http://riak-users.197444.n3.nabble.com/Java-client-querying-using-domain-bucket-and-2i-tp4030476.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Rak 1.3.1 error on start
We have /usr/lib/riak/erts-5.9.1/ installed from official Riak apt package 1.3.1, and we've been using it on numerous installs. No other erlang packages have been installed. Is this the version you are talking about or should we upgrade it? D. -- View this message in context: http://riak-users.197444.n3.nabble.com/Rak-1-3-1-error-on-start-tp4030339p4030368.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Rak 13.1 error on start
Hi Eric
Thanks for that. We change the name during installation but not to
riak_maint_2989@10.173.240.5 which looks like automatically generated with
PID?
Anyways that error looks more like .erlang_cookie file not being written
correctly (empty). Would it happen due to node name change?
D
On 27 January 2014 15:27, Eric Redmond wrote:
> You started Riak under one name, then changed the name later. However, the
> ring doesn't know about this change, hence the error. If you need to change
> a node name, look into the "riak-admin cluster replace" command.
>
>
> http://docs.basho.com/riak/latest/ops/running/tools/riak-admin/#cluster-replace
>
> Eric
> On Jan 27, 2014 7:04 AM, "Daniel Iwan" wrote:
>
>> I just got this right after installing Riak and restarting (Ubuntu
>> 12.04.2)
>>
>> Node name should be riak@10.173.240.5 but is different in this error msg.
>> Vm.args had a correct name ie. riak@10.173.240.5
>> Moving content /var/lib/riak, killing all riak processes and manual launch
>> via riak start
>> fixed it, but still I would like to know what could have caused that.
>>
>> .erlang_cookie was 0 bytes which I guess is the problem.
>> How this can happen?
>> Strange thing is that beam process was running
>>
>> =ERROR REPORT 27-Jan-2014::14:15:48 ===
>> Too short cookie stringescript: exception error: no match of right hand
>> side
>> value
>> {error,
>> {shutdown,
>> {child,undefined,net_sup_dynamic,
>> {erl_distribution,start_link,
>>
>> [['riak_maint_2989@10.173.240.5',longnames]]},
>> permanent,1000,supervisor,
>> [erl_distribution]}}}
>> in function nodetool__escript__1390__832147__907784:process_args/3
>> (/usr/lib/riak/erts-5.9.1/bin/nodetool, line 126)
>> in call from nodetool__escript__1390__832147__907784:main/1
>> (/usr/lib/riak/erts-5.9.1/bin/nodetool, line 15)
>> in call from escript:run/2 (escript.erl, line 727)
>> in call from escript:start/1 (escript.erl, line 277)
>> in call from init:start_it/1
>> in call from init:start_em/1
>> riak_kv is not up:
>> =ERROR REPORT 27-Jan-2014::14:15:51 ===
>>
>>
>> Regards
>> Daniel
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://riak-users.197444.n3.nabble.com/Rak-13-1-error-on-start-tp4030339.html
>> Sent from the Riak Users mailing list archive at Nabble.com.
>>
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Rak 13.1 error on start
I just got this right after installing Riak and restarting (Ubuntu 12.04.2)
Node name should be riak@10.173.240.5 but is different in this error msg.
Vm.args had a correct name ie. riak@10.173.240.5
Moving content /var/lib/riak, killing all riak processes and manual launch
via riak start
fixed it, but still I would like to know what could have caused that.
.erlang_cookie was 0 bytes which I guess is the problem.
How this can happen?
Strange thing is that beam process was running
=ERROR REPORT 27-Jan-2014::14:15:48 ===
Too short cookie stringescript: exception error: no match of right hand side
value
{error,
{shutdown,
{child,undefined,net_sup_dynamic,
{erl_distribution,start_link,
[['riak_maint_2989@10.173.240.5',longnames]]},
permanent,1000,supervisor,
[erl_distribution]}}}
in function nodetool__escript__1390__832147__907784:process_args/3
(/usr/lib/riak/erts-5.9.1/bin/nodetool, line 126)
in call from nodetool__escript__1390__832147__907784:main/1
(/usr/lib/riak/erts-5.9.1/bin/nodetool, line 15)
in call from escript:run/2 (escript.erl, line 727)
in call from escript:start/1 (escript.erl, line 277)
in call from init:start_it/1
in call from init:start_em/1
riak_kv is not up:
=ERROR REPORT 27-Jan-2014::14:15:51 ===
Regards
Daniel
--
View this message in context:
http://riak-users.197444.n3.nabble.com/Rak-13-1-error-on-start-tp4030339.html
Sent from the Riak Users mailing list archive at Nabble.com.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Listing all keys and 2i $key query on a bucket
How "heavy" for the cluster are those two operations for Riak cluster 3-5 nodes? Listing all keys and filtering on client side is definitely not recommended but is 2i query via $key for given bucket equally heavy and not recommended? On related note is there a $bucket query to find all the buckets in the cluster and if there is how heavy is that operation? Thanks Daniel -- View this message in context: http://riak-users.197444.n3.nabble.com/Listing-all-keys-and-2i-key-query-on-a-bucket-tp4030332.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak Search and Yokozuna Backup Strategy
Any comment on that approach? http://hackingdistributed.com/2014/01/14/back-that-nosql-up/ Snippet: HyperDex uses HyperLevelDB as its storage backend, which, in turn, constructs an LSM-tree on disk. The majority of data stored within HyperLevelDB is stored within immutable .sst files. Once written, these files are never overwritten. The backup feature within HyperDex can extract a snapshot of these files via an instantenous hard link in the filesystem. This state, once snapshotted so efficiently, can then be transferred elsewhere at will. At the cluster level, backups pause writes for a short period of time while the HyperDex daemons take their individual snaphsots. Because individual snapshots are efficient, the pause generally takes less than a second. Regards Daniel -- View this message in context: http://riak-users.197444.n3.nabble.com/Riak-Search-and-Yokozuna-Backup-Strategy-tp4030242p4030288.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Bucket properties not updated
There is no coordination between servers so concurrent update of properties is possible. That would certainly explain a lot. In my case though I'm setting allow_mult back to true so eventually that should win? Or would propagation through ring potentially break that logic and allow_multi = false could win? Also could clock drift on the servers be problematic here? It's good to hear you plan to solve that. I'm assuming it's going to be some sort of vclock causality applied to properties? D. On 10 October 2013 21:06, Jordan West wrote: > Hi Daniel, > > You mentioned this was running on three servers simultaneously. Is it > possible that two of the bucket property updates were truly concurrent? If > so, an implementation detail of bucket properties: they are last-write-wins > (and unfortunately the resolution is in milliseconds)*. So its feasible > that in some cases the allow_mult=false would win. > > Cheers, > Jordan > > *this is something we plan to address in future releases. > > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Bucket properties not updated
Hi I found a place in my code where allow_mult is switched to false (during boot) and then back to true. After removing that I could not reproduce the problem (so far). Looks like it may be related to problems Jeremiah reported, and allow_mult getting stuck in false. Thanks for that hint. D. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Bucket properties not updated
Hi I found a place in my code where allow_mult is switched to false (during boot) and then back to true. After removing that I could not reproduce the problem (so far). Looks like it may be related to problems Jeremiah reported, and allow_mult getting stuck in false. Thanks for that hint. D. -- View this message in context: http://riak-users.197444.n3.nabble.com/Bucket-properties-not-updated-tp4029397p4029425.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Bucket properties not updated
Hi I found a place in my code where allow_mult is switched to false (during boot) and then back to true. After removing that I could not reproduce the problem (so far). Looks like it may be related to problems Jeremiah reported, and allow_mult getting stuck in false. Thanks for that hint. D. -- View this message in context: http://riak-users.197444.n3.nabble.com/Bucket-properties-not-updated-tp4029397p4029424.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Delete deleted object
Sometimes I get siblings like this - original object - object modified from machine1 - object modified from machine2 - deleted object 4 siblings for one object. Delete happens only if both machines made modifications to the object, so clearly object was deleted but not removed from Riak db. In my log I see that during delete new vclock is used (the one returned during store in return body phase) which I think is correct behaviour. I thought with that vclock Riak should figure it out that delete was latest operation? What is the best way of handling those scenarios? Should I fetch it and delete it again? Regards Daniel ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Bucket properties not updated
It does sound similar, although in my case I just create brand new bucket
with allow_multi=true, which does not seem to
be accepted (although client says it is). I'm not trying to reset bucket
properties, but maybe it's a good test to to that to see i it gets unstuck.
D.
On 9 October 2013 20:42, Jeremiah Peschka wrote:
> Could this be a duplicate of bug 660?
> https://github.com/basho/riak_kv/issues/660
>
> I can reliably reproduce this on a local 5 node stagedevrel cluster on
> Riak 1.4.2 and develop.
>
> ---
> Jeremiah Peschka - Founder, Brent Ozar Unlimited
> MCITP: SQL Server 2008, MVP
> Cloudera Certified Developer for Apache Hadoop
>
>
> On Wed, Oct 9, 2013 at 12:35 PM, Daniel Iwan wrote:
>
>> Thank for reply.
>>
>> The thing is that bucket never converges. The allow_mult remains false
>> even several minutes after handoff had finished.
>> I don't know whether it's because properties get overwritten by defaults
>> during gossip or it's because message from client(s) never reaches Riak.
>>
>> This situation is potentially very dangerous for us. As I have no way of
>> checking if allow_mult has incorrect value (Riak client returns true)
>> it simply means write loss during updates.
>>
>> Is there a way to debug what's happening or check what's in the ring?
>>
>> Regards
>> Daniel Iwan
>>
>>
>>
>> On 9 October 2013 20:08, Joe Caswell wrote:
>>
>>> Daniel,
>>>
>>> Custom (i.e. non-default) bucket properties are currently stored in
>>> the ring. Once you set a bucket property on one node, the updated ring
>>> must be gossiped to the other nodes. I assume by 'vnode transfer' you mean
>>> handoff. An ownership handoff would also generate changes to the ring
>>> which need to be gossiped, and so could slow the propagation of the bucket
>>> data. If you are setting bucket properties at multiple nodes
>>> simultaneously, there will be additional steps to resolve the change in
>>> order for the ring to converge. If all nodes are online and reachable, the
>>> change should complete very quickly (I would consider 1 second somewhat
>>> slow).
>>>
>>> Joe Caswell
>>>
>>> From: Daniel Iwan
>>> Date: Wednesday, October 9, 2013 7:39 AM
>>> To: "riak-users@lists.basho.com"
>>> Subject: Bucket properties not updated
>>>
>>> Hi
>>>
>>> With Java client 1.1.3 and Riak 1.3.1
>>> I'm doing:
>>>
>>> WriteBucket wb =
>>> iclient.createBucket(BUCKET_NAME).nVal(3).allowSiblings(true);
>>> Bucket b = wb.execute();
>>>
>>> _logger.fine("Regular bucket: " + bucket + ", allows siblings? " +
>>> bucket.getAllowSiblings());
>>>
>>> DomainBucketBuilder< MyObjectInRiak > builder =
>>> DomainBucket.builder(bucket, MyObjectInRiak.class);
>>>
>>> builder.r(RIAK_TX_R_VALUE);
>>> builder.w(RIAK_TX_W_VALUE);
>>> builder.rw(RIAK_TX_RW_VALUE);
>>> builder.returnDeletedVClock(true);
>>> builder.returnBody(true);
>>> builder.basicQuorum(false);
>>>
>>> with custom resolver, mutator and converter.
>>> This code runs on 3 servers at the same time during a test and I know
>>> Riak db is empty
>>> and bucket BUCKET_NAME does not exist.
>>>
>>> log shows that all 3 servers see allowSibbling set to true, but
>>> occasionally test fails and curl shows allow_multi is false on that bucket!
>>>
>>> Is there any delay, race condition etc in Riak, or client library. that
>>> would explain why that bucket property is not set?
>>> During a test vnode transfer occurs but that should not matter I guess?
>>>
>>> Regards
>>> Daniel
>>>
>>>
>>> ___ riak-users mailing list
>>> riak-users@lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>
>>
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Bucket properties not updated
Thank for reply.
The thing is that bucket never converges. The allow_mult remains false even
several minutes after handoff had finished.
I don't know whether it's because properties get overwritten by defaults
during gossip or it's because message from client(s) never reaches Riak.
This situation is potentially very dangerous for us. As I have no way of
checking if allow_mult has incorrect value (Riak client returns true)
it simply means write loss during updates.
Is there a way to debug what's happening or check what's in the ring?
Regards
Daniel Iwan
On 9 October 2013 20:08, Joe Caswell wrote:
> Daniel,
>
> Custom (i.e. non-default) bucket properties are currently stored in the
> ring. Once you set a bucket property on one node, the updated ring must be
> gossiped to the other nodes. I assume by 'vnode transfer' you mean
> handoff. An ownership handoff would also generate changes to the ring
> which need to be gossiped, and so could slow the propagation of the bucket
> data. If you are setting bucket properties at multiple nodes
> simultaneously, there will be additional steps to resolve the change in
> order for the ring to converge. If all nodes are online and reachable, the
> change should complete very quickly (I would consider 1 second somewhat
> slow).
>
> Joe Caswell
>
> From: Daniel Iwan
> Date: Wednesday, October 9, 2013 7:39 AM
> To: "riak-users@lists.basho.com"
> Subject: Bucket properties not updated
>
> Hi
>
> With Java client 1.1.3 and Riak 1.3.1
> I'm doing:
>
> WriteBucket wb =
> iclient.createBucket(BUCKET_NAME).nVal(3).allowSiblings(true);
> Bucket b = wb.execute();
>
> _logger.fine("Regular bucket: " + bucket + ", allows siblings? " +
> bucket.getAllowSiblings());
>
> DomainBucketBuilder< MyObjectInRiak > builder =
> DomainBucket.builder(bucket, MyObjectInRiak.class);
>
> builder.r(RIAK_TX_R_VALUE);
> builder.w(RIAK_TX_W_VALUE);
> builder.rw(RIAK_TX_RW_VALUE);
> builder.returnDeletedVClock(true);
> builder.returnBody(true);
> builder.basicQuorum(false);
>
> with custom resolver, mutator and converter.
> This code runs on 3 servers at the same time during a test and I know Riak
> db is empty
> and bucket BUCKET_NAME does not exist.
>
> log shows that all 3 servers see allowSibbling set to true, but
> occasionally test fails and curl shows allow_multi is false on that bucket!
>
> Is there any delay, race condition etc in Riak, or client library. that
> would explain why that bucket property is not set?
> During a test vnode transfer occurs but that should not matter I guess?
>
> Regards
> Daniel
>
>
> ___ riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Bucket properties not updated
Hi
With Java client 1.1.3 and Riak 1.3.1
I'm doing:
WriteBucket wb =
iclient.createBucket(BUCKET_NAME).nVal(3).allowSiblings(true);
Bucket b = wb.execute();
_logger.fine("Regular bucket: " + bucket + ", allows siblings? " +
bucket.getAllowSiblings());
DomainBucketBuilder< MyObjectInRiak > builder =
DomainBucket.builder(bucket, MyObjectInRiak.class);
builder.r(RIAK_TX_R_VALUE);
builder.w(RIAK_TX_W_VALUE);
builder.rw(RIAK_TX_RW_VALUE);
builder.returnDeletedVClock(true);
builder.returnBody(true);
builder.basicQuorum(false);
with custom resolver, mutator and converter.
This code runs on 3 servers at the same time during a test and I know Riak
db is empty
and bucket BUCKET_NAME does not exist.
log shows that all 3 servers see allowSibbling set to true, but
occasionally test fails and curl shows allow_multi is false on that bucket!
Is there any delay, race condition etc in Riak, or client library. that
would explain why that bucket property is not set?
During a test vnode transfer occurs but that should not matter I guess?
Regards
Daniel
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak Java client not returning deleted sibling
I still did not figure it out what's happening. I wrote a simple test that uses 3 Riak Java clients running on single machine. Each client connects to a different Riak node and uses the same converters, mutators etc as original code. I'm explicitly setting clientId and started updating single key. I can see that siblings are generated and resolved and I always get full set of meta that is merged correctly during resolve At the end of test I can see there are still 4-6 siblings that are resolved but not resolved value not stored (that's what test does) So in the end every client can see all updates and I cannot reproduce problem I see when clients are running directly on Riak nodes. For me it looks like data loss as after store each client can see only it's local changes, and curl shows there are no siblings. Next fetch using Java client shows the same value as curl. I'm using N=3, storing and reading with quorum on r, w and rw, bacisQuorum is false. I would think with those settings clients should see changes from all clients. Any idea why it's not happening and also why I cannot reproduce that with unit test? Thanks Daniel I see that On 7 October 2013 21:21, Daniel Iwan wrote: > I tested that with curl. Should've mentioned that. > The output shows there is no siblings for the key and returned header > looks like this: > > < HTTP/1.1 200 OK > < X-Riak-Vclock: > a85hYGBgymDKBVIc84WrPgUFPS3IYEpkymNlWJZ4/DQfVIpNStsDKsUIlFoOksoCAA== > < x-riak-index-oid_bin: 2a9691be-2f70-11e3-b746-c91c38008704-0 > < x-riak-index-bucket_bin: 2a9691be-2f70-11e3-b746-c91c38008704 > < Vary: Accept-Encoding > < Server: MochiWeb/1.1 WebMachine/1.9.2 (someone had painted it blue) > < my_meta: 5c3ff653-21cb-11e3-96a0-9a712d0ac54a > < Link: ; rel="up" > < Last-Modified: Mon, 07 Oct 2013 16:47:35 GMT > < ETag: "aNfE9KrKGTR7qCW2IDPuf" > < Date: Mon, 07 Oct 2013 17:17:55 GMT > < Content-Type: application/json > < Content-Length: 2223 > > I was thinking that my mutator ConditionalStoreMutation returns false in > hasMutated() > but then I should not see new vclock in result of Store right? > > > D. > > > > > > > On 7 October 2013 20:12, Brian Roach wrote: > >> Daniel - >> >> Unfortunately returning the body from a store operation may not >> reflect all the replicas (and in the case of a concurrent write on two >> different nodes "may not" really means "probably doesn't"). >> >> If you do a subsequent fetch after sending both your writes you'll get >> back a single vclock with siblings. >> >> Thanks, >> - Roach >> >> On Mon, Oct 7, 2013 at 12:37 PM, Daniel Iwan >> wrote: >> > Hi Brian >> > >> > Thanks for update. >> > I'm using 1.1.3 now and still have some issues sibling related >> > >> > Two clients are updating the same key. >> > Updated is my custom meta field, which should be merged to contain >> values >> > from both clients (set) >> > I see both client are doing fetch, resolving sibling (only 1 i.e. no >> > siblings), apply mutation (their own values for meta field). After that >> > object is converted using fromDomain() in my converter using vclock >> provided >> > Both nodes use vclock >> > 6bce61606060cc60ca05521cf385ab3e05053d2dc8604a64cc6365589678fc345f1600 >> > >> > So far so god. >> > But the next step is toDomain (which is pare of Store I think since I'm >> > using withBody) and looks like each node contains info only about >> > it own changes. >> > Client one sees vclock >> > 6bce61606060cc60ca05521cf385ab3e05053d2dc8604a64ca6365589678fc345f1600 >> > Client 2 sees vclock >> > >> 6bce61606060ca60ca05521cf385ab3e05053d2dc8604a64ca6365589678fc341f548a4d4adb032ac508945a0e92ca0200 >> > >> > Both of vclocks are different than original vclock given during store, >> which >> > I assume means RIak accepted write. >> > Resolve is called on both machines but there is only one sibling. >> > >> > I guess the fact that I'm changing only meta field should not matter >> here >> > and I should see 2 siblings? >> > allow_multi is of course true and lastWriteWins is false on that bucket >> > >> > Any help much appreciated >> > >> > >> > Regards >> > Daniel >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > On 4 Oc
Re: Riak Java client not returning deleted sibling
I tested that with curl. Should've mentioned that. The output shows there is no siblings for the key and returned header looks like this: < HTTP/1.1 200 OK < X-Riak-Vclock: a85hYGBgymDKBVIc84WrPgUFPS3IYEpkymNlWJZ4/DQfVIpNStsDKsUIlFoOksoCAA== < x-riak-index-oid_bin: 2a9691be-2f70-11e3-b746-c91c38008704-0 < x-riak-index-bucket_bin: 2a9691be-2f70-11e3-b746-c91c38008704 < Vary: Accept-Encoding < Server: MochiWeb/1.1 WebMachine/1.9.2 (someone had painted it blue) < my_meta: 5c3ff653-21cb-11e3-96a0-9a712d0ac54a < Link: ; rel="up" < Last-Modified: Mon, 07 Oct 2013 16:47:35 GMT < ETag: "aNfE9KrKGTR7qCW2IDPuf" < Date: Mon, 07 Oct 2013 17:17:55 GMT < Content-Type: application/json < Content-Length: 2223 I was thinking that my mutator ConditionalStoreMutation returns false in hasMutated() but then I should not see new vclock in result of Store right? D. On 7 October 2013 20:12, Brian Roach wrote: > Daniel - > > Unfortunately returning the body from a store operation may not > reflect all the replicas (and in the case of a concurrent write on two > different nodes "may not" really means "probably doesn't"). > > If you do a subsequent fetch after sending both your writes you'll get > back a single vclock with siblings. > > Thanks, > - Roach > > On Mon, Oct 7, 2013 at 12:37 PM, Daniel Iwan > wrote: > > Hi Brian > > > > Thanks for update. > > I'm using 1.1.3 now and still have some issues sibling related > > > > Two clients are updating the same key. > > Updated is my custom meta field, which should be merged to contain values > > from both clients (set) > > I see both client are doing fetch, resolving sibling (only 1 i.e. no > > siblings), apply mutation (their own values for meta field). After that > > object is converted using fromDomain() in my converter using vclock > provided > > Both nodes use vclock > > 6bce61606060cc60ca05521cf385ab3e05053d2dc8604a64cc6365589678fc345f1600 > > > > So far so god. > > But the next step is toDomain (which is pare of Store I think since I'm > > using withBody) and looks like each node contains info only about > > it own changes. > > Client one sees vclock > > 6bce61606060cc60ca05521cf385ab3e05053d2dc8604a64ca6365589678fc345f1600 > > Client 2 sees vclock > > > 6bce61606060ca60ca05521cf385ab3e05053d2dc8604a64ca6365589678fc341f548a4d4adb032ac508945a0e92ca0200 > > > > Both of vclocks are different than original vclock given during store, > which > > I assume means RIak accepted write. > > Resolve is called on both machines but there is only one sibling. > > > > I guess the fact that I'm changing only meta field should not matter here > > and I should see 2 siblings? > > allow_multi is of course true and lastWriteWins is false on that bucket > > > > Any help much appreciated > > > > > > Regards > > Daniel > > > > > > > > > > > > > > > > > > > > > > > > > > On 4 October 2013 21:41, Brian Roach wrote: > >> > >> Hey - > >> > >> I'm releasing 1.1.3 and 1.4.2 but it'll take a while for them to get > >> staged at maven central so I can post an "official" release to the > >> mailing list. > >> > >> I've gone ahead and uploaded the jar-with-dependencies to the usual > >> place for you- > >> > >> > >> > http://riak-java-client.s3.amazonaws.com/riak-client-1.1.3-jar-with-dependencies.jar > >> > >> It fixes up the DomainBucket stuff and the JSONConverter. > >> > >> Thanks, > >> - Roach > >> > >> On Fri, Oct 4, 2013 at 2:58 AM, Daniel Iwan > wrote: > >> > Thanks Brian for putting fix together so quickly. > >> > > >> > I think I found something else though. > >> > In JSONConverter I don't see vclock being set in toDomain() when > >> > converting > >> > deleted sibling? > >> > That vclock should be used for following delete if I understood it > >> > correctly? > >> > > >> > Also where can I download latest build? I tried > >> > > >> > > http://riak-java-client.s3.amazonaws.com/riak-client-1.1.3-jar-with-dependencies.jar > >> > but access is denied > >> > > >> > Cheers > >> > Daniel > >> > > >> > > >> > On 3 October 2013 19:36, Brian Roach wrote: > >> >> > >> >> On Thu, Oct 3,
Re: Riak Java client not returning deleted sibling
Hi Brian Thanks for update. I'm using 1.1.3 now and still have some issues sibling related Two clients are updating the same key. Updated is my custom meta field, which should be merged to contain values from both clients (set) I see both client are doing fetch, resolving sibling (only 1 i.e. no siblings), apply mutation (their own values for meta field). After that object is converted using fromDomain() in my converter using vclock provided Both nodes use vclock 6bce61606060cc60ca05521cf385ab3e05053d2dc8604a64cc6365589678fc345f1600 So far so god. But the next step is toDomain (which is pare of Store I think since I'm using withBody) and looks like each node contains info only about it own changes. Client one sees vclock 6bce61606060cc60ca05521cf385ab3e05053d2dc8604a64ca6365589678fc345f1600 Client 2 sees vclock 6bce61606060ca60ca05521cf385ab3e05053d2dc8604a64ca6365589678fc341f548a4d4adb032ac508945a0e92ca0200 Both of vclocks are different than original vclock given during store, which I assume means RIak accepted write. Resolve is called on both machines but there is only one sibling. I guess the fact that I'm changing only meta field should not matter here and I should see 2 siblings? allow_multi is of course true and lastWriteWins is false on that bucket Any help much appreciated Regards Daniel On 4 October 2013 21:41, Brian Roach wrote: > Hey - > > I'm releasing 1.1.3 and 1.4.2 but it'll take a while for them to get > staged at maven central so I can post an "official" release to the > mailing list. > > I've gone ahead and uploaded the jar-with-dependencies to the usual > place for you- > > > http://riak-java-client.s3.amazonaws.com/riak-client-1.1.3-jar-with-dependencies.jar > > It fixes up the DomainBucket stuff and the JSONConverter. > > Thanks, > - Roach > > On Fri, Oct 4, 2013 at 2:58 AM, Daniel Iwan wrote: > > Thanks Brian for putting fix together so quickly. > > > > I think I found something else though. > > In JSONConverter I don't see vclock being set in toDomain() when > converting > > deleted sibling? > > That vclock should be used for following delete if I understood it > > correctly? > > > > Also where can I download latest build? I tried > > > http://riak-java-client.s3.amazonaws.com/riak-client-1.1.3-jar-with-dependencies.jar > > but access is denied > > > > Cheers > > Daniel > > > > > > On 3 October 2013 19:36, Brian Roach wrote: > >> > >> On Thu, Oct 3, 2013 at 10:32 AM, Daniel Iwan > >> wrote: > >> > Thanks Brian for quick response. > >> > > >> > As a side question, what is the best way to delete such an object i.e. > >> > once > >> > I know one of the siblings has 'deleted' flag true because I fetched > it? > >> > Should I just use DomainBucket.delete(key) without providing any > vclock? > >> > Would it wipe it from Riak or create yet another sibling? > >> > >> You should always use vclocks when possible, which in the case it is. > >> There are additional issues around doing the delete without a vclock > >> and if there's concurrently a store operation occurring. > >> > >> Ideally you should look at why you're getting that tombstone sibling. > >> If it's simply a case of high write concurrency and you're using > >> vclocks with your writes, then there's not much you can do except > >> resolve it later (without changing how you're using the DB)... but > >> usually these things are caused by writes without a vclock. > >> > >> Thanks, > >> - Roach > >> > >> > >> > >> > >> > > >> > Regards > >> > Daniel > >> > > >> > > >> > On 3 October 2013 17:20, Brian Roach wrote: > >> >> > >> >> Daniel - > >> >> > >> >> Yeah, that is the case. When the ability to pass fetch/store/delete > >> >> meta was added to DomainBucket way back when it appears that was > >> >> missed. > >> >> > >> >> I'll add it and forward-port to 1.4.x as well and cut new jars. > Should > >> >> be avail by tomorrow morning at the latest. > >> >> > >> >> Thanks! > >> >> - Roach > >> >> > >> >> On Thu, Oct 3, 2013 at 9:38 AM, Daniel Iwan > >> >> wrote: > >> >> > Hi I'm using Riak 1.3.1 and Java client 1.1.2 > >> >> > > >>
Re: Riak Java client not returning deleted sibling
Thanks Brian for putting fix together so quickly.
I think I found something else though.
In JSONConverter I don't see vclock being set in toDomain() when converting
deleted sibling?
That vclock should be used for following delete if I understood it
correctly?
Also where can I download latest build? I tried
http://riak-java-client.s3.amazonaws.com/riak-client-1.1.3-jar-with-dependencies.jar
but access is denied
Cheers
Daniel
On 3 October 2013 19:36, Brian Roach wrote:
> On Thu, Oct 3, 2013 at 10:32 AM, Daniel Iwan
> wrote:
> > Thanks Brian for quick response.
> >
> > As a side question, what is the best way to delete such an object i.e.
> once
> > I know one of the siblings has 'deleted' flag true because I fetched it?
> > Should I just use DomainBucket.delete(key) without providing any vclock?
> > Would it wipe it from Riak or create yet another sibling?
>
> You should always use vclocks when possible, which in the case it is.
> There are additional issues around doing the delete without a vclock
> and if there's concurrently a store operation occurring.
>
> Ideally you should look at why you're getting that tombstone sibling.
> If it's simply a case of high write concurrency and you're using
> vclocks with your writes, then there's not much you can do except
> resolve it later (without changing how you're using the DB)... but
> usually these things are caused by writes without a vclock.
>
> Thanks,
> - Roach
>
>
>
>
> >
> > Regards
> > Daniel
> >
> >
> > On 3 October 2013 17:20, Brian Roach wrote:
> >>
> >> Daniel -
> >>
> >> Yeah, that is the case. When the ability to pass fetch/store/delete
> >> meta was added to DomainBucket way back when it appears that was
> >> missed.
> >>
> >> I'll add it and forward-port to 1.4.x as well and cut new jars. Should
> >> be avail by tomorrow morning at the latest.
> >>
> >> Thanks!
> >> - Roach
> >>
> >> On Thu, Oct 3, 2013 at 9:38 AM, Daniel Iwan
> wrote:
> >> > Hi I'm using Riak 1.3.1 and Java client 1.1.2
> >> >
> >> > Using http and curl I see 4 siblings for an object one of which has
> >> > X-Riak-Deleted: true
> >> > but when I'm using Java client with DomainBucket my Converter's method
> >> > toDomain is called only 3 times.
> >> >
> >> > I have set the property
> >> >
> >> > builder.returnDeletedVClock(true);
> >> >
> >> > on my DomainBuilder which I keep reusing for all queries and store
> >> > operations (I guess that's good practice btw.?)
> >> >
> >> >
> >> > I run that under debugger and it seems raw client sees 4 siblings but
> >> > passes
> >> > over only 3 due to bug (I think) in DomainBucket.fetch() method which
> >> > should
> >> > have
> >> >
> >> > if (fetchMeta.hasReturnDeletedVClock()) {
> >> >
> >> >
> so.returnDeletedVClock(fetchMeta.getReturnDeletedVClock());
> >> >
> >> > }
> >> >
> >> > at the end, as store() method has.
> >> >
> >> > Could you confirm or I'm I completely wrong?
> >> >
> >> >
> >> > Regards
> >> >
> >> > Daniel
> >> >
> >> >
> >> > ___
> >> > riak-users mailing list
> >> > riak-users@lists.basho.com
> >> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >> >
> >
> >
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak Java client not returning deleted sibling
Thanks Brian for quick response.
As a side question, what is the best way to delete such an object i.e. once
I know one of the siblings has 'deleted' flag true because I fetched it?
Should I just use DomainBucket.delete(key) without providing any vclock?
Would it wipe it from Riak or create yet another sibling?
Regards
Daniel
On 3 October 2013 17:20, Brian Roach wrote:
> Daniel -
>
> Yeah, that is the case. When the ability to pass fetch/store/delete
> meta was added to DomainBucket way back when it appears that was
> missed.
>
> I'll add it and forward-port to 1.4.x as well and cut new jars. Should
> be avail by tomorrow morning at the latest.
>
> Thanks!
> - Roach
>
> On Thu, Oct 3, 2013 at 9:38 AM, Daniel Iwan wrote:
> > Hi I'm using Riak 1.3.1 and Java client 1.1.2
> >
> > Using http and curl I see 4 siblings for an object one of which has
> > X-Riak-Deleted: true
> > but when I'm using Java client with DomainBucket my Converter's method
> > toDomain is called only 3 times.
> >
> > I have set the property
> >
> > builder.returnDeletedVClock(true);
> >
> > on my DomainBuilder which I keep reusing for all queries and store
> > operations (I guess that's good practice btw.?)
> >
> >
> > I run that under debugger and it seems raw client sees 4 siblings but
> passes
> > over only 3 due to bug (I think) in DomainBucket.fetch() method which
> should
> > have
> >
> > if (fetchMeta.hasReturnDeletedVClock()) {
> >
> > so.returnDeletedVClock(fetchMeta.getReturnDeletedVClock());
> >
> > }
> >
> > at the end, as store() method has.
> >
> > Could you confirm or I'm I completely wrong?
> >
> >
> > Regards
> >
> > Daniel
> >
> >
> > ___
> > riak-users mailing list
> > riak-users@lists.basho.com
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak Java client not returning deleted sibling
Hi I'm using Riak 1.3.1 and Java client 1.1.2
Using http and curl I see 4 siblings for an object one of which
has X-Riak-Deleted: true
but when I'm using Java client with DomainBucket my Converter's method
toDomain is called only 3 times.
I have set the property
builder.returnDeletedVClock(true);
on my DomainBuilder which I keep reusing for all queries and store
operations (I guess that's good practice btw.?)
I run that under debugger and it seems raw client sees 4 siblings but
passes over only 3 due to bug (I think) in DomainBucket.fetch() method
which should have
if (fetchMeta.hasReturnDeletedVClock()) {
so.returnDeletedVClock(fetchMeta.getReturnDeletedVClock());
}
at the end, as store() method has.
Could you confirm or I'm I completely wrong?
Regards
Daniel
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
VNodes distribution on the ring
Is there anywhere a pseudo-code or description of the algorithm how vnodes (primaries and replicas) would be distributed if I had 3, 4 and more nodes in the cluster? Does it depend in any way on the node name or any other setting, or is it only a function of number of physical nodes? Regards Daniel ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Getting largest key
You can store revertIndex = (MAX_KEY_VALUE - keyvaluefromberkley) in Riak as a secondary index for every object. Then get a full range for that index limiting results to 1. In this way you'll get one result with max keyvaluefromberkley. Reversing order in a nutshell, because I think values for 2i in Riak are always ascending. Regards Daniel -- View this message in context: http://riak-users.197444.n3.nabble.com/no-subject-tp4029126p4029153.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Empty bucket disappears
Makes perfect sense. Now I understand why is highly discouraged do to that Thanks for quick reply. Daniel On 5 September 2013 18:11, Brian Sparrow wrote: > Daniel, > > When you do a bucket list you are actually listing all keys in 1/N > partitions and extracting the bucket names from the keys(which are > bucket++key). So, when you remove all the keys in a bucket you effectively > remove the bucket because no keys in the keylist will return as being > members of that bucket. > > -- > Brian Sparrow > Developer Advocate > Basho Technologies > > Sent with Sparrow <http://www.sparrowmailapp.com/?sig> > > On Thursday, September 5, 2013 at 12:53 PM, Daniel Iwan wrote: > > If I remove all keys from a bucket that bucket is not visible when I do > > curl http://127.0.0.1:8098/buckets?buckets=true > > I know buckets are only prefixes for keys so in theory bucket does not > know if it's empty (or maybe it does) but to me it looks like only buckets > with keys are visible. > > Is there any purging mechanism in Riak to remove unused buckets? > > Regards > Daniel > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Empty bucket disappears
If I remove all keys from a bucket that bucket is not visible when I do curl http://127.0.0.1:8098/buckets?buckets=true I know buckets are only prefixes for keys so in theory bucket does not know if it's empty (or maybe it does) but to me it looks like only buckets with keys are visible. Is there any purging mechanism in Riak to remove unused buckets? Regards Daniel ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: riak-admin diag output
Thanks Jared I'm aware of limitations of 3-node cluster. If I understand it correctly there are some corner cases where certain copies for some vnodes can land on the same physical node. But I would assume there is no case where all 3 copies (for N=3) should land on the same physical node. Hence I'm thinking 1-node failure means we are still safe. riak-admin diag output seems bit strange to me showing particular vnode twice. Any reason for that? Also, where can I find description of algorithm which vnodes are assigned to which physical node? I think there was blog post about it... Daniel ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
riak-admin diag output
Hi my riak admin diag shows output as below(3-node cluster)
I'm assuming long numbers are vnodes. Strange thing is:
5708990770823839524233143877797980545530986496 exist twice for the same node
19981467697883438334816003572292931909358452736 once on the list
How do I interpret this?
How can I list all vnodes and nodes that they are exist on, riak-admin
vnode-status
shows only primary locations, what about copies?
me@node3:~$ riak-admin diag
Attempting to restart script through sudo -H -u riak
14:19:51.277 [critical] vm.swappiness is 1, should be no more than 0)
14:19:51.277 [critical] net.core.wmem_default is 229376, should be at least
8388608)
14:19:51.278 [critical] net.core.rmem_default is 229376, should be at least
8388608)
14:19:51.278 [critical] net.core.wmem_max is 131071, should be at least
8388608)
14:19:51.278 [critical] net.core.rmem_max is 131071, should be at least
8388608)
14:19:51.278 [critical] net.core.netdev_max_backlog is 1000, should be at
least 1)
14:19:51.278 [critical] net.core.somaxconn is 128, should be at least 4000)
14:19:51.278 [critical] net.ipv4.tcp_max_syn_backlog is 2048, should be at
least 4)
14:19:51.278 [critical] net.ipv4.tcp_fin_timeout is 60, should be no more
than 15)
14:19:51.278 [critical] net.ipv4.tcp_tw_reuse is 0, should be 1)
14:19:51.278 [warning] The following preflists do not satisfy the n_val:
[[{0,'riak@10.173.240.1'},{2854495385411919762116571938898990272765493248,'
riak@10.173.240.2'},{5708990770823839524233143877797980545530986496,'
riak@10.173.240.2'}],[{2854495385411919762116571938898990272765493248,'
riak@10.173.240.2'},{5708990770823839524233143877797980545530986496,'
riak@10.173.240.2'},{8563486156235759286349715816696970818296479744,'
riak@10.173.240.3'}],[{11417981541647679048466287755595961091061972992,'
riak@10.173.240.1'},{14272476927059598810582859694494951363827466240,'
riak@10.173.240.2'},{17126972312471518572699431633393941636592959488,'
riak@10.173.240.2'}],[{14272476927059598810582859694494951363827466240,'
riak@10.173.240.2'},{17126972312471518572699431633393941636592959488,'
riak@10.173.240.2'},{19981467697883438334816003572292931909358452736,'
riak@10.173.240.3'}],[{22835963083295358096932575511191922182123945984,'
riak@10.173.240.1'},{25690458468707277859049147450090912454889439232,'
riak@10.173.240.2'},{28544953854119197621165719388989902727654932480,'
riak@10.173.240.2'}],[{25690458468707277859049147450090912454889439232,'
riak@10.173.240.2'},{28544953854119197621165719388989902727654932480,'
riak@10.173.240.2'},{3139944923953111738328229132793000420425728,'
riak@10.173.240.3'}],[{34253944624943037145398863266787883273185918976,'
riak@10.173.240.1'},{37108440010354956907515435205686873545951412224,'
riak@10.173.240.2'},{39962935395766876669632007144585863818716905472,'
riak@10.173.240.2'}],[{37108440010354956907515435205686873545951412224,'
riak@10.173.240.2'},{39962935395766876669632007144585863818716905472,'
riak@10.173.240.2'},{42817430781178796431748579083484854091482398720,'
riak@10.173.240.3'}],[{45671926166590716193865151022383844364247891968,'
riak@10.173.240.1'},{48526421552002635955981722961282834637013385216,'
riak@10.173.240.2'},{51380916937414555718098294900181824909778878464,'
riak@10.173.240.2'}],[{48526421552002635955981722961282834637013385216,'
riak@10.173.240.2'},{51380916937414555718098294900181824909778878464,'
riak@10.173.240.2'},{54235412322826475480214866839080815182544371712,'
riak@10.173.240.3'}],[{57089907708238395242331438777979805455309864960,'
riak@10.173.240.1'},{59944403093650315004448010716878795728075358208,'
riak@10.173.240.2'},{6279889847906223476656458265586000840851456,'
riak@10.173.240.2'}],[{59944403093650315004448010716878795728075358208,'
riak@10.173.240.2'},{6279889847906223476656458265586000840851456,'
riak@10.173.240.2'},{65653393864474154528681154594676776273606344704,'
riak@10.173.240.3'}],[{68507889249886074290797726533575766546371837952,'
riak@10.173.240.1'},{71362384635297994052914298472474756819137331200,'
riak@10.173.240.2'},{74216880020709913815030870411373747091902824448,'
riak@10.173.240.2'}],[{71362384635297994052914298472474756819137331200,'
riak@10.173.240.2'},{74216880020709913815030870411373747091902824448,'
riak@10.173.240.2'},{77071375406121833577147442350272737364668317696,'
riak@10.173.240.3'}],[{79925870791533753339264014289171727637433810944,'
riak@10.173.240.1'},{82780366176945673101380586228070717910199304192,'
riak@10.173.240.2'},{85634861562357592863497158166969708182964797440,'
riak@10.173.240.2'}],[{82780366176945673101380586228070717910199304192,'
riak@10.173.240.2'},{85634861562357592863497158166969708182964797440,'
riak@10.173.240.2'},{88489356947769512625613730105868698455730290688,'
riak@10.173.240.3'}],[{91343852333181432387730302044767688728495783936,'
riak@10.173.240.1'},{9419834771859335214984687398379001261277184,'
riak@10.173.240.2'},{97052843104005271911963445922565669274026770432,'
riak@10.173.240.2'}],[{94
Re: Riak node joining
Four days passed and node is still joining.
I haven't tried to restart it (which would probably fix the issue) as I
would like to find out what was the real reason of that stall and what to
do
to avoid it in the future.
Any suggestions?
Daniel
On 27 June 2013 00:19, Daniel Iwan wrote:
> Hi all
>
> I see node stalled at 'joining' for good 8 hours now:
> 3-node cluster v1.3.1, 512 vnodes (way too high but that's another
> matter), leveldb backend
> Cluster was originally 2-nodes only and after upgrading to 1.3.1 we
> attached another node
> No active transfers on the nodes at the moment, but from error log looks
> like vnode transfer stopped
>
> riak-admin member-status
> Attempting to restart script through sudo -H -u riak
> = Membership
> ==
> Status RingPendingNode
>
> ---
> joining33.2% -- 'riak@10.173.240.3'
> valid 33.6% -- 'riak@10.173.240.2'
> valid 33.2% -- 'riak@10.173.240.21'
> ---
> Valid:2 / Leaving:0 / Exiting:0 / Joining:1 / Down:0
>
>
>
> riak-admin ring-status
> Attempting to restart script through sudo -H -u riak
> == Claimant
> ===
> Claimant: 'riak@10.173.240.21'
> Status: up
> Ring Ready: true
>
> == Ownership Handoff
> ==
> No pending changes.
>
> == Unreachable Nodes
> ==
> All nodes are up and reachable
>
>
>
> Node that is joining shows
> *error.log*
> 2013-06-26 16:35:50.351 [error] <0.7222.0> gen_fsm <0.7222.0> in state
> ready terminated with reason:
> {timeout,{gen_fsm,sync_send_all_state_event,[<0.7223.0>,stop]}}
> 2013-06-26 16:35:51.549 [error] <0.7222.0> CRASH REPORT Process <0.7222.0>
> with 0 neighbours exited with reason:
> {timeout,{gen_fsm,sync_send_all_state_event,[<0.7223.0>,stop]}} in
> gen_fsm:handle_msg/7 line 511
> 2013-06-26 16:35:52.460 [error] <0.5316.0> gen_fsm <0.5316.0> in state
> ready terminated with reason:
> {timeout,{gen_fsm,sync_send_all_state_event,[<0.5317.0>,stop]}}
> 2013-06-26 16:35:52.696 [error] <0.5316.0> CRASH REPORT Process <0.5316.0>
> with 0 neighbours exited with reason:
> {timeout,{gen_fsm,sync_send_all_state_event,[<0.5317.0>,stop]}} in
> gen_fsm:handle_msg/7 line 511
> 2013-06-26 16:35:52.867 [error] <0.9088.0> gen_fsm <0.9088.0> in state
> ready terminated with reason:
> {timeout,{gen_fsm,sync_send_all_state_event,[<0.9089.0>,stop]}}
> 2013-06-26 16:35:53.007 [error] <0.9088.0> CRASH REPORT Process <0.9088.0>
> with 0 neighbours exited with reason:
> {timeout,{gen_fsm,sync_send_all_state_event,[<0.9089.0>,stop]}} in
> gen_fsm:handle_msg/7 line 511
> 2013-06-26 16:35:53.127 [error] <0.5794.0> gen_fsm <0.5794.0> in state
> ready terminated with reason:
> {timeout,{gen_fsm,sync_send_all_state_event,[<0.5795.0>,stop]}}
> 2013-06-26 16:35:53.248 [error] <0.5794.0> CRASH REPORT Process <0.5794.0>
> with 0 neighbours exited with reason:
> {timeout,{gen_fsm,sync_send_all_state_event,[<0.5795.0>,stop]}} in
> gen_fsm:handle_msg/7 line 511
> 2013-06-26 16:35:53.368 [error] <0.3990.0> gen_fsm <0.3990.0> in state
> ready terminated with reason:
> {timeout,{gen_fsm,sync_send_all_state_event,[<0.3991.0>,stop]}}
> 2013-06-26 16:35:53.498 [error] <0.3990.0> CRASH REPORT Process <0.3990.0>
> with 0 neighbours exited with reason:
> {timeout,{gen_fsm,sync_send_all_state_event,[<0.3991.0>,stop]}} in
> gen_fsm:handle_msg/7 line 511
> 2013-06-26 16:35:53.619 [error] <0.7017.0> gen_fsm <0.7017.0> in state
> ready terminated with reason:
> {timeout,{gen_fsm,sync_send_all_state_event,[<0.7018.0>,stop]}}
> 2013-06-26 16:35:53.739 [error] <0.7017.0> CRASH REPORT Process <0.7017.0>
> with 0 neighbours exited with reason:
> {timeout,{gen_fsm,sync_send_all_state_event,[<0.7018.0>,stop]}} in
> gen_fsm:handle_msg/7 line 511
> 2013-06-26 16:35:53.859 [error] <0.5120.0> gen_fsm <0.5120.0> in state
> ready terminated with reason:
> {timeout,{gen_fsm,sync_send_all_state_event,[<0.5121.0>,stop]}}
> 2013-06-26 16:35:53.980 [error] <0.5120.0> CRASH REPORT Process <0.5120.0>
> with 0 neighbours exited with reason:
> {timeout,{g
Riak node joining
Hi all
I see node stalled at 'joining' for good 8 hours now:
3-node cluster v1.3.1, 512 vnodes (way too high but that's another matter),
leveldb backend
Cluster was originally 2-nodes only and after upgrading to 1.3.1 we
attached another node
No active transfers on the nodes at the moment, but from error log looks
like vnode transfer stopped
riak-admin member-status
Attempting to restart script through sudo -H -u riak
= Membership
==
Status RingPendingNode
---
joining33.2% -- 'riak@10.173.240.3'
valid 33.6% -- 'riak@10.173.240.2'
valid 33.2% -- 'riak@10.173.240.21'
---
Valid:2 / Leaving:0 / Exiting:0 / Joining:1 / Down:0
riak-admin ring-status
Attempting to restart script through sudo -H -u riak
== Claimant
===
Claimant: 'riak@10.173.240.21'
Status: up
Ring Ready: true
== Ownership Handoff
==
No pending changes.
== Unreachable Nodes
==
All nodes are up and reachable
Node that is joining shows
*error.log*
2013-06-26 16:35:50.351 [error] <0.7222.0> gen_fsm <0.7222.0> in state
ready terminated with reason:
{timeout,{gen_fsm,sync_send_all_state_event,[<0.7223.0>,stop]}}
2013-06-26 16:35:51.549 [error] <0.7222.0> CRASH REPORT Process <0.7222.0>
with 0 neighbours exited with reason:
{timeout,{gen_fsm,sync_send_all_state_event,[<0.7223.0>,stop]}} in
gen_fsm:handle_msg/7 line 511
2013-06-26 16:35:52.460 [error] <0.5316.0> gen_fsm <0.5316.0> in state
ready terminated with reason:
{timeout,{gen_fsm,sync_send_all_state_event,[<0.5317.0>,stop]}}
2013-06-26 16:35:52.696 [error] <0.5316.0> CRASH REPORT Process <0.5316.0>
with 0 neighbours exited with reason:
{timeout,{gen_fsm,sync_send_all_state_event,[<0.5317.0>,stop]}} in
gen_fsm:handle_msg/7 line 511
2013-06-26 16:35:52.867 [error] <0.9088.0> gen_fsm <0.9088.0> in state
ready terminated with reason:
{timeout,{gen_fsm,sync_send_all_state_event,[<0.9089.0>,stop]}}
2013-06-26 16:35:53.007 [error] <0.9088.0> CRASH REPORT Process <0.9088.0>
with 0 neighbours exited with reason:
{timeout,{gen_fsm,sync_send_all_state_event,[<0.9089.0>,stop]}} in
gen_fsm:handle_msg/7 line 511
2013-06-26 16:35:53.127 [error] <0.5794.0> gen_fsm <0.5794.0> in state
ready terminated with reason:
{timeout,{gen_fsm,sync_send_all_state_event,[<0.5795.0>,stop]}}
2013-06-26 16:35:53.248 [error] <0.5794.0> CRASH REPORT Process <0.5794.0>
with 0 neighbours exited with reason:
{timeout,{gen_fsm,sync_send_all_state_event,[<0.5795.0>,stop]}} in
gen_fsm:handle_msg/7 line 511
2013-06-26 16:35:53.368 [error] <0.3990.0> gen_fsm <0.3990.0> in state
ready terminated with reason:
{timeout,{gen_fsm,sync_send_all_state_event,[<0.3991.0>,stop]}}
2013-06-26 16:35:53.498 [error] <0.3990.0> CRASH REPORT Process <0.3990.0>
with 0 neighbours exited with reason:
{timeout,{gen_fsm,sync_send_all_state_event,[<0.3991.0>,stop]}} in
gen_fsm:handle_msg/7 line 511
2013-06-26 16:35:53.619 [error] <0.7017.0> gen_fsm <0.7017.0> in state
ready terminated with reason:
{timeout,{gen_fsm,sync_send_all_state_event,[<0.7018.0>,stop]}}
2013-06-26 16:35:53.739 [error] <0.7017.0> CRASH REPORT Process <0.7017.0>
with 0 neighbours exited with reason:
{timeout,{gen_fsm,sync_send_all_state_event,[<0.7018.0>,stop]}} in
gen_fsm:handle_msg/7 line 511
2013-06-26 16:35:53.859 [error] <0.5120.0> gen_fsm <0.5120.0> in state
ready terminated with reason:
{timeout,{gen_fsm,sync_send_all_state_event,[<0.5121.0>,stop]}}
2013-06-26 16:35:53.980 [error] <0.5120.0> CRASH REPORT Process <0.5120.0>
with 0 neighbours exited with reason:
{timeout,{gen_fsm,sync_send_all_state_event,[<0.5121.0>,stop]}} in
gen_fsm:handle_msg/7 line 511
2013-06-26 16:41:27.819 [error]
<0.10046.0>@riak_core_handoff_sender:start_fold:226 hinted_handoff transfer
of riak_kv_vnode from 'riak@10.173.240.3'
961964944883816959833284743408959721921971224576 to 'riak@10.173.240.2'
961964944883816959833284743408959721921971224576 failed because of
exit:{noproc,{riak_core_gen_server,call,[{riak_kv_handoff_listener,'
riak@10.173.240.2'},handoff_port,infinity]}}
[{riak_core_gen_server,call,3,[{file,"src/riak_core_gen_server.erl"},{line,214}]},{riak_core_handoff_sender,start_fold,5,[{file,"src/riak_core_handoff_sender.erl"},{line,84}]}]
2013-06-26 16:41:27.819 [error]
<0.10045.0>@riak_core_handoff_sender:start_fold:226 hinted_handoff transfer
of riak_kv_vnode from 'riak@10.173.240.3'
790695221759101774106290427075020305556041629696 to 'riak@10.173.240.2'
790695221759101774106290427075020305556041629696 failed because of
exit:{noproc,{riak_core_gen_server,call,[{riak_kv_handoff_listener,'
riak@10.173.240.2'},handoff_port,infinity]}}
[{riak_cor
Re: Reformatting 2i in Riak 1.3.1
Thanks Russell for quick reply Regarding intermediate upgrade I was under the impression that Riak upgrades are supported from one version behind so 1.0 -> 1.1, 1.1 -> 1.2 1.2 -> 1.3.x are allowed but 1.1 -> 1.3 is not Am I wrong here? I could not find relevant info in Rolling Upgrades doc, I think I've read it somewhere on mailing list. Daniel On 30 April 2013 09:59, Russell Brown wrote: > > On 30 Apr 2013, at 09:47, Daniel Iwan wrote: > > > When doing migration from pre-1.3.1 do I run > > > > riak-admin reformat-indexes [] [] > > > > on every node that is part of the cluster or just one and then it > magically applies change to all of them? Changelog says: > > Riak 1.3.1 includes a utility, as part of riak-admin, that will perform > the reformatting of these indexes while the node is online > > > > which suggests I need to run it on every node. > > Yes, please run it on every node. > > > > > Also can I upgrade from 1.2 to 1.3.1 or do I have to go via intermediate > upgrade 1.3.0 ? > > Yes. You should. > > Cheers > > Russell > > > > > Thanks > > Daniel > > ___ > > riak-users mailing list > > riak-users@lists.basho.com > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reformatting 2i in Riak 1.3.1
When doing migration from pre-1.3.1 do I run riak-admin reformat-indexes [] [] on every node that is part of the cluster or just one and then it magically applies change to all of them? Changelog says: Riak 1.3.1 includes a utility, as part of riak-admin, that will perform the reformatting of these indexes while the node is online which suggests I need to run it on every node. Also can I upgrade from 1.2 to 1.3.1 or do I have to go via intermediate upgrade 1.3.0 ? Thanks Daniel ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Keys not listed during vnode transfers
Hi Mark Thanks for info. For the moment I thought we were the only one experiencing this. At the moment the only workaround for us is to wait until vnode transfer is finished. I suspect attaching a node also will trigger that behaviour. Considering that expanding cluster from 3 to 4 nodes can take quite some time the fact that we are not able to receive all keys via 2i means system is not available during attaching a node(s). I'll keep an eye on the issue and I'll put as much information I can. Daniel. On 2 April 2013 22:46, Mark Phillips wrote: > Hi Daniel, > > Sorry for the delay here. > > It looks like it's going to take a more investigation to nail down the > cause. Engel just opened an issue you can keep an eye on: > > https://github.com/basho/riak/issues/305 > > Thanks for reporting. Feel free to add any relevant information to the > issue. > > Mark > > On Thursday, March 14, 2013, Daniel Iwan wrote: > >> Maybe someone from Basho could shed some light on that issue? >> >> Regards >> Daniel >> >> >> On 12 March 2013 11:55, Daniel Iwan wrote: >> >>> Just to add to that. >>> Further test shows that 2i searches aso suffer form the problem of not >>> showing all results durring active vnode transfer. >>> Is this a known issue with Riak 1.2.1? Has it been solved in 1.3? >>> >>> Anyone else experienced that? I guess attaching a node would trigger >>> that as well, maybe in less severe way. >>> Also I've read somewhere that you should attach one node at a time to >>> Riak cluster, and wait until vnode transfer completes. >>> I think it's no longer true in 1.2 since you have a plan that you >>> commit, but attaching a node and shuffling vnodes will cause problem >>> described >>> >>> What is the solution here? Waiting until vnode transfer finishes is not >>> acceptable (availability) and recent findings show it may take a while on >>> big clusters. >>> >>> Regards >>> Daniel >>> >>> >>> >>> On 11 March 2013 23:06, Daniel Iwan wrote: >>> >>>> I'm aware that listing keys is not for production. >>>> I'm using it mainly during testing, which started to be unreliable >>>> after changes described above. >>>> >>>> What I was not expecting at all was that some of the keys won't be >>>> listed. >>>> I'm not sure if that is stated in documentation to tell the truth. >>>> To me it looks like it should be called 'listSomeKeys' >>>> >>>> About alternatives. >>>> Some time ago I did comparison of listing and 2i and MapReduce and >>>> surprisingly listing was quickest. >>>> I'm not sure why it was happening. I did similar tests today and what I >>>> got is: >>>> >>>> 1000 keys, grouped with 2i into 10 equal groups, each value < 1kB >>>> Listing: >>>> - via listkeys 276ms >>>> - via keyindex $key: 255ms >>>> - via 2i (10 calls), total 2480ms >>>> >>>> So for that simple case 2i is 10 times slower. >>>> Further test shows that 100k keys (100 groups, 1000 each) gives query >>>> response between 250-5500ms. >>>> Not good. It's almost silly NOT to use listing keys. >>>> >>>> I may need to do that test on different hardware to compare. At the >>>> moment I'm using just one 7200rpm HDD for Riak db. >>>> >>>> Daniel >>>> >>>> >>> >> ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Using withoutFetch with DomainBucket
Hi Brian Thanks for your detailed response. Nothing detects whether there is a vclock or not. If there isn't one > provided (the value is `null` in Java), then one isn't sent to Riak - > it is not a requirement for a store operation for it to be present. If > an object exists when such a store is performed and allow_multi=true > for the bucket, then a sibling is created. > By detection I meant that when withoutFetch is used StoreObject would throw an exception if no RiakVClock annotation exist. I checked the code and it looks like getRiakVClock will not throw it and as you say VClock null will be used > > The .withoutFetch() method was added to the StoreObject as a requested > feature. It is meant for when you are storing an object that was > previously fetched from Riak and want to avoid doing another fetch. If > that previous fetch returned nothing (the key was not found) then the > vector clock will be null. > I guess another use case would be storing an object if I'm sure key does not exist i.e. I can guarantee key uniqueness. For that usage I will use VClock field with null. In that caes if sibblings are created, each of them will have vtag generated, and riak's java client VClock field is in fact vtag? According to what Sean Cribbs is saying, client sees only one vclock http://markmail.org/message/n65vsfsqrjpduhwd#query:+page:1+mid:j6ob5r3dtq3nzgxo+state:results Am I understanding that correctly? When talking about deleted keys ... unless you change the default > `delete_mode` in Riak's app.config, you're not usually going to get a > tombstone - they are reaped after 3s. Unless either you do a fetch > immediately following a delete, you're doing store operations without > vclocks with allow_multi=true for the bucket (which is basically > "doing it wrong") immediately after a delete and a sibling gets > created, or hit a very small window with multiple writers under heavy > load where the read/write cycle interleaves with a delete and a > tombstone sibling gets created. > I was thinking about that 3rd case actually, where interleaved delete happens, or case where fetch is executed and another node does delete. 3s is plenty of time. I can imagine it would be fairly easy to reproduce that case. > With that being said, yes, unless you set 'returnDeletedVClock(true)` > they are silently discarded by the Java client and not passed to the > Converter. If that has been set, the default JSONConverter will return > a new instance of whatever POJO is being used (if possible - if > there's not a default constructor it will throw an exception) and then > set a @RiakTombstone annotated boolean field to `true` if one exists. > It detects this by calling the .isDeleted() method of the returned > IRiakObject. > Understood. I think in case of custom converter I need to implement it myself in similar fashion i.e. using isDeleted() and setting it in my POJO when fromDomain() gets called. To handle deleted objects correctly (i.e. do not resurrect them) in in those corner cases mentioned above I assume I need to set returnDeletedVClock(true) and during sibbling resolving pick the one that has isDeleted equal true. At this point should I store it again with its vclock, or better abort store by using MutationWithCondition? I guess 2nd option is better since I want that key to remain 'deleted' and 1st option would resurrect it? When I have a moment I'm willing to make changes for DomainBucket to support withoutFetch unless you have it on your plate already. But first I need to learn how to use git... Daniel ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Keys not listed during vnode transfers
Maybe someone from Basho could shed some light on that issue? Regards Daniel On 12 March 2013 11:55, Daniel Iwan wrote: > Just to add to that. > Further test shows that 2i searches aso suffer form the problem of not > showing all results durring active vnode transfer. > Is this a known issue with Riak 1.2.1? Has it been solved in 1.3? > > Anyone else experienced that? I guess attaching a node would trigger that > as well, maybe in less severe way. > Also I've read somewhere that you should attach one node at a time to Riak > cluster, and wait until vnode transfer completes. > I think it's no longer true in 1.2 since you have a plan that you commit, > but attaching a node and shuffling vnodes will cause problem described > > What is the solution here? Waiting until vnode transfer finishes is not > acceptable (availability) and recent findings show it may take a while on > big clusters. > > Regards > Daniel > > > > On 11 March 2013 23:06, Daniel Iwan wrote: > >> I'm aware that listing keys is not for production. >> I'm using it mainly during testing, which started to be unreliable after >> changes described above. >> >> What I was not expecting at all was that some of the keys won't be listed. >> I'm not sure if that is stated in documentation to tell the truth. >> To me it looks like it should be called 'listSomeKeys' >> >> About alternatives. >> Some time ago I did comparison of listing and 2i and MapReduce and >> surprisingly listing was quickest. >> I'm not sure why it was happening. I did similar tests today and what I >> got is: >> >> 1000 keys, grouped with 2i into 10 equal groups, each value < 1kB >> Listing: >> - via listkeys 276ms >> - via keyindex $key: 255ms >> - via 2i (10 calls), total 2480ms >> >> So for that simple case 2i is 10 times slower. >> Further test shows that 100k keys (100 groups, 1000 each) gives query >> response between 250-5500ms. >> Not good. It's almost silly NOT to use listing keys. >> >> I may need to do that test on different hardware to compare. At the >> moment I'm using just one 7200rpm HDD for Riak db. >> >> Daniel >> >> > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Using withoutFetch with DomainBucket
Brian, Where I got lost was the fact that I was using custom Converter and I did not do anything with vclock passed into fromDomain(). That was undetected because at the same time I wasn't using withoutFetch, which I believe is the only moment where missing @RiakVClock annotation can be detected. Normally when JSONConverter is used missing @RiakVClock would also be detected. Could you confirm? Few additional, related questions: - if I use byte[] or VClock field and use withoutFetch() what is default value it should be set to (since it will be extracted via StoreObject)? - if I want to avoid overwriting deleted keys, I guess I need to set returnDeletedVClock as below, DomainBucketBuilder builder = DomainBucket.builder(bucket, Custom.class) builder.returnDeletedVClock(true); and then check isDeleted on sibblings and use ConditionalStoreMutation to return false id one of he sibblings has that flag set to true? I believe it needs to use VClock of deleted sibbling as well? Thanks Daniel The .withoutFetch() method isn't available when using the DomanBucket. > > As for the vector clock, when using .withoutFetch() the .execute() > method of StoreObject is going to extract the vector clock from the > POJO returned from your Mutation by looking for a VectorClock or > byte[] field that is annotated with @RiakVClock. It is then passed to > the Converter's .fromDomain() method as an argument. If you are > storing an object you previously fetched from Riak, that vector clock > and annotation needs to be there. > > The easiest way to implement that is: > 1. Have a VectorClock or byte[] field in your POJO annotated with > @RiakVClock > > 2. When you fetch, in the .toDomain() method of your Converter have > the line of code you noted. > > 3. When you store, the vector clock stored in that field will be > passed to the .fromDomain() method of your Converter. Make sure to > call the .withVClock(vclock) method of the RiakObjectBuilder or > explicitly set it in the IRiakObject being returned. > > - Roach > > > On Fri, Mar 8, 2013 at 3:31 PM, Daniel Iwan wrote: > > Somehow I cannot find a way to avoid pre-fetch during store operation > (Java > > client). > > I know in StoreObject there is withoutFetch method for that purpose but I > > cannot find corresponding method/property in DomainBucket or > > DomainBucketBuilder > > > > Am I missing something? > > > > Also on related note when withoutFetch is used I guess I need to provide > > annotated RiakVClock field and use something like: > > > > VClockUtil.setVClock(domainObject, riakObject.getVClock()); > > > > in my Converter. Is that right or is there better way to do it? > > > > > > I'm using Riak Java client 1.1.0 > > > > Thanks > > Daniel > > > > ___ > > riak-users mailing list > > riak-users@lists.basho.com > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Keys not listed during vnode transfers
Just to add to that. Further test shows that 2i searches aso suffer form the problem of not showing all results durring active vnode transfer. Is this a known issue with Riak 1.2.1? Has it been solved in 1.3? Anyone else experienced that? I guess attaching a node would trigger that as well, maybe in less severe way. Also I've read somewhere that you should attach one node at a time to Riak cluster, and wait until vnode transfer completes. I think it's no longer true in 1.2 since you have a plan that you commit, but attaching a node and shuffling vnodes will cause problem described What is the solution here? Waiting until vnode transfer finishes is not acceptable (availability) and recent findings show it may take a while on big clusters. Regards Daniel On 11 March 2013 23:06, Daniel Iwan wrote: > I'm aware that listing keys is not for production. > I'm using it mainly during testing, which started to be unreliable after > changes described above. > > What I was not expecting at all was that some of the keys won't be listed. > I'm not sure if that is stated in documentation to tell the truth. > To me it looks like it should be called 'listSomeKeys' > > About alternatives. > Some time ago I did comparison of listing and 2i and MapReduce and > surprisingly listing was quickest. > I'm not sure why it was happening. I did similar tests today and what I > got is: > > 1000 keys, grouped with 2i into 10 equal groups, each value < 1kB > Listing: > - via listkeys 276ms > - via keyindex $key: 255ms > - via 2i (10 calls), total 2480ms > > So for that simple case 2i is 10 times slower. > Further test shows that 100k keys (100 groups, 1000 each) gives query > response between 250-5500ms. > Not good. It's almost silly NOT to use listing keys. > > I may need to do that test on different hardware to compare. At the moment > I'm using just one 7200rpm HDD for Riak db. > > Daniel > > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Keys not listed during vnode transfers
I'm aware that listing keys is not for production. I'm using it mainly during testing, which started to be unreliable after changes described above. What I was not expecting at all was that some of the keys won't be listed. I'm not sure if that is stated in documentation to tell the truth. To me it looks like it should be called 'listSomeKeys' About alternatives. Some time ago I did comparison of listing and 2i and MapReduce and surprisingly listing was quickest. I'm not sure why it was happening. I did similar tests today and what I got is: 1000 keys, grouped with 2i into 10 equal groups, each value < 1kB Listing: - via listkeys 276ms - via keyindex $key: 255ms - via 2i (10 calls), total 2480ms So for that simple case 2i is 10 times slower. Further test shows that 100k keys (100 groups, 1000 each) gives query response between 250-5500ms. Not good. It's almost silly NOT to use listing keys. I may need to do that test on different hardware to compare. At the moment I'm using just one 7200rpm HDD for Riak db. Daniel ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Delete keys still in Riak db
Thanks Jeremy In our code I'm discarding ghost keys although I'm quite sure default settings in Java client should not return tombstones. I think my bug in the code contributed to the problems I've observed. I'm using DomainBucket and custom converter and in that case I think I need to explicitly set VClock on the POJO object. I need to investigate that. Thanks for the tips with map-reduce. For the tim being I think I'll stick to merging existing/deleted values hopefully in a reliable way. I just need to make sure I won't resurrect deleted keys. Daniel ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Using withoutFetch with DomainBucket
Somehow I cannot find a way to avoid pre-fetch during store operation (Java client). I know in StoreObject there is withoutFetch method for that purpose but I cannot find corresponding method/property in DomainBucket or DomainBucketBuilder Am I missing something? Also on related note when withoutFetch is used I guess I need to provide annotated RiakVClock field and use something like: VClockUtil.setVClock(domainObject, riakObject.getVClock()); in my Converter. Is that right or is there better way to do it? I'm using Riak Java client 1.1.0 Thanks Daniel ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Keys not listed during vnode transfers
Right after setting up the cluster of 3 nodes before riak finishes vnode transfers (ring 512) I store 3000 keys On some occasions instead of having 3000 keys listed I have 2999 or 2989. After transfer is finished we have 3000 keys visible via listing. Why is that happening and what's the best way to approach that? I assume it will happen also when new node attached to the cluster in production... Riak 1.2.1 Regards Daniel ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Delete keys still in Riak db
Hi
What worries me though is:
1) number of keys changes when i do listing, shouldn't that number be
constant?
If I do:
http://127.0.0.1:8098/buckets/TX/index/$key/0/zzz' | grep keys | awk
'{split($0,a,/,/); for (i in a) print a[i]}' | wc -l
I'm getting 12, 15 or 20 keys randomly. I believe all of them has been
deleted
2) Very often when I do get query e.g
curl '
http://127.0.0.1:8098/riak/TX/eb602276-15c0-2756-ee54-d04b77230afd:af79a2e4-874a-11e2-ba5c-bb1a97c2eacb:1362676559911
'
I will correctly get 'not found'
But subsequent list as above would still show that key as being in the
bucket.
I believe most of the time key would disappear but very often they just
seem to linger.
Is it possible that list somehow revives the keys?
D.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Delete keys still in Riak db
In our tests we are adding 3000 keys into 3-node Riak db right after nodes have joined. For each key one node reads it and modifies it and another node does the same but also deletes the key when it sees other change (key is no longer needed). After all keys are processed our test framework checks if all keys have been processed and deleted (bucket listing) Now, since deletion is not instant in Riak I believe here is 3 second delay, we are doing that check in a loop listing and counting number of keys in a bucket to check the progress. So far we've seen number of keys going down, eventually reaching zero. Sometimes the number of keys remained greater than zero, but after attempt to get a value we got null (Java client) which indicates key has been deleted. It's not perfect but it worked so far. When we changed to ring size 512 things got bit hairy. Our test fails and we have 24, 37 or 44 keys hanging around never going down to zero, but rather going up and down between those numbers. Test fails eventually. After that when I checked riak-admin transfer I could see 96 partitions to be transfer on every node. BTW Active Transfer section was showing nothing (I'm pretty sure there should be MB/s now) In the end partitions pending for transfer hit zero few minuter after test failed but that did not change anything. Listing keys in the bucket via http client showed between 24-44 keys. Manual reading via http client shows keys does not exist, and it looks like read-repair removes it, but somehow number of keys cannot reach 0 When those keys will be removed? Do I need to get every key to trigger read-repair? Any suggestions what to change to improve that behaviour? Bucket has allow_multi set to true, which I believe is necessary for that kind of usage. Riak db is in version 1.2.1 and backend LevelDB Regards Daniel ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak vnodes not available
Awesome stuff Shane! Thanks for sharing. We were thinking about the same approach so that will save us some work. Also we were planning to add some code to get/put some fixed value into Riak to check if that succeeds, but I'm not sure if that would work considering Riak's HA. I suspect that even if put will succeed key listing still may fail if not all nodes went up. BTW. we've noticed that after initial test setup code fails always on getBucket() with the stack as I posted before. That's quite surprising I must say. Regards Daniel ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak vnodes not available
Hi
In our test setup (3 nodes) we've changed number of vnodes from default 64
to 512.
We've noticed increased riak start up time (by 7 seconds) and failures in
our test framework due to that. Our test framework wipes Riak cluster and
recreates it, and then our application starts.
Application (each server) loads some keys on start via local riak node
using 127.0.0.1 interface ,
Most queries are on buckets with default N=3 and quorum 2.
Since we've changed to 512 vnodes, we get:
Unable to retrieve the riak object! com.google.protobuf.ByteString@1a292c9d,
com.google.protobuf.ByteString@3e62f0b1
com.basho.riak.pbc.RiakError: {insufficient_vnodes,0,need,2}
at com.basho.riak.pbc.RiakConnection.receive(RiakConnection.java:125)
at com.basho.riak.pbc.RiakClient.processFetchReply(RiakClient.java:282)
at com.basho.riak.pbc.RiakClient.fetch(RiakClient.java:275)
This is true both for Riak 1.2.1 and 1.3
So obviously nodes are not ready (or in the middle of vnode transfer?).
This happens not only during cluster set up but also after reboot.
What is the correct approach to make sure nodes are ready and Riak service
is fully up and running and can take request?
SInce our app depends on Riak we need to somehow make sure Riak is ready
before our services start.
Daniel
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak Java client 100% CPU
Hi
Thanks for that and also for building riak-client with all dependencies.
But I'm afraid S3 bucket is password protected or link expired since I'm
getting AccessDenied on that 1.1.0 jar
Daniel
On 14 February 2013 17:22, Brian Roach wrote:
> Daniel -
>
> Yes, sorry about that. This has been corrected in the current master
> on github and version 1.1.0 of the client will be released today.
> https://github.com/basho/riak-java-client/pull/212
>
> Thanks!
> Brian Roach
>
> On Thu, Feb 14, 2013 at 9:31 AM, Daniel Iwan
> wrote:
> > I see 100% CPU very regularly on one of the Riak client (v1.0.7) threads.
> > I think the place where it spins is connection reaper in
> RiakConnectionPool
> >
> > I looked at it briefly and it seems that when it finds first connection
> > using peek but that does not expired it can spin in tight while loop.
> > I guess second peek() should be outside if block?
> >
> > private synchronized void doStart() {
> > if (idleConnectionTTLNanos > 0) {
> > idleReaper.scheduleWithFixedDelay(new Runnable() {
> > public void run() {
> > RiakConnection c = available.peek();
> > while (c != null) {
> > long connIdleStartNanos =
> c.getIdleStartTimeNanos();
> > if (connIdleStartNanos + idleConnectionTTLNanos <
> > System.nanoTime()) {
> > if (c.getIdleStartTimeNanos() ==
> > connIdleStartNanos) {
> > // still a small window, but better than
> > locking
> > // the whole pool
> > boolean removed = available.remove(c);
> > if (removed) {
> > c.close();
> > permits.release();
> > }
> > }
> > c = available.peek();
> > }
> > }
> > }
> > }, idleConnectionTTLNanos, idleConnectionTTLNanos,
> > TimeUnit.NANOSECONDS);
> > }
> >
> > state = State.RUNNING;
> > }
> >
> >
> > Regards
> > Daniel Iwan
> >
> >
> > ___
> > riak-users mailing list
> > riak-users@lists.basho.com
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak Java client 100% CPU
I see 100% CPU very regularly on one of the Riak client (v1.0.7) threads.
I think the place where it spins is connection reaper in RiakConnectionPool
I looked at it briefly and it seems that when it finds first connection
using peek but that does not expired it can spin in tight while loop.
I guess second peek() should be outside if block?
private synchronized void doStart() {
if (idleConnectionTTLNanos > 0) {
idleReaper.scheduleWithFixedDelay(new Runnable() {
public void run() {
RiakConnection c = available.peek();
while (c != null) {
long connIdleStartNanos = c.getIdleStartTimeNanos();
if (connIdleStartNanos + idleConnectionTTLNanos <
System.nanoTime()) {
if (c.getIdleStartTimeNanos() ==
connIdleStartNanos) {
// still a small window, but better than
locking
// the whole pool
boolean removed = available.remove(c);
if (removed) {
c.close();
permits.release();
}
}
c = available.peek();
}
}
}
}, idleConnectionTTLNanos, idleConnectionTTLNanos,
TimeUnit.NANOSECONDS);
}
state = State.RUNNING;
}
Regards
Daniel Iwan
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak fails to start
Thanks Jon
Is there any timeframe for 1.3 release with maybe a glimpse of what
features we may expect?
Regards
Daniel
On 21 January 2013 14:42, Jon Meredith wrote:
> Thanks for reporting the issue and sorry you got bitten by it. We have
> committed a fix to master a couple of months ago and it will appear in the
> 1.3 release.
>
> Jon
>
> On Jan 21, 2013, at 3:58 AM, Daniel Iwan wrote:
>
> THe issue was that one of the ring snapshot files had size 0
>
> user@node1:~$ ls -la /var/lib/riak/ring/
> total 32
> drwxr-xr-x 2 riak riak 138 Jan 17 10:27 .
> drwxr-xr-x 7 riak riak 129 Jan 17 13:22 ..
> -rw-rw-r-- 1 riak riak 12584 Jan 15 17:19
> riak_core_ring.default.20130115171955
> -rw-rw-r-- 1 riak riak 12584 Jan 17 10:27
> riak_core_ring.default.20130117102756
> -rw-rw-r-- 1 riak riak 0 Jan 17 10:27
> riak_core_ring.default.20130117102757
>
> After deleting it riak process started correctly.
> I guess riak process should check if ring file is corrupted or 0 bytes
> long, delete/rename it automatically
> and use previous version?
>
> Regards
> Daniel
>
>
>
> On 17 January 2013 14:00, Daniel Iwan wrote:
>
>> One of our nodes fails to start
>>
>> $ sudo riak console
>> Attempting to restart script through sudo -H -u riak
>> Exec: /usr/lib/riak/erts-5.9.1/bin/erlexec -boot
>> /usr/lib/riak/releases/1.2.1/riak -embedded -config
>> /etc/riak/app.config -pa /usr/lib/riak/lib/basho-patches
>> -args_file /etc/riak/vm.args -- console
>> Root: /usr/lib/riak
>> Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:4:4] [async-threads:64]
>> [kernel-poll:true]
>>
>> ** /usr/lib/riak/lib/basho-patches/etop_txt.beam hides
>> /usr/lib/riak/lib/observer-1.1/ebin/etop_txt.beam
>> ** Found 1 name clashes in code paths
>> 13:22:47.424 [info] Application lager started on node 'riak@10.173.240.1'
>> 13:22:47.466 [error] CRASH REPORT Process <0.149.0> with 0 neighbours
>> exited with reason: bad argument in call to erlang:binary_to_term(<<>>) in
>> riak_core_ring_manager:read_ringfile/1 line 154 in gen_server2:init_it/6
>> line 384
>> /usr/lib/riak/lib/os_mon-2.2.9/priv/bin/memsup: Erlang has closed.
>>Erlang
>> has closed
>>
>> {"Kernel pid terminated",application_controller,"{application_
>> start_failure,riak_core,{shutdown,{riak_core_app,start,[normal,[]]}}}"}
>>
>> Crash dump was written to: /var/log/riak/erl_crash.dump
>> Kernel pid terminated (application_controller)
>> ({application_start_failure,riak_core,{shutdown,{riak_
>> core_app,start,[normal,[]]}}})
>>
>>
>> Which suggest that something happened to ring files?
>>
>> Any idea what and why it happened?
>>
>> Dump file available at:
>> http://www.filedropper.com/erlcrash
>>
>> Regards
>> Daniel
>>
>>
>>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak fails to start
THe issue was that one of the ring snapshot files had size 0
user@node1:~$ ls -la /var/lib/riak/ring/
total 32
drwxr-xr-x 2 riak riak 138 Jan 17 10:27 .
drwxr-xr-x 7 riak riak 129 Jan 17 13:22 ..
-rw-rw-r-- 1 riak riak 12584 Jan 15 17:19
riak_core_ring.default.20130115171955
-rw-rw-r-- 1 riak riak 12584 Jan 17 10:27
riak_core_ring.default.20130117102756
-rw-rw-r-- 1 riak riak 0 Jan 17 10:27
riak_core_ring.default.20130117102757
After deleting it riak process started correctly.
I guess riak process should check if ring file is corrupted or 0 bytes
long, delete/rename it automatically
and use previous version?
Regards
Daniel
On 17 January 2013 14:00, Daniel Iwan wrote:
> One of our nodes fails to start
>
> $ sudo riak console
> Attempting to restart script through sudo -H -u riak
> Exec: /usr/lib/riak/erts-5.9.1/bin/erlexec -boot
> /usr/lib/riak/releases/1.2.1/riak -embedded -config
> /etc/riak/app.config -pa /usr/lib/riak/lib/basho-patches
> -args_file /etc/riak/vm.args -- console
> Root: /usr/lib/riak
> Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:4:4] [async-threads:64]
> [kernel-poll:true]
>
> ** /usr/lib/riak/lib/basho-patches/etop_txt.beam hides
> /usr/lib/riak/lib/observer-1.1/ebin/etop_txt.beam
> ** Found 1 name clashes in code paths
> 13:22:47.424 [info] Application lager started on node 'riak@10.173.240.1'
> 13:22:47.466 [error] CRASH REPORT Process <0.149.0> with 0 neighbours
> exited with reason: bad argument in call to erlang:binary_to_term(<<>>) in
> riak_core_ring_manager:read_ringfile/1 line 154 in gen_server2:init_it/6
> line 384
> /usr/lib/riak/lib/os_mon-2.2.9/priv/bin/memsup: Erlang has closed.
>Erlang
> has closed
>
> {"Kernel pid terminated",application_controller,"{application_
> start_failure,riak_core,{shutdown,{riak_core_app,start,[normal,[]]}}}"}
>
> Crash dump was written to: /var/log/riak/erl_crash.dump
> Kernel pid terminated (application_controller) ({application_start_failure,
> riak_core,{shutdown,{riak_core_app,start,[normal,[]]}}})
>
>
> Which suggest that something happened to ring files?
>
> Any idea what and why it happened?
>
> Dump file available at:
> http://www.filedropper.com/erlcrash
>
> Regards
> Daniel
>
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak fails to start
One of our nodes fails to start
$ sudo riak console
Attempting to restart script through sudo -H -u riak
Exec: /usr/lib/riak/erts-5.9.1/bin/erlexec -boot
/usr/lib/riak/releases/1.2.1/riak -embedded -config
/etc/riak/app.config -pa /usr/lib/riak/lib/basho-patches
-args_file /etc/riak/vm.args -- console
Root: /usr/lib/riak
Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:4:4] [async-threads:64]
[kernel-poll:true]
** /usr/lib/riak/lib/basho-patches/etop_txt.beam hides
/usr/lib/riak/lib/observer-1.1/ebin/etop_txt.beam
** Found 1 name clashes in code paths
13:22:47.424 [info] Application lager started on node 'riak@10.173.240.1'
13:22:47.466 [error] CRASH REPORT Process <0.149.0> with 0 neighbours
exited with reason: bad argument in call to erlang:binary_to_term(<<>>) in
riak_core_ring_manager:read_ringfile/1 line 154 in gen_server2:init_it/6
line 384
/usr/lib/riak/lib/os_mon-2.2.9/priv/bin/memsup: Erlang has closed.
Erlang
has closed
{"Kernel pid terminated",application_controller,"{application_
start_failure,riak_core,{shutdown,{riak_core_app,start,[normal,[]]}}}"}
Crash dump was written to: /var/log/riak/erl_crash.dump
Kernel pid terminated (application_controller) ({application_start_failure,
riak_core,{shutdown,{riak_core_app,start,[normal,[]]}}})
Which suggest that something happened to ring files?
Any idea what and why it happened?
Dump file available at:
http://www.filedropper.com/erlcrash
Regards
Daniel
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Java riak-client 1.0.7 build with dependencies
Is there a repository/location where I could download 1.0.7 Java riak-client without building it myself? Thanks Daniel ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Listing keys again
I hope someone could shed some light on this issue
Part of our dev code is using Java RiakClient like this
KeySource fetched = getRiakClient().listKeys(bucket);
while (fetched.hasNext()) {
result.add(fetched.next().toStringUtf8());
}
where getRiakClient() returns instance of com.basho.riak.pbc.RiakClient
Aside of not using IRiakClient this lists keys in all keyspace and
filters them to get those falling into the bucket.
We are using Riak 1.2 and Riak client 1.0.5 and LevelDB
Now, I'm trying to see how this listing (which is done only when
machine boots) behaves when there is lots of keys in Riak
I load some keys into it - 100k per node (3 servers) and I'm rebooting
nodes to see what's the loading time.
Surprisingly it's always under 50ms per bucket (I have dozen of those)
Is there some automatic 2i creation to speed up listing keys or
something? Some improvements in 1.2?
Or is Riak is that quick and I should expect to fall apart on listing
keys once I reach certain number of keys in Riak?
Or is there other explanation for this?
It is quite a surprise especially that as far as I know Riak client
still have some features missing (native support of 2i?)
Regards
Daniel
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Node cannot join
Thanks Ryan!
I'll give it a try. Where I can find details about it, parameters options etc.?
D.
On 21 August 2012 14:36, Ryan Zezeski wrote:
> Daniel,
>
> Cluster operations (invoked by `riak-admin cluster`) are now multi-phase.
> Instead of calling an operation and it taking effect immediately you need to
> stage your changes, view your changes, and then commit them. E.g.
>
> riak-admin cluster join blah
>
> riak-admin cluster plan
>
> riak-admin cluster commit
>
> There is an `-f` option to keep the old behavior but we recommend against
> this. Staged clustering was put in place to keep users from hurting their
> clusters and to make multiple changes more efficient.
>
> -Z
>
> On Tue, Aug 21, 2012 at 9:28 AM, Daniel Iwan wrote:
>>
>> Hi
>>
>> In my setup everything worked fine until I upgraded to riak 1.2
>> (although this may be a coincidence)
>> Nodes are installed from scratch with changes only to db backend (I'm
>> using eLevelDB)
>> and names.
>> For some reason node cannot join to another.
>> What am I doing wrong?
>>
>> I'm using Ubuntu 10.04 but I've seen the same behaviour on Ubuntu 12.04
>> I may miss openssl dependency. I don't know if that matters
>>
>>
>> user@node2:~$ riak-admin cluster join riak@10.173.240.1
>> Attempting to restart script through sudo -H -u riak
>> Success: staged join request for 'riak@10.173.240.2' to
>> 'riak@10.173.240.1'
>>
>>
>> user@node1:~$ riak-admin member-status
>> Attempting to restart script through sudo -H -u riak
>> = Membership
>> ==
>> Status RingPendingNode
>>
>> ---
>> joining 0.0% -- 'riak@10.173.240.2'
>> valid 100.0% -- 'riak@10.173.240.1'
>>
>> ---
>>
>> Valid:1 / Leaving:0 / Exiting:0 / Joining:1 / Down:0
>> user@node1:~$
>> user@node1:~$ riak-admin transfers
>>
>> Attempting to restart script through sudo -H -u riak
>> No transfers active
>>
>> Active Transfers:
>>
>> user@node1:~$
>>
>> riak-admin status
>> Attempting to restart script through sudo -H -u riak
>> 1-minute stats for 'riak@10.173.240.1'
>> ---
>> vnode_gets : 0
>> vnode_gets_total : 0
>> vnode_puts : 0
>> vnode_puts_total : 0
>> vnode_index_reads : 0
>> vnode_index_reads_total : 0
>> vnode_index_writes : 0
>> vnode_index_writes_total : 0
>> vnode_index_writes_postings : 0
>> vnode_index_writes_postings_total : 0
>> vnode_index_deletes : 0
>> vnode_index_deletes_total : 0
>> vnode_index_deletes_postings : 0
>> vnode_index_deletes_postings_total : 0
>> node_gets : 0
>> node_gets_total : 0
>> node_get_fsm_siblings_mean : 0
>> node_get_fsm_siblings_median : 0
>> node_get_fsm_siblings_95 : 0
>> node_get_fsm_siblings_99 : 0
>> node_get_fsm_siblings_100 : 0
>> node_get_fsm_objsize_mean : 0
>> node_get_fsm_objsize_median : 0
>> node_get_fsm_objsize_95 : 0
>> node_get_fsm_objsize_99 : 0
>> node_get_fsm_objsize_100 : 0
>> node_get_fsm_time_mean : 0
>> node_get_fsm_time_median : 0
>> node_get_fsm_time_95 : 0
>> node_get_fsm_time_99 : 0
>> node_get_fsm_time_100 : 0
>> node_puts : 0
>> node_puts_total : 0
>> node_put_fsm_time_mean : 0
>> node_put_fsm_time_median : 0
>> node_put_fsm_time_95 : 0
>> node_put_fsm_time_99 : 0
>> node_put_fsm_time_100 : 0
>> read_repairs : 0
>> read_repairs_total : 0
>> coord_redirs_total : 0
>> executing_mappers : 0
>> precommit_fail : 0
>> postcommit_fail : 0
>> pbc_active : 0
>> pbc_connects : 0
>> pbc_connects_total : 0
>> cpu_nprocs : 340
>> cpu_avg1 : 182
>> cpu_avg5 : 136
>> cpu_avg15 : 92
>> mem_total : 4154609664
>> mem_allocated : 1475477504
>> disk : [{"/",54417668,2},
>> {"/dev",2024188,1},
>> {"/dev/shm",2028616,0},
>> {"/var/run",2028616,1},
>> {"/var/lock",2028616,0},
>> {"/lib/init/rw",2028616,0},
>> {"/var/lib/ureadahead/debugfs",54417668,2},
>> {"/media/ARRAY1",10739982336,1},
>> {"/boot",186663,22}]
>> nodename : &
Node cannot join
Hi
In my setup everything worked fine until I upgraded to riak 1.2
(although this may be a coincidence)
Nodes are installed from scratch with changes only to db backend (I'm
using eLevelDB)
and names.
For some reason node cannot join to another.
What am I doing wrong?
I'm using Ubuntu 10.04 but I've seen the same behaviour on Ubuntu 12.04
I may miss openssl dependency. I don't know if that matters
user@node2:~$ riak-admin cluster join riak@10.173.240.1
Attempting to restart script through sudo -H -u riak
Success: staged join request for 'riak@10.173.240.2' to 'riak@10.173.240.1'
user@node1:~$ riak-admin member-status
Attempting to restart script through sudo -H -u riak
= Membership ==
Status RingPendingNode
---
joining 0.0% -- 'riak@10.173.240.2'
valid 100.0% -- 'riak@10.173.240.1'
---
Valid:1 / Leaving:0 / Exiting:0 / Joining:1 / Down:0
user@node1:~$
user@node1:~$ riak-admin transfers
Attempting to restart script through sudo -H -u riak
No transfers active
Active Transfers:
user@node1:~$
riak-admin status
Attempting to restart script through sudo -H -u riak
1-minute stats for 'riak@10.173.240.1'
---
vnode_gets : 0
vnode_gets_total : 0
vnode_puts : 0
vnode_puts_total : 0
vnode_index_reads : 0
vnode_index_reads_total : 0
vnode_index_writes : 0
vnode_index_writes_total : 0
vnode_index_writes_postings : 0
vnode_index_writes_postings_total : 0
vnode_index_deletes : 0
vnode_index_deletes_total : 0
vnode_index_deletes_postings : 0
vnode_index_deletes_postings_total : 0
node_gets : 0
node_gets_total : 0
node_get_fsm_siblings_mean : 0
node_get_fsm_siblings_median : 0
node_get_fsm_siblings_95 : 0
node_get_fsm_siblings_99 : 0
node_get_fsm_siblings_100 : 0
node_get_fsm_objsize_mean : 0
node_get_fsm_objsize_median : 0
node_get_fsm_objsize_95 : 0
node_get_fsm_objsize_99 : 0
node_get_fsm_objsize_100 : 0
node_get_fsm_time_mean : 0
node_get_fsm_time_median : 0
node_get_fsm_time_95 : 0
node_get_fsm_time_99 : 0
node_get_fsm_time_100 : 0
node_puts : 0
node_puts_total : 0
node_put_fsm_time_mean : 0
node_put_fsm_time_median : 0
node_put_fsm_time_95 : 0
node_put_fsm_time_99 : 0
node_put_fsm_time_100 : 0
read_repairs : 0
read_repairs_total : 0
coord_redirs_total : 0
executing_mappers : 0
precommit_fail : 0
postcommit_fail : 0
pbc_active : 0
pbc_connects : 0
pbc_connects_total : 0
cpu_nprocs : 340
cpu_avg1 : 182
cpu_avg5 : 136
cpu_avg15 : 92
mem_total : 4154609664
mem_allocated : 1475477504
disk : [{"/",54417668,2},
{"/dev",2024188,1},
{"/dev/shm",2028616,0},
{"/var/run",2028616,1},
{"/var/lock",2028616,0},
{"/lib/init/rw",2028616,0},
{"/var/lib/ureadahead/debugfs",54417668,2},
{"/media/ARRAY1",10739982336,1},
{"/boot",186663,22}]
nodename : 'riak@10.173.240.1'
connected_nodes : ['riak@10.173.240.2']
sys_driver_version : <<"2.0">>
sys_global_heaps_size : 0
sys_heap_type : private
sys_logical_processors : 4
sys_otp_release : <<"R15B01">>
sys_process_count : 1392
sys_smp_support : true
sys_system_version : <<"Erlang R15B01 (erts-5.9.1) [source] [64-bit]
[smp:4:4] [async-threads:64] [kernel-poll:true]">>
sys_system_architecture : <<"x86_64-unknown-linux-gnu">>
sys_threads_enabled : true
sys_thread_pool_size : 64
sys_wordsize : 8
ring_members : ['riak@10.173.240.1','riak@10.173.240.2']
ring_num_partitions : 64
ring_ownership : <<"[{'riak@10.173.240.1',64}]">>
ring_creation_size : 64
storage_backend : riak_kv_eleveldb_backend
erlydtl_version : <<"0.7.0">>
riak_control_version : <<"1.2.0">>
cluster_info_version : <<"1.2.2">>
riak_api_version : <<"1.2.0">>
riak_search_version : <<"1.2.0">>
merge_index_version : <<"1.2.0">>
riak_kv_version : <<"1.2.0">>
riak_pipe_version : <<"1.2.0">>
riak_core_version : <<"1.2.0">>
lager_version : <<"1.2.0">>
syntax_tools_version : <<"1.6.8">>
compiler_version : <<"4.8.1">>
bitcask_version : <<"1.5.1">>
basho_stats_version : <<"1.0.2">>
luke_version : <<"0.2.5">>
webmachine_version : <<"1.9.2">>
mochiweb_version : <<"1.5.1">>
inets_version : <<"5.9">>
erlang_js_version : <<"1.2.0">>
runtime_tools_version : <<"1.8.8">>
os_mon_version : <<"2.2.9">>
riak_sysmon_version : <<"1.1.2">>
ssl_version : <<"5.0.1">>
public_key_version : <<"0.15">>
crypto_version : <<"2.1">>
sasl_version : <<"2.2.1">>
stdlib_version : <<"1.18.1">>
kernel_version : <<"2.15.1">>
memory_total : 26853928
memory_processes : 6744452
memory_processes_used : 6744438
memory_system : 20109476
memory_atom : 429569
memory_atom_used : 406875
memory_binary : 58504
memory_code : 9277444
memory_ets : 774080
D.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Java clientId
I've read somewhere some time ago that each client connecting to Riak cluster should have unique id to help with resolving conflicts. Is it still the case and if yes, what would be a recommended way of selecting such id? I just found in RawClient and in IRiakClient /** * If you don't set a client id explicitly at least call this to set one. It * generates the 4 byte ID and sets that Id on the client IE you *don't* * need to call setClientId() with the result of generate. * * @return the generated clientId for the client */ byte[] generateAndSetClientId() throws IOException; Would that be a way to go or is it now all automatically resolved via vector clocks? Daniel ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Non-standard ring size
Hello riak users I'm trying to get my head around the partitioning in Riak Quite recent thread was very helpful http://thread.gmane.org/gmane.comp.db.riak.user/6207/focus=6266 Let's say I install Riak on 3 nodes (initially), which will possibly grow to 10 or more. Default partition size is 64 and I store some data into bucket with n=2. Does it mean that both copies of keys that are in 64th partition would land on the same physical machine (node 1)? >From what I understood keys from 64th partition land in 1st partition as well, which in that case would be on node1 If the answer is Yes, would it be better if I set the max ring size to 72 which divides nicely by 3? Basho wiki states that ring size should be power of 2. Is there any performance impact of setting it to such a non-standard value? Or is there any better approach if I know that I will start with 3 machines? Apart from changing n to 3 of course. Thanks D. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
