Re: is it possible to use FQDN instead of IP in riak.conf for "listener.http.$name" field?
That could be a problem if your box(es) is(are) dual/multi homed. Be awares!
-Alexander
@siculars
http://siculars.posthaven.com
Sent from my iRotaryPhone
> On Jul 27, 2015, at 02:16, Toby Corkindale wrote:
>
> Hi Roman,
> I just set the IP to 0.0.0.0 as then it'll bind to whatever IPs are
> available, even if they change.
>
> Obviously you want to make sure your firewall isn't allowing random internet
> addresses into the Riak ports, though.
>
> -Toby
>
>> On Tue, 21 Jul 2015 at 06:48 ROMAN SHESTAKOV
>> wrote:
>>
>> Hello,
>>
>> is there any way to specify in riak.conf in field "listener.http.$name” FQDN
>> instead of IP address?
>>
>> >>>listener.http.$name This is the IP address and TCP port to which the
>> >>>Riak HTTP interface will bind. {"127.0.0.1",8098}<<<
>>
>> why is it required to use IP address instead of a host name? In my setting
>> RIAK_KV is deployed on dynamic VMs and IP could be changed if the VM is
>> launched on a different hypervisor.
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Deleted keys come back
Seconded. This makes your cluster so fresh, so clean for new tests. For node in nodes Stop node Do stuff (ie. delete data directory) Start node That general pattern is known as rolling restarts and is more or less how Basho recommends doing maintenance on a Riak cluster. Regards, -Alexander @siculars http://siculars.posthaven.com Sent from my iRotaryPhone > On Oct 7, 2015, at 09:19, Dmitri Zagidulin wrote: > > Also, if you're deleting things for unit tests, there's an easier way. > Instead of deleting the bucket object-by-object, you can just stop the node, > and clear the bitcask (or leveldb) data directory. (That's going to get rid > of all the data in the cluster, which is what you want to do for unit tests > anyways.) ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Java Riak client can't handle a Riak node failure?
Greetings and salutations, Vanessa. I am obliged to point out that running n_val of 3 (or more) is highly detrimental in a cluster size smaller than 5 nodes. Needless to say, it is not recommended. Let's talk about why that is the case for a moment. The default level of abstraction in Riak is the virtual node, or vnode. The vnode represents a segment of the ring (ring_size, configurable in steps of power of 2, default 64, min 8, max 1024.) The "ring" is the number line 0 - 2^160 which represents the output of the SHA hash. Riak achieves high availability through replication. Riak replicates data to vnodes. A hypothetical replica set may be, for example, set[vnode 1, vnode 10, vnode 20]. Note, I said vnodes. Not physical nodes. And therein lies the concern. Considering a default ring size of 64 and a default replica count of 3, the minimum recommended production deployment of a Riak cluster should be 5 due to the fact that in that circumstance every replica set combination is guaranteed to have each vnode on a distinct physical node. Anything less than that will certainly have some combinations of replica sets which have two of its copies on the same physical host. Note I said some combinations. Some fraction of node replica set combinations will have two of their copies allocated to one physical machine. You can see where I'm going with this. Firstly, performance will be negatively impacted when writing more than one copy of data to the same physical hardware, aka disk. But more importantly, you are negating Riak's high availability mechanic. If you lost any given physical node you would lose access to two copies of the set of data which had two replicas on that node. Riak is designed to withstand loss of any two physical nodes while maintaining access to 100% of your corpus assuming the fact that you are running the default settings and have deployed 5 nodes. Here is the rule of thumb that I recommend (me personally, not Basho) to folks looking to deploy clusters with less than 5 nodes: 1,2 nodes: n_val 1 3,4 nodes: n_val 2 5+ nodes: n_val 3 In summary, please consider reconfiguring your production deployment. Sincerely, Alexander @siculars http://siculars.posthaven.com Sent from my iRotaryPhone On Oct 7, 2015, at 19:56, Vanessa Williams wrote: Hi Dmitri, what would be the benefit of r=2, exactly? It isn't necessary to trigger read-repair, is it? If it's important I'd rather try it sooner than later... Regards, Vanessa On Wed, Oct 7, 2015 at 4:02 PM, Dmitri Zagidulin wrote: > Glad you sorted it out! > > (I do want to encourage you to bump your R setting to at least 2, though. > Run some tests -- I think you'll find that the difference in speed will not > be noticeable, but you do get a lot more data resilience with 2.) > > On Wed, Oct 7, 2015 at 6:24 PM, Vanessa Williams < > vanessa.willi...@thoughtwire.ca> wrote: > >> Hi Dmitri, well...we solved our problem to our satisfaction but it turned >> out to be something unexpected. >> >> The keys were two properties mentioned in a blog post on "configuring >> Riak’s oft-subtle behavioral characteristics": >> http://basho.com/posts/technical/riaks-config-behaviors-part-4/ >> >> notfound_ok= false >> basic_quorum=true >> >> The 2nd one just makes things a little faster, but the first one is the >> one whose default value of true was killing us. >> >> With r=1 and notfound_ok=true (default) the first node to respond, if it >> didn't find the requested key, the authoritative answer was "this key is >> not found". Not what we were expecting at all. >> >> With the changed settings, it will wait for a quorum of responses and >> only if *no one* finds the key will "not found" be returned. Perfect. >> (Without this setting it would wait for all responses, not ideal.) >> >> Now there is only one snag, which is that if the Riak node the client >> connects to goes down, there will be no communication and we have a >> problem. This is easily solvable with a load-balancer, though for >> complicated reasons we actually don't need to do that right now. It's just >> acceptable for us temporarily. Later, we'll get the load-balancer working >> and even that won't be a problem. >> >> I *think* we're ok now. Thanks for your help! >> >> Regards, >> Vanessa >> >> >> >> On Wed, Oct 7, 2015 at 9:33 AM, Dmitri Zagidulin >> wrote: >> >>> Yeah, definitely find out what the sysadmin's experience was, with the >>> load balancer. It could have just been a wrong configuration or something. >>> >>> And yes, that's the documentation page I recommend - >>> http://docs.basho.com/riak/latest/ops/advanced/configs/load-balancing-proxy/ >>> Just set up HAProxy, and point your Java clients to its IP. >>> >>> The drawbacks to load-balancing on the java client side (yes, the >>> cluster object) instead of a standalone load balancer like HAProxy, are the >>> following: >>> >>> 1) Adding node means code changes (or at very least, config file >>> changes) rolled out to all your clients
Re: Using Bucket Data Types slowed insert performance
Hi Dennis, It's a bit unclear what you are trying to do here. Are you 1. uploading the entire file and saving it to one key with the value being the file? Or are you 2. parsing the file and storing each row as a register in a map? Either of those approaches are not appropriate in Riak KV. For the first case I would point you to Riak S2 which is designed to manage large binary object storage. You can keep the large file as a single addressable entity and access it via Amazon S3 or Swift protocol. For the second case I would consider maintaining one key (map) per row in the file and have a register per column in the row. Or not use Riak data types (maps, sets, registers, flags and counters) and simply keep each row in the file as a KV in Riak either as a raw string or as a serialized json string. ETL'ing out of relational databases and into Riak is a very common use case and often implemented in the fashion I described. As Chris mentioned, soft upper bound on value size should be 1MB. I say soft because we won't enforce it although there are settings in the config that can be changed to enforce it (default 5MB warning, 50MB reject I believe.) Best, Alexander @siculars http://siculars.posthaven.com Sent from my iRotaryPhone > On Oct 20, 2015, at 10:22, Christopher Mancini wrote: > > Hi Dennis, > > I am not the most experienced, but what I do know is that a file that size > causes a great deal of network chatter because it has to handoff that data to > the other nodes in the network and will cause delays in Riak's ability to > send and confirm consistency across the ring. Typically we recommend that you > try to structure your objects to around 1mb or less to ensure consistent > performance. That max object size can vary of course based on your network / > server specs and configuration. > > I hope this helps. > > Chris > >> On Tue, Oct 20, 2015 at 8:18 AM Dennis Nicolay wrote: >> Hi, >> >> >> >> I’m using .net RiakClient 2.0 to insert a 44mb delimited file with 139k rows >> of data into riak. I switched to a map bucket data type with registers. >> It is taking about 3 times longer to insert into this bucket vs non data >> typed bucket. Any suggestions? >> >> >> >> Thanks in advance, >> >> Dennis >> >> ___ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Using Bucket Data Types slowed insert performance
Let's talk about Riak data types for a moment. Riak data types are collectively implementations of what academia refer to as CRDT's (convergent or conflict free replicated data types.) The key benefit a CRDT offers, over a traditional KV by contrast, is in automatic conflict resolution. The various CRDT's provided in Riak have specific conflict resolution strategies. This does not come for free. There is a computational cost associated with CRDT's. If your use case requires automated conflict resolution strategies than CRDT's are a good fit. Internally CRDT's rely on vector clocks (see DVV's in the documentation) to resolve conflict. Considering your ETL use case I'm going to presume that your data is immutable (I could very well be wrong here.) If your data is immutable I would consider simply using a KV and not paying the CRDT computational penalty (and possibly even the write once bucket.) The CRDT penalty you pay is obviously subjective to your use case, configuration, hw deployment etc. Hope that helps! -Alexander @siculars http://siculars.posthaven.com Sent from my iRotaryPhone > On Oct 20, 2015, at 12:39, Dennis Nicolay wrote: > > Hi Alexander, > > I’m parsing the file and storing each row with own key in a map datatype > bucket and each column is a register. > > Thanks, > Dennis > > From: Alexander Sicular [mailto:sicul...@gmail.com] > Sent: Tuesday, October 20, 2015 10:34 AM > To: Dennis Nicolay > Cc: Christopher Mancini; riak-users@lists.basho.com > Subject: Re: Using Bucket Data Types slowed insert performance > > Hi Dennis, > > It's a bit unclear what you are trying to do here. Are you 1. uploading the > entire file and saving it to one key with the value being the file? Or are > you 2. parsing the file and storing each row as a register in a map? > > Either of those approaches are not appropriate in Riak KV. For the first case > I would point you to Riak S2 which is designed to manage large binary object > storage. You can keep the large file as a single addressable entity and > access it via Amazon S3 or Swift protocol. For the second case I would > consider maintaining one key (map) per row in the file and have a register > per column in the row. Or not use Riak data types (maps, sets, registers, > flags and counters) and simply keep each row in the file as a KV in Riak > either as a raw string or as a serialized json string. ETL'ing out of > relational databases and into Riak is a very common use case and often > implemented in the fashion I described. > > As Chris mentioned, soft upper bound on value size should be 1MB. I say soft > because we won't enforce it although there are settings in the config that > can be changed to enforce it (default 5MB warning, 50MB reject I believe.) > > Best, > Alexander > > > @siculars > http://siculars.posthaven.com > > Sent from my iRotaryPhone > > On Oct 20, 2015, at 10:22, Christopher Mancini wrote: > > Hi Dennis, > > I am not the most experienced, but what I do know is that a file that size > causes a great deal of network chatter because it has to handoff that data to > the other nodes in the network and will cause delays in Riak's ability to > send and confirm consistency across the ring. Typically we recommend that you > try to structure your objects to around 1mb or less to ensure consistent > performance. That max object size can vary of course based on your network / > server specs and configuration. > > I hope this helps. > > Chris > > On Tue, Oct 20, 2015 at 8:18 AM Dennis Nicolay wrote: > Hi, > > I’m using .net RiakClient 2.0 to insert a 44mb delimited file with 139k rows > of data into riak. I switched to a map bucket data type with registers. It > is taking about 3 times longer to insert into this bucket vs non data typed > bucket. Any suggestions? > > Thanks in advance, > Dennis > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: cluster health check using riak-java-client
Besides just plainly writing a key, you could also do something like (pseudo
code):
Riak.put(canaryKey, pw=n_val){
If ok -> cool!
If borked -> sad face
}
The important bit is the pw (primary write) equals your replication value. This
means that all copies in the virtual node replica set need to go to virtual
nodes allocated to their primary physical machines. This is a way you can check
cluster status from the app level as in , is the cluster in some kind of borked
state.
-Alexander
@siculars
http://siculars.posthaven.com
Sent from my iRotaryPhone
> On Dec 8, 2015, at 14:13, David Byron wrote:
>
> I'm still curious what people think here. As I stare at this longer, I'd
> like to be able to call RiakNode.checkHealth(), but it's private.
>
> HealthMonitorTask.run that only calls checkHealth some of the time, so
> without the ability to call it directly, I think I'm getting a stale notion
> of health in circumstances like I outlined below -- when the last operation
> was successful, but the node has since gone down.
>
> Thanks for your input.
>
> -DB
>
>> On 12/2/15 10:25 PM, David Byron wrote:
>> I'm implementing a health check for a service of mine that uses riak.
>> I've seen this code from
>> https://github.com/basho/riak-java-client/issues/456:
>>
>> RiakCluster cluster = clientInstance.getRiakCluster();
>> List nodes = cluster.getNodes();
>> for (RiakNode node : nodes)
>> {
>> State state = node.getNodeState();
>> }
>>
>> and it's great. From what I can tell, it depends on some background
>> processing that keeps track of the state of the nodes. I did a quick
>> test though, and if I run 'riak stop' from the command line and then
>> this loop with no intervening operations, the nodes report RUNNING. Even
>> after some time passes (more than three minutes), still RUNNING.
>>
>> However, if I run do run an intervening operation (some actual query of
>> data for example) that fails, the nodes then report HEALTH_CHECKING.
>> Then, after 'riak start', the nodes report RUNNING again. I suppose
>> that's good.
>>
>> So, I'm trying to decide how to implement the health check. The above
>> loop doesn't seem to be enough, but do I really need to do something like:
>>
>> final RiakFuture future = cluster.execute(new PingOperation());
>>
>> try {
>> future.await();
>> future.get();
>> } catch (ExecutionException | InterruptedException e) {
>> // bad
>> }
>> // good
>>
>> Maybe it's sufficient to only do this if all the nodes report RUNNING? I
>> suppose there's always a small window in time where a node could report
>> bad, but via a ping I'd learn it was up...so I'm torn. Any suggestions
>> for whether pinging every time is correct, or there's something more
>> efficient (and safe)?
>>
>> Thanks for your help.
>>
>> -DB
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: riak nodes on vmware
Hi Travis, Beyond performance reasons, this architecture is a bad idea from an availability perspective. If you lose one physical machine you'll lose two segments of your Riak cluster. And that's generally "not a good thing." -Alexander @siculars http://siculars.posthaven.com Sent from my iRotaryPhone > On Dec 7, 2015, at 12:57, Travis Kirstine > wrote: > > We thinking of deploying multiple riak nodes on a few servers using vmware, 2 > vm with riak nodes per physical server. Is this a bad idea? > > > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: A put is not put in Riak
Patrice, since Riak is distributed it uses a quorum (default 2 when n_val is 3) to read your data. Since you're doing a post commit hook to read your data immediately after you write there may be delay in getting the new value from two of three copies written due simply to timing. So the rule of thumb is to do a w+r>n transaction where you ensure that you write/read with some total value more than your number of replicas. Read these posts (part four talks about RYOW): http://basho.com/posts/technical/understanding-riaks-configurable-behaviors-part-1/ http://basho.com/posts/technical/riaks-config-behaviors-part-2/ http://basho.com/posts/technical/riaks-config-behaviors-part-3/ http://basho.com/posts/technical/riaks-config-behaviors-part-4/ -Alexander @siculars http://siculars.posthaven.com Sent from my iRotaryPhone > On Dec 30, 2015, at 11:14, John Daily wrote: > > This smells like a clock problem. Are you confident all of your servers’ > clocks are set to the same time? > > Updates can be treacherous if clocks are not synchronized and multiple writes > to the same key occur in a period of time short enough to fit within the > clock skew. > > I wrote about this after a similar question arose here: > http://basho.com/posts/technical/clocks-are-bad-or-welcome-to-distributed-systems/ > > -John > >> On Dec 8, 2015, at 9:39 AM, Patrice Bruno wrote: >> >> Good-Evening everyone, >> >> I'm reading this maling for minths now wioth great interest, and now we are >> using Riak, I have a question to not say an issue that I can not figure out. >> >> We have migrated leveldb files from Riak 1.3.1 to Riak 2.1.1. >> The clusters has 6 Nodes and 6 computers. >> We are running now on CentOS Linux release 7.1.1503 (Core) (we were under >> Fedora when we are under Riak 1.3.1). >> >> We have kept the default Riak paramaters. We are not using 'bucket_type'. >> The only paramaters we have tuned are: >> erlang.distribution_buffer_size = 128MB >> erlang.async_threads = 128 >> >> My issue is on a Bucket without sibling (default settings) I'm making a put >> and when I do the get the value has not been changed, and old value stayed >> under the key. So, I have made a test: I've made a put of an objet A, I can >> see in the precommit hook that it's well the object A, but in the postcommit >> the object (argument passed) is an old one. >> I thought that I'm not sending the right object, but when I see it in >> precommit hook, I'm sure now that I send what I expect to send. >> >> It's not for all the objects, but it appears time to time under load. >> >> Do you already seen this issue ? >> Do you have an idea or something I can enhance/tune ? >> >> >> Help me mailing list, you are my only hope :-) (dixit Princess Leia). >> >> -- >> >> Best Reagrds, >> >> Patrice Bruno >> Kuantic >> --- >> >> >> ___ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Continuous HTTP POSTs to Riak
Hi Qiang, Check out RYOW semantics in this blog post, part 2 of a 4 part series , http://basho.com/posts/technical/riaks-config-behaviors-part-2/ . -Alexander @siculars http://siculars.posthaven.com Sent from my iRotaryPhone > On Mar 3, 2016, at 20:12, Qiang Cao wrote: > > Thanks, Dave! I'm aware of Riak's eventual consistency. I wonder how I can > get Riak work in the way as I need. I'm in a situation where I expect the > immediate GET reads the value I've just posted. And the GET request isn't > sent out until the POST is done. > > Thanks, > -Qiang > >> On Thu, Mar 3, 2016 at 6:53 PM, Dave King wrote: >> You need to google 'Eventual consistency'. >> >> - Peace >> Dave >> >> >>> On Thu, Mar 3, 2016 at 4:23 PM, Qiang Cao wrote: >>> Hi All, >>> >>> I use Riak as a KV store and send http requests to its REST API to update >>> entries. Instead of using curl, I use Apache HttpClient to send requests. >>> >>> One thing bothering me is that sometimes I find the entry isn't properly >>> updated after an HTTP POST/PUT. For example, if I do an HTTP GET on the >>> same key immediately after an HTTP POST, the chance is good that I get a >>> stale value. This happens when I continuously POST to a multi-node Riak, >>> and some of the posts target the same key. >>> >>> Also, I find I don't have this issue if I use a single-node Riak. So I >>> think something wasn't correct with my multi-node Riak cluster (I set >>> "allow_mult" to false). Any thoughts? >>> >>> Thanks in advance! >>> >>> Qiang >>> >>> ___ >>> riak-users mailing list >>> riak-users@lists.basho.com >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak Recap March 18, 2016
Hello All! Here’s the latest summary of the Riak users mailing list, latest client releases and job openings at Basho. ## Latest Releases - NodeJS Client 2.1.2 - includes improvements to closed connections handling such as with AWS ELB. Release notes [1] and code [2]. - Java Client 2.0.5 - brings a number of fixes including a fix to a DNS refresh issue and enhanced testing. Release notes [3] and maven repo [4]. ## Recently Answered - Magnus and Russell help Oleksiy clear up some confusion about the hidden $bucket index [5] - but only when using leveldb. Nevertheless, perhaps a $bucket query should not be accepted against a bitcask backend [6]. - Robert notes that when joining nodes to clusters they must have the same ring sizes [7]. - Agung is interested in working with riak_ensemble [8][9] and has a few questions about it. Carlos recommends taking a look at nkbase [10], which uses riak_ensemble, and Sargun goes into more detail [11]. ## Open Discussions - Fred and Magnus are working with Oleksiy to determine the root cause of inconsistent Solr search results [12]. - Bartek is looking for assistance on how to feed a mapreduce job with the results of a full text search [13] - Kinson is looking for guidance on implementing Basho Bench [14] - There is an open question regarding whether it’s possible to rebalance Riak KV without adding/removing nodes or making any hardware changes [15] - Luke thinks that Gerhard’s proposed architecture will ensure replicas are distributed to every node, but invites someone with more knowledge of the hashing method to chime in [16] - Robert is looking for guidance on choosing EC2 instance types for use with Riak/LevelDB [17] - Vitaly is helping troubleshoot timeouts in a deployment of Riak KV running on a cluster with 512 partitions [18] - There’s an open question about debugging a post commit hook for a bucket type in Riak 2.1.3 [19] - Christopher explained that when configuring multiple Riak clients in a cluster, you need to have each client use a different name. Chathuri has some follow-up questions about the configuration [20] - Vitaly and Chris are helping troubleshoot testing in a single node implementation of Riak KV [21] ## Jobs at Basho Interested in working on distributed computing related problems? Perhaps these open positions at Basho may be of interest: - Developer Advocate (London and US) [22] - Client Services Engineer (USA) [23] - Consulting Engineer (USA) [24] Enjoy your weekend! Alexander Sicular Solution Architect Basho Technologies @siculars [1] https://github.com/basho/riak-nodejs-client/blob/master/RELNOTES.md [2] https://github.com/basho/riak-nodejs-client/releases/tag/v2.1.2 [3] https://github.com/basho/riak-java-client/blob/develop/RELNOTES.md#205 [4] http://search.maven.org/#artifactdetails%7Ccom.basho.riak%7Criak-client%7C2.0.5%7Cjar [5] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-March/018175.html [6] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-March/018222.html [7] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-March/018189.html [8] https://github.com/basho/riak_ensemble [9] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-March/018191.html [10] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-March/018193.html [11] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-March/018194.html [12] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-March/018226.html [13] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-March/018227.html [14] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-January/018039.html [15] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-February/018074.html [16] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-February/018086.html [17] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-February/018079.html [18] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-February/018105.html [19] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-February/018110.html [20] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-February/018124.html [21] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-February/018139.html [22] http://bashojobs.theresumator.com/apply/nVY7Ea/Developer-Advocate-EMEA [23] http://bashojobs.theresumator.com/apply/0CTNKU/Client-Services-Engineer-Remote [24] http://bashojobs.theresumator.com/apply/lKF2vz/Consulting-Engineer-Remote ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak Recap March 18, 2016
H. Not that I know of and I hadn't heard of JugglingDB previously. What do you find interesting about it, Shailesh? On Monday, March 21, 2016, Shailesh Mangal wrote: > Thanks for the update Alex, Is there a plan to build JugglingDB > <https://www.npmjs.com/package/jugglingdb> adapter for RIAK? > > - Shailesh Mangal > > On 18-Mar-2016, at 10:29 AM, Alexander Sicular > wrote: > > Hello All! > > Here’s the latest summary of the Riak users mailing list, latest client > releases and job openings at Basho. > > ## Latest Releases > > >- NodeJS Client 2.1.2 - includes improvements to closed connections >handling such as with AWS ELB. Release notes [1] and code [2]. >- Java Client 2.0.5 - brings a number of fixes including a fix to a >DNS refresh issue and enhanced testing. Release notes [3] and maven repo >[4]. > > > ## Recently Answered > > >- Magnus and Russell help Oleksiy clear up some confusion about the >hidden $bucket index [5] - but only when using leveldb. Nevertheless, >perhaps a $bucket query should not be accepted against a bitcask backend >[6]. >- Robert notes that when joining nodes to clusters they must have the >same ring sizes [7]. >- Agung is interested in working with riak_ensemble [8][9] and has a >few questions about it. Carlos recommends taking a look at nkbase [10], >which uses riak_ensemble, and Sargun goes into more detail [11]. > > > ## Open Discussions > > >- Fred and Magnus are working with Oleksiy to determine the root cause >of inconsistent Solr search results [12]. >- Bartek is looking for assistance on how to feed a mapreduce job with >the results of a full text search [13] >- Kinson is looking for guidance on implementing Basho Bench [14] >- There is an open question regarding whether it’s possible to >rebalance Riak KV without adding/removing nodes or making any hardware >changes [15] >- Luke thinks that Gerhard’s proposed architecture will ensure >replicas are distributed to every node, but invites someone with more >knowledge of the hashing method to chime in [16] >- Robert is looking for guidance on choosing EC2 instance types for >use with Riak/LevelDB [17] >- Vitaly is helping troubleshoot timeouts in a deployment of Riak KV >running on a cluster with 512 partitions [18] >- There’s an open question about debugging a post commit hook for a >bucket type in Riak 2.1.3 [19] >- Christopher explained that when configuring multiple Riak clients in >a cluster, you need to have each client use a different name. Chathuri has >some follow-up questions about the configuration [20] >- Vitaly and Chris are helping troubleshoot testing in a single node >implementation of Riak KV [21] > > > ## Jobs at Basho > > Interested in working on distributed computing related problems? Perhaps > these open positions at Basho may be of interest: > >- Developer Advocate (London and US) [22] >- Client Services Engineer (USA) [23] >- Consulting Engineer (USA) [24] > > > > > Enjoy your weekend! > > Alexander Sicular > Solution Architect > Basho Technologies > @siculars > > > [1] https://github.com/basho/riak-nodejs-client/blob/master/RELNOTES.md > [2] https://github.com/basho/riak-nodejs-client/releases/tag/v2.1.2 > [3] https://github.com/basho/riak-java-client/blob/develop/RELNOTES.md#205 > [4] > http://search.maven.org/#artifactdetails%7Ccom.basho.riak%7Criak-client%7C2.0.5%7Cjar > [5] > http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-March/018175.html > [6] > http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-March/018222.html > [7] > http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-March/018189.html > [8] https://github.com/basho/riak_ensemble > [9] > http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-March/018191.html > [10] > http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-March/018193.html > [11] > http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-March/018194.html > [12] > http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-March/018226.html > [13] > http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-March/018227.html > [14] > http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-January/018039.html > [15] > http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-February/018074.html > [16] > http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-February/018086.html > [17] > http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016
Riak Recap March 18, 2016
Hi Shailesh, I hadn't heard of JugglingDB. Looks interesting. There is no riak adapter for it as far as I know. There is, however, a riak client library for nodejs[0]. So I imagine an adapter could be written. Looking through the examples in the read me it looks like solr would have to be used to implement certain features. How many folks out there would like to use something like juggling or waterline? -Alexander [0] https://github.com/basho/riak-nodejs-client On Wednesday, March 30, 2016, Shailesh Mangal > wrote: > Hi Alex, > > We are integrating with a framework that has its persistence built around > jugglingDB. Having RIAK plugin for JugglingDB, would come out real handy > for us to implement our RIAK with this framework. > > - Shailesh Mangal > > > On 21-Mar-2016, at 10:18 PM, Alexander Sicular wrote: > > H. Not that I know of and I hadn't heard of JugglingDB previously. > What do you find interesting about it, Shailesh? > > On Monday, March 21, 2016, Shailesh Mangal > wrote: > >> Thanks for the update Alex, Is there a plan to build JugglingDB >> <https://www.npmjs.com/package/jugglingdb> adapter for RIAK? >> >> - Shailesh Mangal >> >> On 18-Mar-2016, at 10:29 AM, Alexander Sicular >> wrote: >> >> Hello All! >> >> Here’s the latest summary of the Riak users mailing list, latest client >> releases and job openings at Basho. >> >> ## Latest Releases >> >> >>- NodeJS Client 2.1.2 - includes improvements to closed connections >>handling such as with AWS ELB. Release notes [1] and code [2]. >>- Java Client 2.0.5 - brings a number of fixes including a fix to a >>DNS refresh issue and enhanced testing. Release notes [3] and maven repo >>[4]. >> >> >> ## Recently Answered >> >> >>- Magnus and Russell help Oleksiy clear up some confusion about the >>hidden $bucket index [5] - but only when using leveldb. Nevertheless, >>perhaps a $bucket query should not be accepted against a bitcask backend >>[6]. >>- Robert notes that when joining nodes to clusters they must have the >>same ring sizes [7]. >>- Agung is interested in working with riak_ensemble [8][9] and has a >>few questions about it. Carlos recommends taking a look at nkbase [10], >>which uses riak_ensemble, and Sargun goes into more detail [11]. >> >> >> ## Open Discussions >> >> >>- Fred and Magnus are working with Oleksiy to determine the root >>cause of inconsistent Solr search results [12]. >>- Bartek is looking for assistance on how to feed a mapreduce job >>with the results of a full text search [13] >>- Kinson is looking for guidance on implementing Basho Bench [14] >>- There is an open question regarding whether it’s possible to >>rebalance Riak KV without adding/removing nodes or making any hardware >>changes [15] >>- Luke thinks that Gerhard’s proposed architecture will ensure >>replicas are distributed to every node, but invites someone with more >>knowledge of the hashing method to chime in [16] >>- Robert is looking for guidance on choosing EC2 instance types for >>use with Riak/LevelDB [17] >>- Vitaly is helping troubleshoot timeouts in a deployment of Riak KV >>running on a cluster with 512 partitions [18] >>- There’s an open question about debugging a post commit hook for a >>bucket type in Riak 2.1.3 [19] >>- Christopher explained that when configuring multiple Riak clients >>in a cluster, you need to have each client use a different name. Chathuri >>has some follow-up questions about the configuration [20] >>- Vitaly and Chris are helping troubleshoot testing in a single node >>implementation of Riak KV [21] >> >> >> ## Jobs at Basho >> >> Interested in working on distributed computing related problems? Perhaps >> these open positions at Basho may be of interest: >> >>- Developer Advocate (London and US) [22] >>- Client Services Engineer (USA) [23] >>- Consulting Engineer (USA) [24] >> >> >> >> >> Enjoy your weekend! >> >> Alexander Sicular >> Solution Architect >> Basho Technologies >> @siculars >> >> >> [1] https://github.com/basho/riak-nodejs-client/blob/master/RELNOTES.md >> [2] https://github.com/basho/riak-nodejs-client/releases/tag/v2.1.2 >> [3] >> https://github.com/basho/riak-java-client/blob/develop/RELNOTES.md#205 >> [4] >> http://search.maven.org/#artifactdet
Riak Recap April 1st, 2016
My Fellow Riak People! Here is your latest summary of the Riak users mailing list, a community highlight and a couple of openings at Basho. ## Community Highlights - Erlang Solutions announced the latest version of MongooseIM, vs. 1.6.2, which enables running all major functionality against a Riak backend [1]. ## Recently Answered - In response to a question on inconsistent search results, Magnus offers Oleksiy a lengthy response on zeroing out and rebuilding AAE trees [2]. With a followup from Oleksiy [3]. - A few answers from Mark and Vitally on John’s question about checking for when a riak service is available [4]. ## Open Discussions - Fred asks Anil for more details on his solr duplicate records issue [5]. - Luke is looking for more information from Joe on his indexing design question [6]. - Michael is looking for some guidance on properly sizing a Riak S2 cluster [7]. ## Jobs at Basho Interested in working on distributed computing related problems? Perhaps these open positions at Basho may be of interest: - Developer Advocate (London and US) [8] - Client Services Engineer (USA) [9] Are you working on something Riak related and would like to be highlighted? Send me a note and let me know what you’re up to. Weekend time! -Alexander Sicular @siculars [1] https://www.erlang-solutions.com/blog/mongooseim-1-6-2-is-out-time-to-upgrade.html [2] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-March/018235.html [3] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-March/018239.html [4] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-April/018246.html [5] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-March/018230.html [6] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-March/018241.html [7] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-March/018242.html [8] http://bashojobs.theresumator.com/apply/nVY7Ea/Developer-Advocate-EMEA [9] http://bashojobs.theresumator.com/apply/0CTNKU/Client-Services-Engineer-Remote ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: how to really delete keys from Riak
Hello Shifeng, Riak uses Bitcask as its default data backend, it may also use leveldb as a backend but must be configured to do so[0][1]. Both bitcask[2] and leveldb[3] do not reclaim disk space immediately when a delete is issued. Incidentally, neither do updates. Because Riak is non-destructive, aka append only, aka no in-place updates, mutation operations like updates or deletes are actually writes. For example, a 10KB object that is updated four times in some short interval will consume 50KB space (original 10KB + 4 x 10KB updates) until merge/compaction occurs. A delete is actually a special zero byte value write indicating a key is marked for deletion, aka a tombstone. Bitcask will reclaim disk space via merge operations which may be controlled by the user. Leveldb reclaims disk space via compaction operations. Architects and Developers should take into account their update/delete ratios and consider how those will affect disk space consumption and consider how deletes operate in distributed systems in general and Riak in specific from an application perspective. John Daily wrote a must read four part series on Riak's configurable behaviors that all Riak developers should be familiar with. Part 3 deals specifically with delete related considerations[4]. Thank you, Alexander [0] http://docs.basho.com/riak/kv/latest/setup/planning/backend/leveldb/#installing-eleveldb [1] http://docs.basho.com/riak/kv/latest/configuring/backend/ [2] http://docs.basho.com/riak/kv/latest/setup/planning/backend/bitcask/ [3] http://docs.basho.com/riak/kv/latest/setup/planning/backend/leveldb/ [4] http://basho.com/posts/technical/riaks-config-behaviors-part-3/ On Wed, Apr 20, 2016 at 8:17 AM, Daniel Abrahamsson wrote: > Hi, > > What riak version and what riak backend are you using? For leveldb you have > to take compaction into account, and for older bitcask versions there are > known issues with data not being properly deleted. > > //Daniel > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: how to really delete keys from Riak
Check out bitcask TTL, expiry. -Alexander @siculars http://siculars.posthaven.com Sent from my iRotaryPhone > On Apr 20, 2016, at 12:20, Shifeng Xiao wrote: > > Thanks for the information Alexander and Daniel, I appreciated it. > > We are using Riak2.0 and bitcask as backend, and we are writing the keys once > into bucket, keep it for a few hours, then delete it, never update the > existing data. > > >> On Wed, Apr 20, 2016 at 11:32 AM, Alexander Sicular >> wrote: >> Hello Shifeng, >> >> Riak uses Bitcask as its default data backend, it may also use leveldb >> as a backend but must be configured to do so[0][1]. Both bitcask[2] >> and leveldb[3] do not reclaim disk space immediately when a delete is >> issued. Incidentally, neither do updates. Because Riak is >> non-destructive, aka append only, aka no in-place updates, mutation >> operations like updates or deletes are actually writes. For example, a >> 10KB object that is updated four times in some short interval will >> consume 50KB space (original 10KB + 4 x 10KB updates) until >> merge/compaction occurs. A delete is actually a special zero byte >> value write indicating a key is marked for deletion, aka a tombstone. >> Bitcask will reclaim disk space via merge operations which may be >> controlled by the user. Leveldb reclaims disk space via compaction >> operations. >> >> Architects and Developers should take into account their update/delete >> ratios and consider how those will affect disk space consumption and >> consider how deletes operate in distributed systems in general and >> Riak in specific from an application perspective. John Daily wrote a >> must read four part series on Riak's configurable behaviors that all >> Riak developers should be familiar with. Part 3 deals specifically >> with delete related considerations[4]. >> >> Thank you, >> Alexander >> >> [0] >> http://docs.basho.com/riak/kv/latest/setup/planning/backend/leveldb/#installing-eleveldb >> [1] http://docs.basho.com/riak/kv/latest/configuring/backend/ >> [2] http://docs.basho.com/riak/kv/latest/setup/planning/backend/bitcask/ >> [3] http://docs.basho.com/riak/kv/latest/setup/planning/backend/leveldb/ >> [4] http://basho.com/posts/technical/riaks-config-behaviors-part-3/ >> >> On Wed, Apr 20, 2016 at 8:17 AM, Daniel Abrahamsson >> wrote: >> > Hi, >> > >> > What riak version and what riak backend are you using? For leveldb you have >> > to take compaction into account, and for older bitcask versions there are >> > known issues with data not being properly deleted. >> > >> > //Daniel >> > >> > ___ >> > riak-users mailing list >> > riak-users@lists.basho.com >> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> > > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak Recap April 22nd, 2016
Hello All, Here’s what’s been going on over the last few weeks. A bug fix release, a new product release, a number of talks, a bunch of questions and an open position. Hey you! Ya, you! We want to hear from you. Are you working on or speaking about something Riak related and would like to be highlighted? Send me a note, let me know what you’re up to and I’ll get your talk or blog post in the next Riak Recap. ## Announcements - Riak KV 2.1.4 has been released and is available for download [1]. This release contains fixes for the Riak init file Product Advisory and the leveldb segfault Product Advisory. Please review the Release Notes [2] for more details before upgrading. - Redis Add-on 1.1 is now available for Riak Enterprise [3]. Redis Add-on allows you to integrate Redis as a read-through or write-around cache into your data pipeline. ## Community Events - Seema Jethani, Director of Product, @seemaj, will be speaking at Data By the Bay in San Francisco on May 19th on working with time series data from the London Air Quality Network in Riak TS and other projects [4]. - Yours truly (me!) will be speaking in Dallas on May 5th about Riak TS architecture at the North Texas DAMA meetup [5]. Come say hello. - Basho Engineer Jason Voegele, @jvoegele, will be giving a talk at LambdaConf in Boulder, CO entitled “Dialyzer: Optimistic Type Checking for Erlang and Elixir” [6]. ## Recently Answered - Saran was able to fix his Solr not starting issue [7] by correcting the internal ip address in the configuration file [8]. - Because Riak is written in Erlang you are able to modify a number of config parameters without bouncing the VM. Nevertheless, there are a few parameters that can not be changed without a bump. Luke confirms Edgar’s concerns that, at the moment, changing an ip address requires a reboot of the Erlang virtual machine [9]. - Fasil is looking for ways to remotely configure bucket types [10]. Vitaly confirms that certain operations are only available via the command line and not in the API [11]. - Jim is experiencing a particular flavor of Solr timeouts [12] which Fred pinpoints to certain areas in the code and recommends making changes to hard coded timeouts [13]. Jim verifies the issue but is unhappy about making those changes, testing continues with a new JVM version [14]. - Alex was having some issues testing against his single node cluster [15]. Luke verified that the ring file needed to be reset [16]. - In case you were wondering, Riak S2 is the same as Riak CS [17]. Riak CS has been rebranded Riak S2. - Luke helps Satish work through some ulimit issues [18]. Riak noms fd’s… feed it! - Surajit found a broken link in the docs [19]. The documentation team always welcomes community feedback. If you see something - say something! [20]. - Jared is looking for enhancements to the .NET client [21]. Luke drops him an example and lets us know that enhancements are on the way in the next version [22]. - Shifeng is looking for details on how to delete and reclaim used disk in Riak [23]. Alexander (me!) answers with additional food for thought [24]. - Harjot has some requirements for his time series related project and is looking for some feedback on the Riak TS product roadmap [25]. Seema, Director of Product at Basho lets us know what’s coming up next [26]. If you have a project that needs certain features - let us know! ## Open Discussions - In this long running thread Fred drops some Erlang which gets the build times of individual partition hash trees to help debug Oleksiy’s inconsistent Solr search results [27]. - Alex is looking for advice regarding whether or not Riak TS would be better suited in certain use cases over Riak KV for his social network project [28]. - Fred asks Anil for more details on his solr duplicate records issue [29]. - Luke is looking for more information from Joe on his indexing design question [30]. - Michael is looking for some guidance on properly sizing a Riak S2 cluster [31]. ## Jobs at Basho Interested in working on distributed computing related problems? Perhaps these open positions at Basho may be of interest: - Client Services Engineer (USA) [32] Till next time, -Alexander Sicular Solution Architect, Basho @siculars [1] http://docs.basho.com/riak/kv/2.1.4/downloads/ [2] http://docs.basho.com/riak/kv/2.1.4/release-notes/ [3] http://docs.basho.com/riak/kv/2.1.4/add-ons/redis/redis-add-on-features/ [4] https://databythebay2016.sched.org/event/6ERC/know-the-air-you-are-breathing [5] https://www.eventbrite.com/e/north-texas-dama-chapter-meeting-may-2016-tickets-24381313164 [6] http://lambdaconf.us/#schedule [7] http://lists.basho.com/pipermail/riak-users_lists.basho.com
Re: Very slow acquisition time (99 percentile) while fast median times
I believe you should be looking for the get_fsm_objsize stats listed here: http://docs.basho.com/riak/kv/2.1.4/using/cluster-operations/inspecting-node/#get-fsm-objsize . Unless you are using consistent bucket types or write once bucket types. -Alexander Alexander Sicular Solutions Architect Basho Technologies 9175130679 @siculars On Mon, May 2, 2016 at 10:43 AM, Guillaume Boddaert < guilla...@lighthouse-analytics.co> wrote: > Here we go for a complete round of my hosts, all are objsize : 0 > > Here is a sample answer (headers only, that are followed by the full set > of JSON content) from the RIAK5 host > > HTTP/1.1 200 OK > X-Riak-Vclock: a85hYGBgzGDKBVI8xTxKnGbpn7QYuPafyWBKZMxjZXjyYfYFviwA > Vary: Accept-Encoding > Server: MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho) > Link: ; rel="up" > Last-Modified: Mon, 02 May 2016 15:40:20 GMT > ETag: "2l2QODpewyBZQFqDnyEy3F" > Date: Mon, 02 May 2016 15:40:20 GMT > Content-Type: application/json > Content-Length: 10722 > > Below the riak-admin status output. > > > admin@riak1:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put' > consistent_put_objsize_100 : 0 > consistent_put_objsize_95 : 0 > consistent_put_objsize_99 : 0 > consistent_put_objsize_mean : 0 > consistent_put_objsize_median : 0 > write_once_put_objsize_100 : 0 > write_once_put_objsize_95 : 0 > write_once_put_objsize_99 : 0 > write_once_put_objsize_mean : 0 > write_once_put_objsize_median : 0 > > admin@riak2:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put' > consistent_put_objsize_100 : 0 > consistent_put_objsize_95 : 0 > consistent_put_objsize_99 : 0 > consistent_put_objsize_mean : 0 > consistent_put_objsize_median : 0 > write_once_put_objsize_100 : 0 > write_once_put_objsize_95 : 0 > write_once_put_objsize_99 : 0 > write_once_put_objsize_mean : 0 > write_once_put_objsize_median : 0 > > admin@riak3:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put' > consistent_put_objsize_100 : 0 > consistent_put_objsize_95 : 0 > consistent_put_objsize_99 : 0 > consistent_put_objsize_mean : 0 > consistent_put_objsize_median : 0 > write_once_put_objsize_100 : 0 > write_once_put_objsize_95 : 0 > write_once_put_objsize_99 : 0 > write_once_put_objsize_mean : 0 > write_once_put_objsize_median : 0 > > admin@riak4:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put' > > consistent_put_objsize_100 : 0 > consistent_put_objsize_95 : 0 > consistent_put_objsize_99 : 0 > consistent_put_objsize_mean : 0 > consistent_put_objsize_median : 0 > write_once_put_objsize_100 : 0 > write_once_put_objsize_95 : 0 > write_once_put_objsize_99 : 0 > write_once_put_objsize_mean : 0 > write_once_put_objsize_median : 0 > > admin@riak5:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put' > > consistent_put_objsize_100 : 0 > consistent_put_objsize_95 : 0 > consistent_put_objsize_99 : 0 > consistent_put_objsize_mean : 0 > consistent_put_objsize_median : 0 > write_once_put_objsize_100 : 0 > write_once_put_objsize_95 : 0 > write_once_put_objsize_99 : 0 > write_once_put_objsize_mean : 0 > write_once_put_objsize_median : 0 > > On 02/05/2016 17:32, Luke Bakken wrote: > >> Could you please check the objsize stats on every Riak node? If they >> are all zero then ... >> -- >> Luke Bakken >> Engineer >> lbak...@basho.com >> >> >> On Mon, May 2, 2016 at 8:26 AM, Guillaume Boddaert >> wrote: >> >>> My clients are working through an haproxy box configured on round-robin. >>> >>> I've switched from PBC to HTTP to provide you this: >>> >>> May 2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53456 >>> [02/May/2016:15:24:12.390] riak_rest riak_rest_backend/riak2 6/0/1/54/61 >>> 503 >>> 222 - - 5/4/2/1/0 0/0 "PUT >>> >>> /types/activity_fr/buckets/twitter/keys/727126697699594240?dw=1&timeout=50&returnbody=true >>> HTTP/1.1" >>> May 2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53455 >>> [02/May/2016:15:24:12.339] riak_rest riak_rest_backend/riak3 >>> 57/0/0/54/111 >>> 503 222 - - 5/4/1/1/0 0/0 "PUT >>> >>> /types/activity_fr/buckets/twitter/keys/727126698291023872?dw=1&timeout=50&returnbody=true >>> HTTP/1.1" >>> May 2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53454 >>> [02/May/2016:15:24:12.285] riak_rest riak_rest_backend/riak5 >>> 63/0/0/106/169 >>> 503 222 - - 5/4/0/1/0 0/0 "PUT >>&g
Re: How to increase Riak write performance for sequential alpha-numeric keys
Hello, Leveldb is a sorted backend by key name (and bucket.) I don't know the details on exactly how it does the sort but there may be considerations there. Another question is why is your ring size 128 on a 3 node cluster? Ring size determines the number of virtual nodes in the cluster which are then equally allocated to physical nodes in the cluster. In your config, each node will have about 42 vnodes. With leveldb each vnode wants a minimum of 300MB ram (with performance increasing through 2.5GB ram.) There is only one configurable leveldb memory setting which defaults to allocating 70% system ram to leveldb. If you do the math you'll see you're way short. This memory issue might not have anything to do with alpha sorting but it will catch up with you eventually. -Alexander On Thursday, May 5, 2016, alexc155 wrote: > We're using Riak as a simple key value store and we're having write > performance problems which we think is due to the format of our keys which > we can't easily change because they're tied into different parts of the > business and systems. > > We're not trying to do anything complicated with Riak: No solr, secondary > indexes or map reducing - just simple keys to strings of around 10Kb of > JSON. > > We've got upwards of 3 billion records to store so we've opted for LevelDb > as the backend. > > It's a 3 node cluster running on 3 dedicated Ubuntu VMs each with 16 cpu > cores and 12GB memory backed by SSDs on a 10Gb network. > > Using basho bench we know that it's capable of speeds upwards of *5000 > rows per sec* when using randomised keys, but the problem comes when we > use our actual data. > > The keys are formatted using the following pattern: > > USC~1930~1~1~001 > USC~1930~1~1~002 > USC~1930~1~1~003 > USC~1930~1~1~004 > > Most of the long key stays the same with numbers at the end going up. (The > "~" are changeable - we can set them to whatever character. They're just > delimiters in the keys) > > Using these keys, write performance is a tenth of the speed at *400 rows > per sec*. > > We don't need to worry about different versions of the data so we've set > the following in our bucket type: > > "allow_mult": false > "last_write_wins": true > "DW": 0 > "n_val": 2 > "w": 1 > "r": 1 > "basic_quorum": false > > On the riak servers we've set the ulimit to: > > riak soft nofile 32768 > riak hard nofile 65536 > > and other settings like this: > > ring_size = 128 > protobuf.backlog = 1024 > anti_entropy = passive > > We're using the v2 .net client from basho to do the putting which runs in > an API on 3 machines. > > We've checked all the usual bottlenecks: CPU, memory, network IO, disk IO > and throttles on the Riak servers and windows API servers. > > As a kicker, if we reverse the keys e.g. > > 100~1~1~0391~CSU > > speed goes up to over *3000 rows*, but that's a dirty kludge. > > Can anyone explain why Riak doesn't like sequential alpha-numeric keys and > what we can change to improve performance? > > Thanks! > > -- > View this message in context: How to increase Riak write performance for > sequential alpha-numeric keys > <http://riak-users.197444.n3.nabble.com/How-to-increase-Riak-write-performance-for-sequential-alpha-numeric-keys-tp4034219.html> > Sent from the Riak Users mailing list archive > <http://riak-users.197444.n3.nabble.com/> at Nabble.com. > > -- Alexander Sicular Solutions Architect Basho Technologies 9175130679 @siculars ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak Recap, May 6th, 2016
Hello All, Just in time for a well deserved weekend, here’s what’s been going on since the last recap. Rejoice - Riak TS 1.3 has been open sourced! Such applause and much appreciation to all the engineers, documentors, testers and everyone else at Basho who worked to get this mega release out the door. Let’s not forget all the customers who had input in shaping the product roadmap, a hearty thank you to you as well. Thank you all! As always, we want to hear from you. If you are working on or speaking about something Riak related and would like to be highlighted - send me a note! Let me know what you’re up to and I’ll get your talk or blog post in the next Riak Recap. ## Announcements - Riak TS 1.3 has been released. This is the first open source release and we are all very excited to see what you do with it. There are a ton of new features so please check the release notes [0], read the announcement [1] from our CTO, Dave McCrory, or review my architecture post on Riak TS [2] to learn more. Then download [3] Riak TS and let us know what you think! ## Community Events - Craig Vitter will be speaking in NYC this Tuesday, May 10th, on Riak TS in a talk entitled “Easy Time Series Analysis with Riak TS, Python, Pandas & Jupyter” [4]. This will be a great introduction to working with Riak TS. - Alexander Sicular will be speaking in Dallas on May 13th about Riak KV and assorted friends, Solr, Redis and Spark at the Global Big Data Conference in Dallas [5]. That’s me, come say hello. - Seema Jethani, Director of Product, @seemaj, will be speaking at Data By the Bay in San Francisco on May 19th on working with time series data from the London Air Quality Network in Riak TS and other projects [6]. - Basho Engineer Jason Voegele, @jvoegele, will be giving a talk at LambdaConf in Boulder, CO entitled “Dialyzer: Optimistic Type Checking for Erlang and Elixir” [7]. ## Jobs at Basho Interested in working on distributed computing related problems or helping our community with theirs? Perhaps these open positions at Basho may be of interest: - Client Services Engineer (USA) [8]. - Developer Advocate (London and US) [9]. Basho is remote friendly! ## Recently Answered - Anil was looking for ways to index various Java data objects in Riak via Solr but Fred lets us know that Riak communicates with Solr over http and not SolrJ. You would either need to build your own custom extractor (which is possible and has been done) or serialize to a neutral format like JSON [10]. - Doug confirms some versioning ambiguities for David [11]. Riak is comprised of modules that get their own version numbers independent from Riak KV (or S2 or TS) version numbers. - Fred helps answer Fasil’s question on modifying the solrconfig.xml but it’s a bit involved [12]. Luke proposes a simpler solution - consider modifying the input data so as not to require modifying the config file [13]. - Fred provides a link to Riak’s code in github to help answer Alexander’s question on POST-ing to the Solr http end point [14]. - Magnus points Sanket in the right direction regarding his question on working with CRDT’s from the erlang client [15]. - Fred points out to Ricardo that Riak loves eating fd’s (file descriptors) … so bump up those ulimits! [16]. - Luke confirms that “not found’s” need to be accounted for when executing map/reduce operations. They will throw a “not_found” error which needs to be managed [17]. ## Open Discussions - Alex is looking for advice regarding whether or not Riak TS would be better suited in certain use cases over Riak KV for his social network project [18]. - Michael is looking for some guidance on properly sizing a Riak S2 cluster [19]. - Psterk is looking for help on using hadoops’s distcpy to copy files from Riak S2 to hadoop [20][21]. - Basho engineer and resident CRDT expert, Russell Brown, is soliciting feedback [22] on Riak DT [23], the module that provides CRDT cpabilities in Riak. - A number of folks are working with Guillaume to get to the bottom of slow 99th percentile performance in a lengthy thread that touches on levelDB and Solr based enhancements [24]. - Johnny is looking for help working with credentials in Riak S2 [25]. - A number of folks contribute to a great thread helping Alex on his question about write performance when writing sequential alpha-numeric keys [26]. There is a sidebar on CRDT’s vs write_once buckets in there as well [27]. Till next time, -Alexander Sicular Solution Architect, Basho @siculars [0] http://docs.basho.com/riak/ts/1.3.0/releasenotes/ [1] http://basho.com/posts/business/riak-ts-1-3-is-now-open-source-what-is-here-and-what-is-coming/ [2] http://basho.com/posts/technical/time-series-the-new-s
Re: Schemas, is worth to store values in SOLR?
Those are exactly the two options and opinions vary generally based on use case. Storing the data not only take up more space but also more io which makes things slower not only on read time , but more crucially , at write time. Often people will take a hybrid approach and store certain elements like , say , for blog posts , the author , publish date and title fields. Yet they will leave the body out of the solr index. That way you could quickly generate lists of posts by title and only fetch the body when the post is clicked through. What is your use case? Best, Alexander On Wednesday, May 11, 2016, Alex De la rosa wrote: > Hi all, > > When creating a SOLR schema for Riak Search, we can chose to store or not > the data we are indexing, for example: > > > > I know that the point to have the value stored is to be able to get it > returned automatically when doing a search query... that implies using more > disc to store data that maybe never would be searched and making the return > slower as more bytes are required to get the data. > > Would it be better to just index data but not store the values, returning > only Riak IDs (_yz_id) and then doing a multi-get in the client/API to > fetch the objects for the final response? > > Or would it be better to store the values in SOLR so they will be already > fetched when searching? > > What would give better performance or more sense in terms of disc space on > an application that normally you won't be using much searching (all data is > more or less discoverable without searching using GETs) > > Thanks and Best Regards, > Alex > -- Alexander Sicular Solutions Architect Basho Technologies 9175130679 @siculars ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Schemas, is worth to store values in SOLR?
That's a great use case cause it's not ad hoc (worst case). Your pre compute/cache solution will work whichever approach you take. Then the question just becomes space vs compute. On Wednesday, May 11, 2016, Alex De la rosa wrote: > My use case for searching is mainly for internal purposes, rankings and > statistics (all that data is pre-compiled and stored into final objects for > the app to display)... so I think is best to not store anything in SOLR and > just fetch keys to compile the data when required. > > Thanks, > Alex > > On Wed, May 11, 2016 at 10:40 PM, Alexander Sicular > wrote: > >> Those are exactly the two options and opinions vary generally based on >> use case. Storing the data not only take up more space but also more io >> which makes things slower not only on read time , but more crucially , at >> write time. >> >> Often people will take a hybrid approach and store certain elements like >> , say , for blog posts , the author , publish date and title fields. Yet >> they will leave the body out of the solr index. That way you could quickly >> generate lists of posts by title and only fetch the body when the post is >> clicked through. >> >> What is your use case? >> >> Best, >> Alexander >> >> On Wednesday, May 11, 2016, Alex De la rosa > > wrote: >> >>> Hi all, >>> >>> When creating a SOLR schema for Riak Search, we can chose to store or >>> not the data we are indexing, for example: >>> >>> >>> >>> I know that the point to have the value stored is to be able to get it >>> returned automatically when doing a search query... that implies using more >>> disc to store data that maybe never would be searched and making the return >>> slower as more bytes are required to get the data. >>> >>> Would it be better to just index data but not store the values, >>> returning only Riak IDs (_yz_id) and then doing a multi-get in the >>> client/API to fetch the objects for the final response? >>> >>> Or would it be better to store the values in SOLR so they will be >>> already fetched when searching? >>> >>> What would give better performance or more sense in terms of disc space >>> on an application that normally you won't be using much searching (all data >>> is more or less discoverable without searching using GETs) >>> >>> Thanks and Best Regards, >>> Alex >>> >> >> >> -- >> >> >> Alexander Sicular >> Solutions Architect >> Basho Technologies >> 9175130679 >> @siculars >> >> > -- Alexander Sicular Solutions Architect Basho Technologies 9175130679 @siculars ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak Recap, May 20th, 2016
Hello All, Just in time for the weekend we bring you a rundown of announcements, jobs, some code and, of course, the mailing list review. As always, we want to hear from you. If you’re working on or speaking about something Riak related and would like to be highlighted - send me a note! Let me know what you’re up to and I’ll get your talk or blog post in the next Riak Recap. ## Announcements - Two client library updates this week, PHP and .NET. Check em out: - Riak PHP Client v 3.1.0 adds access to Riak TS via HTTP [0] - Riak .NET Client v 2.3.0 brings Riak TS and external load balancing support [1] - A Riak TS AMI for AWS is now available [2]. ## Community Code and Walkthroughs - Craig Vitter put some code out accompanying his talk last week in NYC on Riak TS. The code walks through using Riak TS with python and jupyter [3]. - Seema Jethani also drops some code with a talk she gave this week at Data By the Bay in San Francisco on working with time series data from the London Air Quality Network in Riak TS and other projects [4]. - Stephen Etheridge shows us how to get started with Spark and Riak TS [5]. - Susan Lee introduces us to Riak TS using Stephens code [6]. ##Community Events - Basho Engineer Jason Voegele, @jvoegele, will be giving a talk at LambdaConf in Boulder, CO entitled “Dialyzer: Optimistic Type Checking for Erlang and Elixir” [7]. ## Jobs at Basho Interested in working on distributed computing related problems or helping our community with theirs? Perhaps these open positions at Basho may be of interest: - Client Services Engineer (USA) [8]. - Developer Advocate (London and US) [9]. Basho is remote friendly! ## Recently Answered - Alex gets help modifying the default Solr schema [10]. - Luke goes above and beyond helping Sanket debug his AWS environment in order to get pre-commit hooks working [11]. If you’re looking to do something similar, instructions are here [12]. - Alex and I have a conversation about Solr schema design. To store or not to store. [13]. - Luke confirms Basho is working on security and drop/truncate table features in upcoming releases [14]. - Daniel helps David grok deleting an object in Riak [15]. Welcome aboard, David! - Kaz helps a different David by pointing out two key Riak KV configuration options when setting up KV for Riak CS [16]. - Vitaly helps Alex with Solr indexing [17] and Drew highlights specific sections of the default Solr schema shipped with Riak KV [18]. - Fred points out to Alex that although you can query Solr directly, there are considerations [19]. John queries Solr directly after caching a copy of the cluster plan every so often [20]. - Vitaly shows Alex how to configure Solr for case insensitive indexing [21]. - Magnus confirms that Basho is working on making Riak compatible with Erlang OTP 18 [22] and highlights a github issue and pull request that help Humberto with his specific compilation issue on OSX. - Luke confirms to one Outback Dingo that Riak TS is not supported on FreeBSD and support for Riak KV on FreeBSD will be discontinued [23]. - Serena is seeking out a Debian Jesse package. Brett assures us a package is on the way [24]. ## Open Discussions - Vladimir asks the list for help with the following warning “perflists do not satisfy the n_val” [25]. - Alex is looking for advice regarding whether or not Riak TS would be better suited in certain use cases over Riak KV for his social network project [26]. - Michael is looking for some guidance on properly sizing a Riak S2 cluster [27]. - Psterk is looking for help on using hadoops’s distcpy to copy files from Riak S2 to hadoop [28][29]. - The conversation around Riak DT [30] continues with Sargun offering to contribute some enhancements [31]. In case you’re wondering, yes, contributions will be highlighted. - Johnny is looking for help working with credentials in Riak S2 [32]. - A number of folks contribute to a great thread helping Alex on his question about write performance when writing sequential alpha-numeric keys [33]. There is a sidebar on CRDT’s vs write_once buckets in there as well [34]. Have a great weekend, -Alexander Sicular Solution Architect, Basho @siculars [0] https://github.com/basho/riak-php-client/releases/tag/v3.1.0 [1] https://github.com/basho/riak-dotnet-client/blob/master/RELNOTES.md [2] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-May/018401.html [3] https://github.com/cvitter/Easy-Time-Series-Analysis-with-Riak-TS [4] https://github.com/svjethani/airquality/blob/master/AirQuality1Q2016.ipynb [5] https://github.com/datalemming/rts12-sc-demo [6] https://www.youtube.com/watch?v=qZ-T4eEdC_g
Re: Riak Cluster Behavior - Clarification
Hi Vikram, If you're using the defaults, two of copies may be on the same machine. When using the default values (ring_size=64, n_val=3) you are not guaranteed copies on distinct physical machines. Implement a back-off retry design pattern. aka, fail once, try again with r=1. Also, a read will trigger a read repair operation which will then copy your data n_val times to surviving members of the cluster. Have you tried that? -Alexander Read these blog posts for more info: http://basho.com/posts/technical/understanding-riaks-configurable-behaviors-part-1/ http://basho.com/posts/technical/riaks-config-behaviors-part-2/ http://basho.com/posts/technical/riaks-config-behaviors-part-3/ http://basho.com/posts/technical/riaks-config-behaviors-part-4/ On Tue, May 24, 2016 at 3:08 PM, Vikram Lalit wrote: > It's returning no object at all for the relevant key. That too is random - > every few calls it returns but then it doesn't. > > On May 24, 2016 4:06 PM, "Sargun Dhillon" wrote: >> >> What do you mean it's not returning? It's returning stale data? Or >> it's erroring? >> >> On Tue, May 24, 2016 at 7:34 AM, Vikram Lalit >> wrote: >> > Hi - I'd appreciate if someone can opine on the below behavior of Riak >> > that >> > I am observing... is that expected, or something wrong in my set-up / >> > understanding? >> > >> > To summarize, I have a 3-node Riak cluster (separate EC2 AWS instances) >> > with >> > a separate chat server connecting to them. When I write data on the Riak >> > nodes, the process is successful and I can read all data correctly. >> > However, >> > as part of my testing, if I deliberately bring down one node (and then >> > remove it from the cluster using riak-admin cluster force-remove / plan >> > / >> > commit), the client API is not able to fetch all the written data. In >> > fact, >> > there is an alternation of success and failure which happens rather >> > randomly. >> > >> > My initial suspicion was that it would be happening only during the time >> > the >> > rebalancing is occurring (i.e. riak-admin ring-status is not fully >> > settled) >> > but I've seen this sporadic behavior post the same too. >> > >> > Does this have to do with the n and r values for the cluster and given >> > that >> > 1 node is down, the cluster does not succeed in returning results >> > reliably? >> > Also, does this mean that during the time a cluster is being rebalanced >> > (even incl. addition of new nodes), the results could be arbitrary - >> > that >> > doesn't sound correct to me? >> > >> > Appreciate if someone can throw some light here? Also, the HTTP API >> > calls to >> > retrieve and set the n / r / w values for a specific bucket - couldn't >> > locate the same! >> > >> > Thanks much! >> > Vikram >> > >> > >> > ___ >> > riak-users mailing list >> > riak-users@lists.basho.com >> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> > > > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Very slow acquisition time (99 percentile) while fast median times
urself. Otherwise, Basho may be releasing prebuilt patches of
> eleveldb 2.0.18 in the near future. But no date is currently set.
>
> Matthew
>
> On May 3, 2016, at 10:50 AM, Luke Bakken <
> lbak...@basho.com
> > wrote:
>
> Guillaume -
>
> You said earlier "My data are stored on an openstack volume that
> support up to 3000IOPS". There is a likelihood that your write load is
> exceeding the capacity of your virtual environment, especially if some
> Riak nodes are sharing physical disk or server infrastructure.
>
> Some suggestions:
>
> * If you're not using Riak Search, set "search = off" in riak.conf
>
> * Be sure to carefully read and apply all tunings:
> http://docs.basho.com/riak/kv/2.1.4/using/performance/
>
> * You may wish to increase the memory dedicated to leveldb:
> http://docs.basho.com/riak/kv/2.1.4/configuring/backend/#leveldb
>
> --
> Luke Bakken
> Engineer
> lbak...@basho.com
>
>
> On Tue, May 3, 2016 at 7:33 AM, Guillaume Boddaert
>
> wrote:
>
> Hi,
>
> Sorry for the delay, I've spent a lot of time trying to understand if the
> problem was elsewhere.
> I've simplified my infrastructure and got a simple layout that don't rely
> anymore on loadbalancer and also corrected some minor performance issue on
> my workers.
>
> At the moment, i have up to 32 workers that are calling riak for writes,
> each of them are set to :
> w=1
> dw=0
> timeout=1000
> using protobuf
> a timeouted attempt is rerun 180s later
>
> From my application server perspective, 23% of the calls are rejected by
> timeout (75446 tries, 57564 success, 17578 timeout).
>
> Here is a sample riak-admin stat for one of my 5 hosts:
>
> node_put_fsm_time_100 : 999331
> node_put_fsm_time_95 : 773682
> node_put_fsm_time_99 : 959444
> node_put_fsm_time_mean : 156242
> node_put_fsm_time_median : 20235
> vnode_put_fsm_time_100 : 5267527
> vnode_put_fsm_time_95 : 2437457
> vnode_put_fsm_time_99 : 4819538
> vnode_put_fsm_time_mean : 175567
> vnode_put_fsm_time_median : 6928
>
> I am using leveldb, so i can't tune bitcask backend as suggested.
>
> I've changed the vmdirty settings and enabled them:
> admin@riak1:~$ sudo sysctl -a | grep dirtyvm.dirty_background_ratio = 0
> vm.dirty_background_bytes = 209715200
> vm.dirty_ratio = 40
> vm.dirty_bytes = 0
> vm.dirty_writeback_centisecs = 100
> vm.dirty_expire_centisecs = 200
>
> I've seen less idle time between writes, iostat is showing near constant
> writes between 20 and 500 kb/s, with some surges around 4000 kb/s. That's
> better, but not that great.
>
> Here is the current configuration for my "activity_fr" bucket type and
> "tweet" bucket:
>
>
> admin@riak1:~$ http localhost:8098/types/activity_fr/props
> HTTP/1.1 200 OK
> Content-Encoding: gzip
> Content-Length: 314
> Content-Type: application/json
> Date: Tue, 03 May 2016 14:30:21 GMT
> Server: MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho)
> Vary: Accept-Encoding
> {
>"props": {
>"active": true,
>"allow_mult": false,
>"basic_quorum": false,
>"big_vclock": 50,
>"chash_keyfun": {
>"fun": "chash_std_keyfun",
>"mod": "riak_core_util"
>},
>"claimant":
>
> "r...@riak2.lighthouse-analytics.co"
> ,
>"dvv_enabled": false,
>"dw": "quorum",
>"last_write_wins": true,
>"linkfun": {
>"fun": "mapreduce_linkfun",
>"mod": "riak_kv_wm_link_walker"
>},
>"n_val": 3,
>"notfound_ok": true,
>"old_vclock": 86400,
>"postcommit": [],
>"pr": 0,
>"precommit": [],
>"pw": 0,
>"r": "quorum",
>"rw": "quorum",
>"search_index": "activity_fr.20160422104506",
>"small_vclock": 50,
>"w": "quorum",
>"young_vclock": 20
>}
> }
>
> admin@riak1:~$ http localhost:8098/types/activity_fr/buckets/tweet/props
> HTTP/1.1 200 OK
> Content-Encoding: gzip
> Content-Length: 322
> Content-Type: application/json
> Date: Tue, 03 May 2016 14:30:02 GMT
> Server: MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho)
> Vary: Accept-Encoding
>
> {
>"prop
Re: Riak not storing data in a round-robin manner across instances
There is a reason Basho's minimum production deployment recommendation is 5 machines. It is to ensure that, when operating with default settings, each replica of any key is stored on distinct physical nodes. It has to do with the allocation of virtual nodes to physical machines and replica sets. Replica sets are sets of virtual nodes, not physical nodes. When using 3 or 4 machines in your cluster reduce your n_val to 2. Cheers, Alexander On Friday, May 27, 2016, DeadZen wrote: > np, number of nodes and ring size play a lot into that. > as does your r,w settings. > might be fun to create a visualization one day ;) > > On Friday, May 27, 2016, Vikram Lalit > wrote: > >> Ok got it thanks. >> On May 27, 2016 4:02 PM, "DeadZen" wrote: >> >>> reiterating my last email >>> >>> > theres no guarantee vnodes are assigned to unique servers.. >>> >>> referencing the docs >>> >>> > Nodes *attempt* to claim their partitions at intervals around the >>> ring such that there is an even distribution amongst the member nodes and >>> that no node is responsible for more than one replica of a key. >>> >>> >>> >>> On Friday, May 27, 2016, Vikram Lalit wrote: >>> >>>> I have a Riak cluster (of 3 nodes, with 64 partitions and n_val = 3) >>>> but I find that for some objects, their hosting partitions / vnodes are not >>>> spread out across the 3 nodes. In some cases, 2 of them are on 1 node and >>>> the third is on a second node. That runs contrary to my understanding (link >>>> here: http://docs.basho.com/riak/kv/2.1.4/learn/concepts/clusters/) >>>> that the data is spread out across partitions in such a way that the >>>> partitions are on different servers. Is there something I'm missing here >>>> please in terms of how Riak works? Thanks... >>>> >>>> -- Alexander Sicular Solutions Architect Basho Technologies 9175130679 @siculars ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak Search and custom Solr components
Solr config is a bit of an acquired skill, a black art even. Would you be willing to shed some light on what you find to work best for you with your use case? Thank you, Alexander @siculars http://siculars.posthaven.com Sent from my iRotaryPhone > On May 31, 2016, at 10:54, Steve Garon wrote: > > Hey Guillaume, > > We had to alter the solrconfig.xml in our deployment because the default > solrconfig.xml does not give good search throughput with a large amount of > documents updating often. I've opened a ticket at basho to get the > solrconfig.xml save into the buckets settings that way they would be the same > across the cluster. I'm not sure when and if they'll have it in. I guess the > more ppl that ask for this, the sooner we'll get it. > > In the meantime, what we did is that during our riak node installation > procedure, we create the index using curl then we override the solrconfig.xml > and then restart riak which in turn restarts solr. From then, the > solrconfig.xml should remain intact. There is one more thing though. > Sometimes, for unknown reasons, riak decides to recreate the solrconfig.xml. > So you might wanna have a script that runs on the node to validate the > solrconfig.xml has not changed and to fix it automatically if it does. > > I know this is not ideal, solrconfig.xml really needs to be saved in bucket > settings like the schema.xml. Lets just hope the Basho works on fixing this > ASAP because for big deployments this is kind of crutial. > > > > Steve > >> On 30 May 2016 at 13:18, Guillaume Boddaert >> wrote: >> Please allow me to bump this previous message that was sent late last friday >> and that didn't attract much attention due to a well deserved week-end. >> >> >>> On 27/05/2016 19:14, Guillaume Boddaert wrote: >>> Hi there, >>> >>> I'm currently testing custom Component in my Riak Search system. As I need >>> a suggestion mechanism from the Solr index, I implemented the Suggester >>> component (https://wiki.apache.org/solr/Suggester). >>> It seems to work correctly, yet I have some question regarding the usage of >>> custom Solr configuration inside of Riak. The only caveat is that Riak >>> commits too often and that the suggestion index must be build manually and >>> not on commit. That's fine for me, i'll cron that once per day. >>> >>> First of all, how do stop/start/reload the solr instance without disturbing >>> the Riak core ? For the time being I'm stuck with service start/stop. How >>> can I reload my solr cores configuration without stopping riak ? >>> >>> Secondly, is that a good pattern to start tweaking defaults from my solr >>> cores (solrconfig.xml) ? Should I stop that right now and consider to use a >>> distinct Solr instance if I require those modifications ? Or should I >>> consider safe to alter cache/performance settings as well for Solr to match >>> my needs ? Can I play with other solr files such as stopwords and stuff >>> like that ? >>> >>> Finally, is there a proper way to alter default solrconfig.xml ? It is >>> auto-generated at index creation by Riak, yet that's a BIG and complex file >>> that may evolve between Riak releases. I'm creating indexes programatically >>> sending my index through Riak interface ( RiakClient.create_search_index in >>> Riak Python lib), but if I start to alter configuration I guess I need more >>> than that. >>> How do you guys would handle this ? >>> >>> Thanks, >>> >>> Guillaume >>> >>> >>> ___ >>> riak-users mailing list >>> riak-users@lists.basho.com >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> >> >> ___ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak Recap, June 3rd, 2016
Hello All, Just in time for the weekend we bring you a rundown of... the mailing list! As always, we want to hear from you. Let me know if you’re working on or speaking about something Riak related and would like a shoutout. I’ll get your talk, project or blog post in the next Riak Recap. ## Community Code and Walkthroughs - Dan Cunnington shows us how to monitor Riak from NodeJS via an Appmetrics integration [0]. ##Community Events - Basho will be at Spark Summit SF next week [1]. Stop by and say hi. ## Jobs at Basho Interested in working on distributed computing related problems or helping our community with theirs? Consider working at Basho. - Client Services Engineer (USA) [2]. - Developer Advocate (London and US) [3]. Yes, we're remote friendly. ## Recently Answered - Guillaume was able to confirm that Solr is a resource hog and indeed needs resources to operate within tolerable limits [4]. - Luke shares some Solr links with Alex to help grok relevancy queries [5]. - Seema lets us know that Basho is no longer supporting FreeBSD [6]. - Luke helps Alex get to the bottom of some Stanchion/RiakS2 (formerly Riak CS) related questions. Yes, Stanchion is an oracle and should only have one running process in your cluster [7]. - Robert has a question about Erlang versions [8]. Sargun points out that kerl [9] is a great way to manage multiple Erlang versions on the same box and Luke notes that Riak packages come bundled with the correct Erlang version [10]. - Vikram was having issues with data availability during different failure scenarios in a three node cluster [11]. DeadZen and I point out that this is most likely a quorum situation and how to ensure you always have access to your data going forward. Incidentally, always being able to access your data is exactly what Riak was designed to do. - Nick answers Richards question on the Riak Erlang clients thread-safeness [12]. Yes, connections may be shared in the riak-erlang-client [13]. - Johnny and Alex are looking for information on changing a clusters ring size [14]. Although it is possible in 2.x, the feature is more or less deprecated. Changing the ring size is not without caveats and performance considerations [15]. - Vikram is concerned with how Riak distributes data in a cluster [16] and Sam provides a number of great links that review the subject [17]. - Norman is interested in using Riak as a session store backing in a NodeJS/Express application [18]. Riak is indeed quite adept at storing session data and Luke points out [19] a possible way forward by using the connect-memcached lib as a starting point by replacing its memchached client lib with the Riak nodejs client lib [20]. ## Open Discussions - Kyle is looking for information on Riak TS drop table functionality and HTTP auth support [21]. - Guillaume kicks off a conversation on custom Solr configs [22] and Luke points to an open issue on github reviewing some considerations [23]. - Anil is looking for ways to copy Solr indecies from one machine to another [24]. - Murali is looking for information around Riak S2 (formerly CS) performance on a single node [25]. - Vladimir asks the list for help with the following warning “perflists do not satisfy the n_val” [26]. - Alex is looking for advice regarding whether or not Riak TS would be better suited in certain use cases over Riak KV for his social network project [27]. - Michael is looking for some guidance on properly sizing a Riak S2 cluster [28]. - Psterk is looking for help on using hadoops’s distcpy to copy files from Riak S2 to hadoop [29][30]. Do have yourselves a great weekend, -Alexander Sicular Solution Architect, Basho @siculars [0] https://developer.ibm.com/open/2016/05/20/riak-monitoring-appmetrics/ [1] https://spark-summit.org/2016/ [2] http://bashojobs.theresumator.com/apply/0CTNKU/Client-Services-Engineer-Remote [3] http://bashojobs.theresumator.com/apply/nVY7Ea/Developer-Advocate-EMEA [4] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-May/018484.html [5] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-May/018481.html [6] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-May/018482.html [7] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-May/018455.html [8] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-May/018456.html [9] https://github.com/kerl/kerl [10] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-May/018459.html [11] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-May/018468.html [12] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-May/018475.html [13] https://github.com/basho/riak-erlang-client [14]
Re: Retrieving objects from Riak
Hi Gianluca, I'll answer inline. My question to you is what is your
use case? In general, the methods you mention could all work but there
are pros and cons with each. -Alexander
On Wed, Jun 15, 2016 at 12:46 PM, Gianluca Padovani wrote:
> Hi all,
> I'm exploring RiakKV and RiakTS and I have some questions:
>
> If I need to find a group of objects based on some properties, are there any
> way to search it on RiakKV?
>
> For example suppose to have some user and create a key as _ so
> I have something as:
> - TX_John
> - TX_May
> - CA_Susan
> - CA_Martin
>
> Can I find all the people of TX? Can I do something as
> users_bucket.get("TX_*") ? I think no.
No, Riak KV is key value and by default keys have no relationship to
one another.
>
> To manage this scenario one possible way is to use a Crdt::Set. I can create
> for every state a set and when I add a new user to users_bucket I add also
> in this list. So If I want all users of TX, I get the set of TX, retrieve
> all the keys and then get all the objects with a mulit_get.
CRDT's may be used. However, performance degrades as you add more
elements to a set. Also, you should be aware of the total key size.
Generally, you don't want keys/values larger than 1MB in Riak. Or more
precisely, you don't want a large discrepancy in the distribution of
your key/value sizes. One 10MB key/value floating around the cluster
may cause thousands of 1KB key/values to get "stuck" behind it in
internal message queues. This ends up introducing a non-deterministic
performance profile.
>
> Another solution is to use 2i, tag every user object with the state, get the
> list of the keys and then retrieve the objects always with a multi_get.
>
> Another options is to use riak search that is based on Solr.
>
2i and Solr may be used here. However Solr is a much better approach
in cases where you may have complex queries going forward. When using
Solr you need to make accommodations for its resource consumption.
Solr like lots of memory.
> Are there any other options? What is the "best options". Some concerns about
> that:
>
> In first solution (set) I need to do 2 write for every write and to get the
> objects I need to do a get and a multi_get, that I think it's a lot of get,
> If I want list it in another way (regarding age for example) I don't have
> any options.
> Using the 2i I should remember to add the 2i to every objects, If I had
> already stored some Object and I want to use a 2i I need to do a map reduce
> to retag every obejcts. Probably riak search is the best solution?
>
If you just use Sole to index and not to store data, Solr will return
a list of keys and you will need to fetch them yourself. The output of
a Solr query is a json object which itself may be cached as a k/v in
Riak for later retrieval. Of course, this all depends on your use case
and whether or not caching is appropriate.
> If I'll user RiakTS, is this search more simple?
>
> If I use state as Partition Key, I can write
>
> select * from users where state="TX"
>
> but, when I want search something I always filter for partition key,
> correct?
SQL queries in Riak TS require that all columns in a primary key be
queried as equalities. The exception to this is the timestamp column,
if you have one, that may be queried as a range. The reason for this
is because the primary key, with the quantum if part of the PK, is
hashed in order to determined where that data has been distributed in
the cluster. You can think of queries against a PK as an index look
up. Other columns in the schema may be queried as well, it is those
columns that are scanned and filtered. You can think of filters
applied against those additional columns as table scans.
If you look at the PK statement in the documentation, you'll notice
there are two lines. The first line is the partition key. The second
line is the local key. The partition key is what gets hashed to
determine distribution in the cluster. The local key is what is used
to sort data on disk and insure uniqueness. So, you may simply use
"state" as your partition key and "state, time" as your local key. So
although you don't need to, in most cases you'd probably want to use a
quantum in your partition key.
> I think that RiakTS doesn't support RiakSearch and 2i. Are there some plans
> for it?
Correct, Riak TS does not support 2i or Solr. Solr is not fast enough.
I believe the roadmap revolves around improving our SQL feature set at
this point.
>
> Is correct use RiakTS without a time series ? :-)
The timestamp data type in Riak TS is an int. You could just push
sequence ints into that column if you wanted.
>
> A long mail I know ...
>
> thanks and bye
> Gianluca
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mai
Riak Recap, June 17th, 2016
hadoops’s distcpy to copy files from Riak S2 to hadoop [33][34]. Peace and love, -Alexander Sicular Solution Architect, Basho @siculars [0] https://github.com/basho/riak-ruby-client/blob/master/RELNOTES.md [1] https://github.com/basho/riak-go-client/blob/master/RELNOTES.md [2] https://github.com/basho/riak-python-client/blob/master/RELNOTES.md [3] https://github.com/basho/riak-nodejs-client/blob/master/RELNOTES.md [4] https://github.com/cvitter/Riak-TS-JDBC-Driver [5] https://github.com/datalemming/PDL-notebooks [6] http://basho.com/posts/technical/what-does-time-series-mean-to-me/ [7] http://conferences.oreilly.com/velocity/devops-web-performance-ca [8] http://bashojobs.theresumator.com/apply/0CTNKU/Client-Services-Engineer-Remote [9] http://bashojobs.theresumator.com/apply/nVY7Ea/Developer-Advocate-EMEA [10] https://stackoverflow.com/questions/37728202/what-does-the-context-for-flags-relate-to-in-riak-kv/37729256#37729256 [11] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-June/018510.html [12] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-June/018518.html [13] https://github.com/basho-labs/riak-mesos [14] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-June/018525.html [15] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-June/018528.html [16] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-June/018538.html [17] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-June/018534.html [18] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-June/018533.html [19] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-June/018535.html [20] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-June/018539.html [21] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-June/018543.html [22] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-June/018513.html [23] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-June/018551.html [24] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-June/018548.html [25] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-May/018389.html [26] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-May/018483.html [27] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-May/018500.html [28] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-May/018469.html [29] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-May/018478.html [30] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-May/018375.html [31] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-April/018279.html [32] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-March/018242.html [33] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-April/018321.html [34] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-April/018322.html ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak Recap, July 1st, 2016
Greetings, For your review this (U.S.) holiday weekend we have some product announcements, a bit of JDBC, and a pinch of Docker along with the indomitable mailing list review. As always, we want to hear from you. Let me know if you’re working on or speaking about something Riak related and would like a shoutout. I’ll get your talk, project or blog post in the next Riak Recap. ## Announcements - Riak KV 2.0.7 [0]. This is an LTS feature release, including new updates to batching and Solr, Cuttlefish configurations, an OTP upgrade, and fixes for two product advisories [1][2]. - Riak TS Product Alert: Default Configuration For Handoff May Cause Data Loss in TS [3][4]. Please click through for mitigation strategy. Riak TS 1.3.1 will be released shortly. ## Community Code and Walkthroughs - Craig Vitter continues to make progress on his JDBC client for Riak TS. Take it out for a spin, contributions welcome [5]. - Jon Brisbin has been hacking together a Riak TS [6] and Riak KV [7] standalone or clustered deployment on Docker. Available in the Docker public repo, feedback welcome. ## Jobs at Basho Interested in working on distributed computing related problems or helping our community with theirs? We’re hiring, here are a few of our open positions. Check out the rest [8]. - Client Services Engineer (USA) [9]. - Developer Advocate (London and US) [10]. - Solution Architect [11]. Yes, you can work from home! ## Recently Answered - Luke Bakken helps Attila Nagy get the memory backend up and running in a multi backend configuration [12][13]. - Michael Gnatz has is interested in dumping data from Riak TS, Stephen Etheridge lets us know that data expiration via TTL will be coming in the next release [14]. ## Open Discussions - Marcin Skoczylas is looking for some insight on the growth of processing queues within Riak [15]. - Michael Gnatz may have found a bug in Riak TS, Gordon Guthrie asks Michael to file a bug report [16]. - Guido Medina is looking for word on an updated Java client library that supports Netty-4.1.x [17]. - Norman Khine is interested in running Riak on Amazon’s recently released Elastic File System (EFS) [18]. As it's brand new and no one has used it in production yet, I’d venture to say time will tell... along with vigorous testing. - Humberto Rodriguez is having some trouble getting Riak TS running on OSX due to naming related issue. Magnus Kessler tries to help him out [19]. - Russell and Luke continue to work with Vladyslav on sibling explosion in Riak CS [20]. - Johnny is looking for some help on his bitcask merge issue [21]. - Vladyslav is looking for clarification on the meaning of “riak-admin node down” [22], particularly in the face of node failure during a cluster resizing operation. - Kyle is looking for information on Riak TS drop table functionality and HTTP auth support [23]. - Guillaume kicks off a conversation on custom Solr configs [24] and Luke points to an open issue on github reviewing some considerations [25]. - Anil is looking for ways to copy Solr indexes from one machine to another [26]. Have a phenomenal holiday weekend (for those in the States!), -Alexander Sicular Solution Architect, Basho @siculars [0] http://docs.basho.com/riak/kv/2.0.7/release-notes [1] http://docs.basho.com/community/productadvisories/leveldbsegfault/ [2] http://docs.basho.com/community/productadvisories/codeinjectioninitfiles/ [3] https://docs.basho.com/community/productadvisories/130-dataloss/ [4] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-June/018571.html [5] https://github.com/cvitter/Riak-TS-JDBC-Driver [6] https://hub.docker.com/r/basho/riak-ts/ [7] https://hub.docker.com/r/basho/riak-kv/ [8] http://bashojobs.theresumator.com [9] http://bashojobs.theresumator.com/apply/0CTNKU/Client-Services-Engineer-Remote [10] http://bashojobs.theresumator.com/apply/nVY7Ea/Developer-Advocate-EMEA [11] http://bashojobs.theresumator.com/apply/EPglZt/Solution-Architect [12] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-June/018556.html [13] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-June/018558.html [14] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-June/018564.html [15] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-June/018557.html [16] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-June/018563.html [17] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-June/018565.html [18] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-June/018566.html [19] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-June/018568.html [20] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-June/0
Riak Recap, July 15, 2016
Hello Friends, For your reading enjoyment this weekend we have a number of announcements and the delectably effervescent list review. As always, we want to hear from you. Let me know if you’re working on or speaking about something Riak related and would like a mention. I’ll try to get your talk, project or blog post in the next Riak Recap. I know a guy. ## Announcements - Riak TS 1.3.1 has been made available and is a bug fix release, get it while it’s hot. [0][1]. - Riak NodeJS Client 2.2.2 fixes a few bugs including Solr return decoding [2][3]. - Riak Erlang Client 2.4.1 supports TS and OTP 19 [4][5]. - Riak Python Client 2.5.4 fixes a couple of bugs found while testing Riak TS and and addresses the issue of time zones when datetime objects are stored in Riak TS [6][7]. - Riak .NET Client 2.4.0 Beta 2 adds TTB support for TS operations [8][9]. - Riak TS LevelDB Expiry Beta 1 brings a rudimentary implementation for expiry to LevelDB. This code is NOT for production use. Please use this list to register your feedback. [10][11]. - We are looking for community feedback on ending support for Java 7 in the client library. Please share your thoughts in Riak Java Client Github issue 635 [12]. ## Community Code and Walkthroughs - Craig Vitter has been announced as a speaker [13] at All Things Open [14] October 26-27 in Raleigh, NC. - Stephen Etheridge pens a blog on our site entitled “How we built an Electric Racing Demo showcasing the Potential of Riak TS” [15] where he walks through how he built a TS demo with a micro-Scalextric car set. ## Jobs at Basho Interested in working on distributed computing related problems or helping our community with theirs? We’re hiring, here are a few of our open positions. Check out the rest [16]. - Client Services Engineer (USA) [17]. - Developer Advocate (London and US) [18]. - Solution Architect [19]. (Join Basho as a Solution Architect and you get to work with me! What could be better than that?) Yes, we do remote! ## Recently Answered - Kyle Nguyen writes in with a question on programmatically creating Riak KV bucket types [20]. Luke Bakken mentions that unfortunately that is not supported in the API however for those who are looking for that feature you are encouraged to support it here in this github issue [21]. - Luke Bakken confirms Henrik Johanssons concerns that Riak will no longer be supported on Solaris/OmniOS/Illumos in future releases [22]. Having said that, the community is invited to package Riak for their platform of choice [23]. - Vikram Lalit brings to our attention a failure scenario when running Riak on low powered VM’s [24]. Matthew Von-Maszewski points out a simple recovery [25] and identifies additional configuration parameters [26] which will allow you to develop/test against Riak running in low memory/low CPU environments. - Henning Verbeek opens a github issue [27] regarding an io.netty.util.concurrent.BlockingOperationException he is seeing in the Riak Java Client [28]. - Adam Winter starts a lengthy conversation on Docker support for Riak [29]. In short, Riak on Docker is doable [30][31], there are concerns, and we don’t officially support it… yet. ## Open Discussions - Jubin Juneja is looking for help updating a an index with a new schema [32]. - Marcin Skoczylas is looking for some insight on the growth of processing queues within Riak [33]. - Michael Gnatz may have found a bug in Riak TS, Gordon Guthrie asks Michael to file a bug report [34]. - Guido Medina is looking for word on an updated Java client library that supports Netty-4.1.x [35]. - Norman Khine is interested in running Riak on Amazon’s recently released Elastic File System (EFS) [36]. As it's brand new and no one has used it in production yet, I’d venture to say time will tell... along with vigorous testing. - Humberto Rodriguez is having some trouble getting Riak TS running on OSX due to naming related issue. Magnus Kessler tries to help him out [37]. Have an outstanding weekend, -Alexander Sicular Solution Architect, Basho @siculars [0] http://docs.basho.com/riak/ts/1.3.1/releasenotes/ [1] http://docs.basho.com/riak/ts/1.3.1/downloads/ [2] https://github.com/basho/riak-nodejs-client/releases/tag/v2.2.2 [3] https://github.com/basho/riak-nodejs-client/blob/master/RELNOTES.md [4] https://github.com/basho/riak-erlang-client/releases/tag/2.4.1 [5] https://github.com/basho/riak-erlang-client/blob/master/RELNOTES.md [6] https://pypi.python.org/pypi/riak/2.5.4 [7] https://github.com/basho/riak-python-client/blob/master/RELNOTES.md [8] https://www.nuget.org/packages/RiakClient/2.4.0-beta2 [9] https://github.com/basho/riak-dotnet-client/blob/master/RELNOTES.md [10] https://githu
Re: Detecting Hinted Handoff
Take a look at the "pw" and "pr" tunable consistency options for gets and puts. The base level of abstraction in Riak is the virtual node - not the physical machine. When data is replicated it is replicated to a replica set of virtual nodes. Those virtual nodes have primary and secondary (due to failures) allocations to physical machines. When using "pr" and "pw" options you instruct Riak to only service the request from virtual nodes that are residing on their primarily allocated physical machines. In short, by abusing pr/pw you can infer the state of your cluster from your application. Obviously, this is not foolproof. There may also be additional 2i specific issues to consider. Nevertheless, I always liked this trick. Also, review this four part series on tunable consistency : http://basho.com/posts/technical/understanding-riaks-configurable-behaviors-part-1/ http://basho.com/posts/technical/riaks-config-behaviors-part-2/ http://basho.com/posts/technical/riaks-config-behaviors-part-3/ http://basho.com/posts/technical/riaks-config-behaviors-part-4/ -Alexander @siculars http://siculars.posthaven.com Sent from my iRotaryPhone > On Jul 22, 2016, at 12:28, Hawk Newton wrote: > > I've got a use case in which I'd like to use a secondary index but can't > tolerate partial result sets caused by hinted handoffs. I'm not currently > running riak search and, as this is a fringe case, would prefer not add the > additional overhead and complexity if I can help it. > > I'd like to detect a hinted handoff operation and throw a 503, if possible. > > Does anyone know of a way I can programatically detect if a hinted handoff > is underway without having to shell out to riak-admin (yuck!) and parse the > results? I'm running riak 2.0.5 at the moment. > > Thank you in advance. > > -- Hawk > > > > -- > View this message in context: > http://riak-users.197444.n3.nabble.com/Detecting-Hinted-Handoff-tp4034489.html > Sent from the Riak Users mailing list archive at Nabble.com. > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak Recap, July 29, 2016
Hello Friends, For your reading enjoyment this weekend we bring you the Basho Academy a community client library and, as always, the mailing list recap. As always, don’t be shy, let us know your thoughts. If you’re working on or speaking about something Riak related and would like a mention. I’ll get your talk, project or blog post in the next Riak Recap. I have connections. ## Announcements - Basho is pleased to bring you the Basho Academy [0], a series of self paced, tutorial style lessons on Riak development and administration. In a blog post [1], Justin Pease walks through the motivation for developing the Basho Academy and what to expect from the Academy going forward. - Luke Bakken released Riak Python Client 2.5.5 [2][3]. This fixes a long standing issue with multi-get and multi-put operations where the thread pool did not shut down cleanly when the interpreter shuts down. Remember to "close()" your RiakClient instances to ensure cleanup [4]. ## Community Code and Walkthroughs - Jason Voegele will be giving a talk entitled “Dialyzer: Optimistic Type Checking for Erlang and Elixir” [5] at ElixirConf [6] in Orlando, FL Aug 31 - Sept 2. - Drew Kerrigan updates the community driven Riak Elixir Client adding Riak TS support [7]. ## Jobs at Basho Interested in working on distributed computing related problems or helping our community with theirs? We’re hiring, here are a few of our open positions. Check out the rest [8]. - Client Services Engineer (USA) [9]. - Developer Advocate (London and US) [10]. - Solution Architect [11]. (Join Basho as a Solution Architect and you get to work with me! What could be better than that?) Yes, we do remote! ## Recently Answered - Travis Kirstine writes in with a question [12] on ram utilization in bitcask. Luke Bakken points out that the bitcask calculator [13] outputs a minimum ram requirement. Riak and your OS will happily eat ram as load increases. Noms. - Henning Verbeek asks about how best to store arbitrarily large Java objects [14]. Alex Moore and Russell Brown offer some guidance. - Guillaume Boddaert is looking for cursor mark support in Solr [15]. A future update to Riak, already being worked on, will include an update to Solr which has this feature. - Vikram Lalit is looking for some clarification on expected behavior when playing with n, r and w values [16]. Tom Santero points out that increasing the n val is generally safe, decreasing it may not be. ## Open Discussions - Magnus Kessler helps “!” with a some high I/O concerns [17]. “!”, please add some more info to your request! - Hawk Newton is looking for some guidance on how best to detect Hinted Handoff programmatically vs parsing the command line output [18]. - Chris Johnson runs into an overload error in Riak TS [19] and continues a conversation on how best to deal with it, take a look at the github issue [20]. - Jubin Juneja is looking for help updating a an index with a new schema [21]. - Marcin Skoczylas is looking for some insight on the growth of processing queues within Riak [22]. - Michael Gnatz may have found a bug in Riak TS, Gordon Guthrie asks Michael to file a bug report [23]. - Guido Medina is looking for word on an updated Java client library that supports Netty-4.1.x [24]. Have a fantastical weekend, -Alexander Sicular Solution Architect, Basho @siculars [0] https://academy.basho.com [1] http://basho.com/posts/business/basho-academy/ [2] https://github.com/basho/riak-python-client/blob/master/RELNOTES.md [3] https://pypi.python.org/pypi/riak/2.5.5 [4] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-July/018616.html [5] http://www.elixirconf.com/2016/speakers.html#voegele [6] http://www.elixirconf.com/ [7] https://github.com/drewkerrigan/riak-elixir-client [8] http://bashojobs.theresumator.com [9] http://bashojobs.theresumator.com/apply/0CTNKU/Client-Services-Engineer-Remote [10] http://bashojobs.theresumator.com/apply/nVY7Ea/Developer-Advocate-EMEA [11] http://bashojobs.theresumator.com/apply/EPglZt/Solution-Architect [12] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-July/018608.html [13] http://docs.basho.com/riak/kv/2.1.4/setup/planning/bitcask-capacity-calc/ [14] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-July/018617.html [15] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-July/018629.html [16] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-July/018631.html [17] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-July/018609.html [18] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-July/018623.html [19] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-July/018634.html
Re: Riak Elixir limit fetch on bucket
Hi churcho, LIMIT has not yet been implemented in Riak TS. It is on the roadmap for a future release. -Alexander On Thu, Jul 28, 2016 at 5:25 AM, churcho wrote: > Due to the nature of what I want to build, I would like to be able to limit > the keys I fetch from a bucket. Is there a facility similat to what SQL has > for fetch limit(n) and if so how is it implemented in elixir? > > > > -- > View this message in context: > http://riak-users.197444.n3.nabble.com/Riak-Elixir-limit-fetch-on-bucket-tp4034505.html > Sent from the Riak Users mailing list archive at Nabble.com. > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak Recap, August 12, 2016
Hello my Denormalizeds, In case you’ve been beaten down by the dog days of summer, fight back by reading about the Spark Riak Connector and the ever present mailing list recap! As always, let us know your thoughts: If you’re working on or speaking about something Riak related and would like a mention, let me know. I’ll get your talk, project or blog post in the next Riak Recap. I know a guy. ## Announcements - The Spark Riak Connector is actively being enhanced [0]. Try it out and let us know what you think. ## Community Code and Walkthroughs - Jason Voegele will be giving a talk entitled “Dialyzer: Optimistic Type Checking for Erlang and Elixir” [1] at ElixirConf [2] in Orlando, FL Aug 31 - Sept 2. - Pavel Hardak guest-published an article on the Databricks blog: “The Quest for Hidden Treasure: An Apache Spark Connector for the Riak NoSQL database.” In it he reviews the architecture of the Spark Riak Connector and how to use it from Spark [3]. - Along with the blog post, we also published a Spark/Riak TS tutorial on the Databricks site. The aforementioned blog details how to get to the tutorial from within Databricks Community Edition or skip all that and follow here -> [4]. ## Jobs at Basho Interested in working on distributed computing related problems or helping our community with theirs? We’re hiring, here are a few of our open positions. Check out the rest [5]. - Client Services Engineer (USA) [6]. - Developer Advocate EMEA (London, UK) [7]. - Solution Architect [8]. (Join Basho as a Solution Architect and you get to work with me! What could be better than that?) Yes, we do remote! ## Recently Answered - In response to churcho, I mention that Riak TS does not yet have LIMIT functionality. LIMIT is on the roadmap and slated for a future release [9]. - Anatoly Smolyaninov asks a question on TTL in Riak TS [10] and Matthew Von-Maszewski lets him know that a global TTL feature will ship in the next release [11]. Here are more details on that [12]. - Henning Verbeek asks a detailed question on maintaining multibackends while upgrading a cluster [13]. Alex Moore and Magnus Kessler help shed some light on how best to handle the situation [14][15], respectively. ## Open Discussions - Magnus Kessler helps “!” with a some high I/O concerns [16]. “!”, please add some more info to your request! - Hawk Newton is looking for some guidance on how best to detect Hinted Handoff programmatically vs parsing the command line output [17]. - Chris Johnson runs into an overload error in Riak TS [18] and continues a conversation on how best to deal with it, take a look at the github issue [19]. - Jubin Juneja is looking for help updating a an index with a new schema [20]. - Marcin Skoczylas is looking for some insight on the growth of processing queues within Riak [21]. - Michael Gnatz may have found a bug in Riak TS, Gordon Guthrie asks Michael to file a bug report [22]. - Guido Medina is looking for word on an updated Java client library that supports Netty-4.1.x [23]. Have a rejuvenating weekend, -Alexander Sicular Solution Architect, Basho @siculars [0] https://github.com/basho/spark-riak-connector [1] http://www.elixirconf.com/2016/speakers.html#voegele [2] http://www.elixirconf.com/ [3] https://databricks.com/blog/2016/08/11/the-quest-for-hidden-treasure-an-apache-spark-connector-for-the-riak-nosql-database.html [4] https://docs.cloud.databricks.com/docs/latest/databricks_guide/index.html#03%20Data%20Sources/5%20Databases%20%26%20Other%20Data%20Sources/7%20RiakTS%20Tutorial.html [5] http://bashojobs.theresumator.com [6] http://bashojobs.theresumator.com/apply/0CTNKU/Client-Services-Engineer-Remote [7] http://bashojobs.theresumator.com/apply/nVY7Ea/Developer-Advocate-EMEA [8] http://bashojobs.theresumator.com/apply/EPglZt/Solution-Architect [9] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-August/018642.html [10] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-August/018643.html [11] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-August/018646.html [12] https://github.com/basho/leveldb/wiki/mv-expiry [13] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-August/018644.html [14] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-August/018645.html [15] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-August/018647.html [16] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-July/018609.html [17] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-July/018623.html [18] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-July/018634.html [19] https://github.com/basho/riak_kv/issues/1418 [20] http://lists.basho.com/pipermail/r
Riak Recap, August 26th, 2016
Hello Gang, For your viewing pleasure this weekend we’ve got a big product bump - Riak TS 1.4 - with an equally large bump to the Riak Java Client to check out along with the effervescent mailing list recap. Take those brand new bits out for a spin. As always, let us know your thoughts: If you’re working on or speaking about something Riak related and would like a mention, let me know. I’ll get your talk, project or blog post in the next Riak Recap. I know a guy. ## Announcements - Riak TS 1.4. There’s lots of new stuff here to get through. GROUP BY [0], ISO8601 support [1] and a global TTL [2] are just some of them. Check out the release notes [3] and give it a try [4]. As always, lots of props to all the engineers involved in making this happen: support, docs, product, time series team, performance and all our users who contributed along the way. Congrats! - Riak Java Client 2.0.7 comes stacked with features, fixes and Riak 1.4 support. Check out the release notes [5], maven [6], the all in one jar [7] and the API docs [8]. ## Jobs at Basho Interested in working on distributed computing related problems or helping our community with theirs? We’re hiring, here are a few of our open positions. Check out the rest [5]. - Client Services Engineer (London) [6]. - Developer Advocate EMEA (London) [7]. - Developer Advocate (US) [8]. - Solution Architect (SF) [9]. (Yes, we do remote!) ## Recently Answered - Kasper Tidemann writes in [10] with a Go client issue and Alex lets him (and the rest of us) know that the Go client only speaks protobuff. Go figure. - For those upgrading to Riak TS 1.4, Chris Johnson writes in with his notes on the process and enhancements to the documentation [11]. Thanks for your input Chris! ## Open Discussions - Ali Rıza KELEŞ is looking to use 3 of his 7 Riak nodes as CS instances and how to appropriately configure his riak.conf/advanced.config files to accomplish this [12]. This is a good time to chime in and simply say Riak doesn’t work that way. One does not simply dictate where data resides in Riak [13]. In other words, you can’t tell Riak to restrain data to a subset of nodes. Create a dedicated cluster. - Kyle Nguyen is working through some certificate based auth issues with the Java client [14] and Jonathan Joseph helped with details on how to turn on SSL debugging [15]. - Magnus Kessler helps “!” with a some high I/O concerns [16]. “!”, please add some more info to your request! - Hawk Newton is looking for some guidance on how best to detect Hinted Handoff programmatically vs parsing the command line output [17]. - Chris Johnson runs into an overload error in Riak TS [18] and continues a conversation on how best to deal with it, take a look at the github issue [19]. Have a fantastic weekend, -Alexander Sicular Solution Architect, Basho @siculars [0] http://docs.basho.com/riak/ts/1.4.0/using/querying/select/group-by/ [1] http://docs.basho.com/riak/ts/1.4.0/using/timerepresentations/ [2] http://docs.basho.com/riak/ts/1.4.0/using/global-object-expiration/ [3] http://docs.basho.com/riak/ts/1.4.0/releasenotes/ [4] http://docs.basho.com/riak/ts/1.4.0/downloads/ [5] https://github.com/basho/riak-java-client/releases/tag/riak-client-2.0.7 [6] http://search.maven.org/#artifactdetails%7Ccom.basho.riak%7Criak-client%7C2.0.7%7Cjar [7] http://riak-java-client.s3.amazonaws.com/index.html [8] http://basho.github.io/riak-java-client/2.0.7/ [5] http://bashojobs.theresumator.com [6] http://bashojobs.applytojob.com/apply/zznz3Z/Client-Services-Engineer [7] http://bashojobs.theresumator.com/apply/nVY7Ea/Developer-Advocate-EMEA [8] http://bashojobs.applytojob.com/apply/pA1Nnk/Developer-Advocate-US [9] http://bashojobs.theresumator.com/apply/EPglZt/Solution-Architect [10] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-August/018653.html [11] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-August/018659.html [12] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-August/018654.html [13] https://imgflip.com/i/19l66m [14] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-August/018661.html [15] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-August/018656.html [16] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-July/018609.html [17] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-July/018623.html [18] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-July/018634.html [19] https://github.com/basho/riak_kv/issues/1418 ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: speeding up bulk loading
Hi Travis,
I also want to confirm that you are spreading your load amongst all nodes
in the cluster. You should be connecting your C client to Riak via a proxy
like nginx/HAproxy/F5 [0]. The proxy will do a round robin/least
connections distribution to all nodes in the cluster. This will greatly
increase performance if you are not already doing it.
-alexander
[0] http://docs.basho.com/riak/kv/2.1.4/configuring/load-balancing-proxy/
Alexander Sicular
Solutions Architect
Basho Technologies
9175130679
@siculars
On Wed, Aug 31, 2016 at 10:41 AM, Travis Kirstine <
tkirst...@firstbasesolutions.com> wrote:
> Magnus
>
>
>
> Thanks for your reply. We’re are using the riack C client library for
> riak (https://github.com/trifork/riack) which is used within an
> application called MapCache to store 256x256 px images with a corresponding
> key within riak. Currently we have 75 million images to transfer from disk
> into riak which is being done concurrently. Periodically this transfer
> process will crash
>
>
>
> Riak is setup using n=3 on 5 nodes with a leveldb backend. Each server
> has 45GB of memory and 16 cores with standard hard drives. We made no
> significant modification to the riak.conf except upping the
> leveldb.maximum_memory.percent to 70 and tweeking the sysctl.conf as follows
>
>
>
> vm.swappiness = 0
>
> net.ipv4.tcp_max_syn_backlog = 4
>
> net.core.somaxconn = 4
>
> net.core.wmem_default = 8388608
>
> net.core.rmem_default = 8388608
>
> net.ipv4.tcp_sack = 1
>
> net.ipv4.tcp_window_scaling = 1
>
> net.ipv4.tcp_fin_timeout = 15
>
> net.ipv4.tcp_keepalive_intvl = 30
>
> net.ipv4.tcp_tw_reuse = 1
>
> net.ipv4.tcp_moderate_rcvbuf = 1
>
> # Increase the open file limit
>
> # fs.file-max = 65536 # current setting
>
>
>
> I have seen this error in the logs
>
> 2016-08-30 22:26:07.180 [error] <0.20777.512> CRASH REPORT Process
> <0.20777.512> with 0 neighbours crashed with reason: no function clause
> matching webmachine_request:peer_from_peername({error,enotconn},
> {webmachine_request,{wm_reqstate,#Port<0.2817336>,[],
> undefined,undefined,undefined,{wm_reqdata,'GET',...},...}}) line 150
>
>
>
> Regards
>
>
>
> *From:* Magnus Kessler [mailto:mkess...@basho.com]
> *Sent:* August-31-16 4:08 AM
> *To:* Travis Kirstine
> *Cc:* riak-users@lists.basho.com
> *Subject:* Re: speeding up bulk loading
>
>
>
> On 26 August 2016 at 22:20, Travis Kirstine com> wrote:
>
> Is there any way to speed up bulk loading? I wondering if I should be
> tweeking the erlang, aae or other config options?
>
>
>
>
>
>
>
> Hi Travis,
>
>
>
> Excuse the late reply; your message had been stuck in the moderation
> queue. Please consider subscribing to this list.
>
>
>
> Without knowing more about how you perform bulk uploads, it's difficult to
> recommend any changes. Are you using the HTTP REST API or one of the client
> libraries, which use protocol buffers by default? What concerns do you have
> about the upload performance? Please let us know a bit more about your
> setup.
>
>
>
> Kind Regards,
>
>
>
> Magnus
>
>
>
>
>
> --
>
> Magnus Kessler
>
> Client Services Engineer
>
> Basho Technologies Limited
>
>
>
> Registered Office - 8 Lincoln’s Inn Fields London WC2A 3BP Reg 07970431
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Spaces in the search string
uot;we rt\"
>> AND age_i:0">>, []).
>> {ok,{search_results,[],0.0,0}}
>> 410>
>> 410>
>> 410> riakc_pb_socket:search(Pid, <<"test_index">>, <<"name_s:we\ rt AND
>> age_i:0">>, []).
>> {error,<<"Query unsuccessful check the logs.">>}
>> 411>
>> 411>
>>
>> Cheers,
>> //Sean.
>>
>>
>> On Tue, Sep 6, 2016 at 2:48 PM, Jason Voegele > > wrote:
>>
>>> Hi Sean,
>>>
>>> Have you tried escaping the space in your query?
>>>
>>> http://stackoverflow.com/questions/10023133/solr-wildcard-qu
>>> ery-with-whitespace
>>>
>>>
>>> On Sep 5, 2016, at 6:24 PM, sean mcevoy >> > wrote:
>>>
>>> Hi List,
>>>
>>> We have a solr index where we store something like:
>>> <<"{\"key_s\":\"ID\",\"body_s\":\"some test string\"}">>}],
>>>
>>> Then we try to do a riakc_pb_socket:search with the pattern:
>>> <<"body_s:*test str*">>
>>>
>>> The request will fail with an error message telling us to check the logs
>>> and in there we find:
>>>
>>> 2016-09-05 13:37:29.271 [error] <0.12067.10>@yz_pb_search:maybe_process:107
>>> {solr_error,{400,"http://localhost:10014/internal_solr/crm_d
>>> b.campaign_index/select",<<"{\"error\":{\"msg\":\"no field name
>>> specified in query and no default specified via 'df'
>>> param\",\"code\":400}}\n">>}} [{yz_solr,search,3,[{file,"src
>>> /yz_solr.erl"},{line,284}]},{yz_pb_search,maybe_process,3,[{
>>> file,"src/yz_pb_search.erl"},{line,78}]},{riak_api_pb_server
>>> ,process_message,4,[{file,"src/riak_api_pb_server.erl"},{
>>> line,388}]},{riak_api_pb_server,connected,2,[{file,"src/
>>> riak_api_pb_server.erl"},{line,226}]},{riak_api_pb_server,de
>>> code_buffer,2,[{file,"src/riak_api_pb_server.erl"},{line,
>>> 364}]},{gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,
>>> 505}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]
>>>
>>>
>>> Through experiment I've figured out that it doesn't like the space as it
>>> seems to think the part of the search string after that space is a new key
>>> to search for. Which seems fair enough.
>>>
>>> Anyone know of a work-around? Or am I formatting my request incorrectly?
>>>
>>> Thanks in advance.
>>> //Sean.
>>>
>>> ___
>>> riak-users mailing list
>>> riak-users@lists.basho.com
>>>
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>>
>>>
>>>
>>>
>>>
>>> ___
>>> riak-users mailing list
>>> riak-users@lists.basho.com
>>>
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>>
>>
>
--
Alexander Sicular
Solutions Architect
Basho Technologies
9175130679
@siculars
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak Recap, September 24, 2016
sions, Guillaume Boddaert opens a new line of questioning around tuning the coverage plan itself [53]. Have a fantastic weekend, -Alexander Sicular Solution Architect, Basho @siculars [0] https://github.com/basho/spark-riak-connector/releases/v1.6.0 [1] https://github.com/basho/spark-riak-connector/blob/master/README.md [2] https://github.com/basho-labs/riak-mesos/releases/tag/1.3.0 [3] https://github.com/basho-labs/riak-mesos-scheduler/releases/1.8.1 [4] https://github.com/basho-labs/riak-mesos-executor/releases/tag/1.7.0 [5] https://github.com/basho-labs/riak_explorer/releases/tag/1.2.1 [6] http://www.erlang-factory.com/euc2016/gordon-guthrie [7] https://www.youtube.com/watch?v=uN0GIPWiRYo [8] http://basho.com/the-scaling-time-series-apps-roadshow/ [9] http://info.basho.com/RiakTSRoadshowChicagoUS092016_Registration.html [10] http://info.basho.com/RiakTSRoadshowSwedenEMEA092016_Registration.html [11] http://info.basho.com/RiakTSRoadshowLondonEMEA092016_Registration.html [12] http://info.basho.com/RiakTSRoadshowAtlantaUS092016_Registration.html [13] http://bashojobs.theresumator.com [14] http://bashojobs.theresumator.com/apply/zznz3Z/Client-Services-Engineer [15] http://bashojobs.theresumator.com/apply/nVY7Ea/Developer-Advocate-EMEA [16] http://bashojobs.theresumator.com/apply/pA1Nnk/Developer-Advocate-US [17] http://bashojobs.theresumator.com/apply/EPglZt/Solution-Architect [18] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-August/018655.html [19] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-August/018682.html [20] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-August/018683.html [21] https://github.com/basho/leveldb/wiki/riak-tuning-2 [22] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-September/018692.html [23] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-September/018690.html [24] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-September/018687.html [25] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-September/018699.html [26] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-September/018688.html [27] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-September/018689.html [28] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-September/018696.html [29] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-September/018693.html [30] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-September/018702.html [31] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-September/018701.html [32] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-September/018711.html [33] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-September/018705.html [34] http://search.maven.org/remotecontent?filepath=com/basho/riak/spark-riak-connector_2.10/1.6.0/spark-riak-connector_2.10-1.6.0-uber.jar [35] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-September/018710.html [36] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-September/018713.html [37] http://docs.basho.com/riak/kv/2.1.4/using/security/ [38] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-September/018715.html [39] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-September/018718.html [40] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-September/018726.html [41] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-September/018729.html [42] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-August/018663.html [43] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-August/018664.html [44] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-August/018666.html [45] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-August/018678.html [46] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-August/018686.html [47] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-September/018712.html [48] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-September/018723.html [49] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-September/018721.html [50] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-September/018722.html [51] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-September/018728.html [52] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-September/018731.html [53] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-September/018730.html ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: High number of Riak buckets
Hi Vikram, Bucket maximums aside, why are you modeling in this fashion? How will you retrieve individual keys if you don't know the time stamp in advance? Do you have a lookup somewhere else? Doable as lookup keys or crdts or other systems. Are you relying on listing all keys in a bucket? Definitely don't do that. Yes, there is a better way. Use Riak TS. Create a table with a composite primary key of topic and time. You can then retrieve by topic equality and time range. You can then cache those results in deterministic keys as necessary. If you don't already know, Riak TS is basically (there are some notable differences) Riak KV plus the time series data model. Riak TS makes all sorts of time series oriented projects easier than modeling them against KV. Oh, and you can also leverage KV buckets alongside TS (resource limitations not withstanding.) Would love to hear more, Alexander @siculars http://siculars.posthaven.com Sent from my iRotaryPhone > On Sep 29, 2016, at 19:42, Vikram Lalit wrote: > > Hi - I am creating a messaging platform wherein am modeling each topic to > serve as a separate bucket. That means there can potentially be millions of > buckets, with each message from a user becoming a value on a distinct > timestamp key. > > My question is there any downside to modeling my data in such a manner? Or > can folks advise a better way of storing the same in Riak? > > Secondly, I would like to modify the default bucket properties (n_val) - I > understand that such 'custom' buckets have a higher performance overhead due > to the extra load on the gossip protocol. Is there a way the default n_val of > newly created buckets be changed so that even if I have the above said high > number of buckets, there is no performance degrade? Believe there was such a > config allowed in app.config but not sure that file is leveraged any more > after riak.conf was introduced. > > Thanks much. > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak Recap, October 7th, 2016
Greetings Team Riak, For your perusing this weekend we’ve got a few blog posts to distract you from CNN’s round-the-clock Matthew coverage along with the presumably informative mailing list recap. Stay dry out there... Are you working on something Riak related? Sharing is caring ;) Let me know what you’re working on and I’ll get your talk, project or blog post in the next Riak Recap. ## Announcements - Craig Vitter writes in opensource.com, “Riak TS for time series analysis at scale” [0], describing his upcoming talk at All Things Open [1]. - Sean Kelly guest blogs on basho.com about pitfalls and misconceptions to watch out for when considering a microservices based architecture [2]. - Basho Engineer Jon Brisbin blogs about Riak on Docker and outlines step by step instruction on how to get up and running in a number of circumstances [3]. For those of you who have expressed interest in Riak on Docker, I encourage you to start with this post. - Viktoria Fordos and Csaba Hoch take us on a harrowing tale of what can go wrong when you’re all deployed and in production and how they reigned it in by using WombatOAM [4] in "Operational nightmare fun: dealing with misconfigured Riak Clusters" [5]. - Tom Sigler & Co. put together the “Riak KV: Operations Intro” course on Basho Academy [6]. - Riak Explorer [7] has been seeing steady improvements. You can now graph Riak stats data gathered from a local instance. - The Ripple project, a modeling layer for Riak in Ruby, has been transferred to its own repo. Those looking to contribute are encouraged to do so there [8]. ## Jobs at Basho Interested in working on distributed computing related problems or helping our community with theirs? We’re hiring, here are a few of our open positions. Check out the rest [9]. - Client Services Engineer (London) [10]. - Developer Advocate EMEA (London) [11]. - Developer Advocate (US) [12]. - Solution Architect (SF) [13]. (Yes, a number of these positions are remote.) ## Recently Answered - Niels Christian Sorensen is interested in certain backup options in Riak S2 [14]. Matthew Von-Maszewski links work being done to make Riak hot-backupable [15]. - Kyle Nguyen is interested in deleting bucket types [16], Magnus Kessler drops the knowledge [17][18]. - Vikram Lalit asks about best practices when modeling a messaging use case that may call for lots of buckets [19]. - Ricardo Mayerhofer has concerns about the continued evolution of Riak KV [20]. Charlie Voiselle, Riak KV wrangler at Basho, replies [21]. ## Open Discussions - Brandon Martin is interested in figuring out how to use map/reduce in the Erlang client when using bucket types [22]. - Sean McEvoy wants to get some more performance out of paginated queries in Solr [23]. Fred Dushin lets us know that cursor marks are the way to go [24]. Unfortunately, the front-end guys always want more [25]. - Lingesh is getting an error when starting Riak and enabling AAE due to 100% i/o utilization [26]. - Following up on previous Solr query, cursor mark and coverage plan discussions, Guillaume Boddaert opens a new line of questioning around tuning the coverage plan itself [27]. Have a safe weekend, -Alexander Sicular Solution Architect, Basho @siculars [0] https://opensource.com/life/16/9/time-series-analysis-riak-ts [1] https://allthingsopen.org/speakers/craig-vitter/ [2] http://basho.com/posts/technical/microservices-please-dont/ [3] http://basho.com/posts/technical/running-riak-in-docker/ [4] https://www.erlang-solutions.com/products/wombat-oam.html [5] https://www.erlang-solutions.com/blog/operational-nightmare-fun-dealing-with-misconfigured-riak-clusters.html [6] https://academy.basho.com/ [7] https://github.com/basho-labs/riak_explorer/ [8] https://github.com/riak-ripple/ripple [9] http://bashojobs.theresumator.com [10] http://bashojobs.theresumator.com/apply/zznz3Z/Client-Services-Engineer [11] http://bashojobs.theresumator.com/apply/nVY7Ea/Developer-Advocate-EMEA [12] http://bashojobs.theresumator.com/apply/pA1Nnk/Developer-Advocate-US [13] http://bashojobs.theresumator.com/apply/EPglZt/Solution-Architect [14] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-September/018733.html [15] https://github.com/basho/leveldb/wiki/mv-hot-backup [16] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-September/018740.html [17] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-September/018741.html [18] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-September/018743.html [19] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-September/018744.html [20] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-October/018750.html [21] http://lists.basho
Re: Riak nodes constantly crashing
Disk, memory or file descriptors would be my guess. Bitcask?
On Monday, October 24, 2016, Ricardo Mayerhofer
wrote:
> Hi all,
> I have a Riak 1.4 where the nodes seems to be constantly crashing. All 5
> nodes are affected.
>
> However it seems Riak manage to get them up again.
>
> Any idea on whats going on? Erros logs below.
>
> Thanks.
>
> error.log
> ...
> 2016-10-24 21:57:29.185 [error] <0.24570.1174> CRASH REPORT Process
> <0.24570.1174> with 0 neighbours crashed with reason: no case clause
> matching {ok,{http_error,"exit\r\n"},<<>>} in mochiweb_http:request/3
> line 107
> 2016-10-24 21:58:29.187 [error] <0.7109.1175> CRASH REPORT Process
> <0.7109.1175> with 0 neighbours crashed with reason: no case clause
> matching {ok,{http_error,"exit\r\n"},<<>>} in mochiweb_http:request/3
> line 107
> 2016-10-24 21:59:29.228 [error] <0.19612.1175> CRASH REPORT Process
> <0.19612.1175> with 0 neighbours crashed with reason: no case clause
> matching {ok,{http_error,"exit\r\n"},<<>>} in mochiweb_http:request/3
> line 107
> 2016-10-24 22:00:29.218 [error] <0.1356.1176> CRASH REPORT Process
> <0.1356.1176> with 0 neighbours crashed with reason: no case clause
> matching {ok,{http_error,"exit\r\n"},<<>>} in mochiweb_http:request/3
> line 107
> 2016-10-24 22:01:29.197 [error] <0.11380.1176> CRASH REPORT Process
> <0.11380.1176> with 0 neighbours crashed with reason: no case clause
> matching {ok,{http_error,"exit\r\n"},<<>>} in mochiweb_http:request/3
> line 107
> 2016-10-24 22:02:29.231 [error] <0.24279.1176> CRASH REPORT Process
> <0.24279.1176> with 0 neighbours crashed with reason: no case clause
> matching {ok,{http_error,"exit\r\n"},<<>>} in mochiweb_http:request/3
> line 107
>
> crash.log
> 2016-10-24 21:51:56 =CRASH REPORT
> crasher:
> initial call: mochiweb_acceptor:init/3
> pid: <0.28136.1621>
> registered_name: []
> exception error: {{case_clause,{ok,{http_error,
> "exit\r\n"},<<>>}},[{mochiweb_http,request,3,[{file,"src/
> mochiweb_http.erl"},{line,107}]},{proc_lib,init_p_do_apply,
> 3,[{file,"proc_lib.erl"},{line,227}]}]}
> ancestors: ['http_0.0.0.0:8098_mochiweb',riak_core_sup,<0.148.0>]
> messages: []
> links: [<0.201.0>,#Port<0.235869290>]
> dictionary: []
> trap_exit: false
> status: running
> heap_size: 377
> stack_size: 24
> reductions: 423
> neighbours:
> 2016-10-24 21:52:56 =CRASH REPORT
> crasher:
> initial call: mochiweb_acceptor:init/3
> pid: <0.7845.1622>
> registered_name: []
> exception error: {{case_clause,{ok,{http_error,
> "exit\r\n"},<<>>}},[{mochiweb_http,request,3,[{file,"src/
> mochiweb_http.erl"},{line,107}]},{proc_lib,init_p_do_apply,
> 3,[{file,"proc_lib.erl"},{line,227}]}]}
> ancestors: ['http_0.0.0.0:8098_mochiweb',riak_core_sup,<0.148.0>]
> messages: []
> links: [<0.201.0>,#Port<0.235879110>]
> dictionary: []
> trap_exit: false
> status: running
> heap_size: 377
> stack_size: 24
> reductions: 406
> --
> Ricardo Mayerhofer
>
--
Alexander Sicular
Solutions Architect
Basho Technologies
9175130679
@siculars
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak nodes constantly crashing
Take a look at the AAE settings here:
http://docs.basho.com/riak/kv/latest/using/cluster-operations/active-anti-entropy/
-Alexander
@siculars
http://siculars.posthaven.com
Sent from my iRotaryPhone
> On Oct 26, 2016, at 16:17, Steven Joseph wrote:
>
> I don't think you should disable AAE, you can tune its frequency.
>
> Steven
>
>
>> On Thu, 27 Oct 2016 03:50 Ricardo Mayerhofer wrote:
>> Yes, I'll check if the problem is the AAE! I will disable it and see the
>> results.
>>
>> Thanks Steven!
>>
>> On Tue, Oct 25, 2016 at 6:54 PM, Steven Joseph wrote:
>> Hi Ricardo,
>>
>> If you are using systemd might have to check LimitNOFILE for your units.
>> Active anti entropy runs periodically.
>>
>> Steven
>>
>>
>> On Wed, 26 Oct 2016 04:36 Ricardo Mayerhofer wrote:
>> What's weird is that the node crashes every minute at the same second. Is
>> there anything Riak may be running every minute?
>>
>> On Mon, Oct 24, 2016 at 8:28 PM, Ricardo Mayerhofer
>> wrote:
>> I'm also pasting the free -m:
>>
>> total used free sharedbuffers cached
>> Mem: 15039 14557482 0 37 4594
>> -/+ buffers/cache: 9925 5114
>> Swap:0 0 0
>>
>> On Mon, Oct 24, 2016 at 8:24 PM, Ricardo Mayerhofer
>> wrote:
>> Hi Alexander,
>> Thanks for your response. We use multi-backend with bitcask and leveldb.
>>
>> - File descriptors seems to be ok, at least the config.
>>
>> ubuntu@ip-10-2-58-5:/var/log/riak$ sudo su riak
>> sudo: unable to resolve host ip-10-2-58-5
>> riak@ip-10-2-58-5:/var/log/riak$ ulimit -n
>> 65535
>>
>> - Memory seems to be ok as well:
>> KiB Mem: 15400916 total, 14493744 used, 907172 free,36244 buffers
>>
>> - Disk is ok
>>
>> /dev/xvda1 20G 4.1G 15G 22% / # root device
>>
>> /dev/xvdb 148G 69G 72G 49% /mnt/riak-data # bitcask and riak
>> data disk
>> /dev/xvdc 296G 23G 258G 8% /mnt/riak-data/leveldb #leveldb disk
>>
>> Any other idea? Thanks.
>>
>> On Mon, Oct 24, 2016 at 8:06 PM, Alexander Sicular
>> wrote:
>> Disk, memory or file descriptors would be my guess. Bitcask?
>>
>>
>> On Monday, October 24, 2016, Ricardo Mayerhofer
>> wrote:
>> Hi all,
>> I have a Riak 1.4 where the nodes seems to be constantly crashing. All 5
>> nodes are affected.
>>
>> However it seems Riak manage to get them up again.
>>
>> Any idea on whats going on? Erros logs below.
>>
>> Thanks.
>>
>> error.log
>> ...
>> 2016-10-24 21:57:29.185 [error] <0.24570.1174> CRASH REPORT Process
>> <0.24570.1174> with 0 neighbours crashed with reason: no case clause
>> matching {ok,{http_error,"exit\r\n"},<<>>} in mochiweb_http:request/3 line
>> 107
>> 2016-10-24 21:58:29.187 [error] <0.7109.1175> CRASH REPORT Process
>> <0.7109.1175> with 0 neighbours crashed with reason: no case clause matching
>> {ok,{http_error,"exit\r\n"},<<>>} in mochiweb_http:request/3 line 107
>> 2016-10-24 21:59:29.228 [error] <0.19612.1175> CRASH REPORT Process
>> <0.19612.1175> with 0 neighbours crashed with reason: no case clause
>> matching {ok,{http_error,"exit\r\n"},<<>>} in mochiweb_http:request/3 line
>> 107
>> 2016-10-24 22:00:29.218 [error] <0.1356.1176> CRASH REPORT Process
>> <0.1356.1176> with 0 neighbours crashed with reason: no case clause matching
>> {ok,{http_error,"exit\r\n"},<<>>} in mochiweb_http:request/3 line 107
>> 2016-10-24 22:01:29.197 [error] <0.11380.1176> CRASH REPORT Process
>> <0.11380.1176> with 0 neighbours crashed with reason: no case clause
>> matching {ok,{http_error,"exit\r\n"},<<>>} in mochiweb_http:request/3 line
>> 107
>> 2016-10-24 22:02:29.231 [error] <0.24279.1176> CRASH REPORT Process
>> <0.24279.1176> with 0 neighbours crashed with reason: no case clause
>> matching {ok,{http_error,"exit\r\n"},<<>>} in mochiweb_http:request/3 line
>> 107
>>
>> crash.log
>> 2016-10-24 21:51:56 =CRASH REPORT
>> crasher:
>> initial call: mochiweb_acceptor:init/3
>> pid: <0.28136.1621>
>> registered_name: []
>> exception error:
>> {{case_clause,{ok,{htt
Riak Recap, October 29th, 2016
Hello Ghouls and Gals,
This weekend I offer you a bit of light reading from our Riak Users email
archive (always available, rarely deletable!) Also, between raiding the
kids’ M&M’s and Skittles, make time to bone up on your Riak skills with
some new courses at the Basho Academy [0]. For those not in the know, we’re
continuously adding quick, self paced tutorials on various Riak related
topics to the Basho Academy. If there is something you think could use some
extra attention let us know and we’ll make it happen.
Are you working on something Riak related? Reply to this missive and I’ll
get your talk, project or blog post in the next Riak Recap. Perhaps you’re
interested in penning a guest post on the Basho blog? [1]
## Announcements
-
Introducing two new training modules on Basho Academy:
-
Riak KV: Operations Intro [2]
-
Riak TS: Spark Riak Connector [3]
## Basho for Business
Let’s assume you use Riak. We want to help you get the most out of it. The
Basho for Business section of the Riak Recap will highlight ways that Basho
can help you do just that.
Did you know that Basho offers Open Source support [4] and professional
services [5] such as developer and operations training? We do, it’s true.
If you've been considering licensing Riak, our sales staff are great people
to work with and can help you get the licenses you need at a price that
works. For example, we offer reduced pricing for startups. If you're
interested in hearing more, with no pressure or obligation, drop us your
details [6] and we'll have someone reach out to answer your questions. Hey,
you might get to talk to me!
## Jobs at Basho
Interested in working on distributed computing related problems or helping
our community with theirs? We’re hiring, here are a few of our open
positions. Check out the rest [7].
-
Client Services Engineer (London) [8].
-
Developer Advocate EMEA (London) [9].
-
Developer Advocate (US) [10].
-
Solution Architect (SF) [11].
(Yes, we are remote friendly.)
## Recently Answered
-
Brandon Martin is interested in figuring out how to use map/reduce in
the Erlang client when using bucket types [12]. Basho engineer Luke Bakken
replies with some recommendations and examples! [13]
-
Damion Junk is looking for guidance on running KV use cases alongside TS
use cases in Riak TS cluster [14]. The official party line from Basho's
John Daily at this time is that although you can do it, Basho does not
currently support it and recommends you split the functionality into two
clusters, KV and TS [15][16]. Damien Krotkine joins the conversation to
report his experience in when doubling up use cases in Riak TS [17]. And I
find it interesting that the Dami{o|e}n's have a Levenshtein distance [18]
of 1.
-
Travis Kirstine has a question on whether or not the Riak claimant node
has higher load than its peers [19]. Magnus Kessler informs us that it does
not [20].
-
AJAX DoneBy Jack (who does AJAX) is interested in knowing how to specify
query fields (qf) in a Solr query via the Riak Java Client. Basho's Magnus
Kessler [21] and Fred Dushin [22] let us know that when communicating with
Riak over protobuf there is no easy way to add certain Solr parameters.
Nevertheless Alex Moore [23] shows us a way to do this by piggybacking the
qf param in the {!dismax} block.
-
Joe Olson brings us a question on max quantum[24] range when querying in
Riak TS [25]. John Daily adds some clarity [26]. For the uninitiated, Riak
TS will execute a range query against a maximum number of quanta
(configurable) [27][28].
-
In answering Rohit Sanbhadti question [29] on webmachine/mochiweb
sockets, Luke Bakken lets us know that there is currently no way to
dynamically modify mochiwebs max connections [30]. An increasing listen()
backlog is generally symptomatic of not being able to handle increased load.
-
AJAX DoneBy Jack is back with a question [31] on Riak Explorer [32] and
SSL. Magnus Kessler lets us know that Riak Explorer hasn't quite figured
out how to speak SSL yet [33].
## Open Discussions
-
Ricardo Mayerhofer is experiencing node crashes [34] in his cluster.
Ricardo, any update?
-
Lingesh is getting an error when starting Riak and enabling AAE due to
100% i/o utilization [35].
-
Following up on previous Solr query, cursor mark and coverage plan
discussions, Guillaume Boddaert opens a new line of questioning around
tuning the coverage plan itself [36].
Have a spooktacular weekend,
-Alexander Sicular
Solution Architect, Basho
@siculars
[0] https://academy.basho.com/
[1] http://basho.com/blog/
[2] https://academy.basho.com/kv-ops-1/
[3] https://academy.basho.com/ts-spark-1/
[4] http://basho.com/products/support/
[5] http://basho.com/products/professional-services/
Re: Hinted handoff failed because of tcp errors
Hi Ryan, yes, you can change a number of settings. Have you had a look
at http://docs.basho.com/riak/kv/2.1.4/using/admin/riak-admin/#transfer-limit
and
http://lists.basho.com/pipermail/riak-users_lists.basho.com/2014-July/015529.html
?
-Alexander
On Tue, Nov 1, 2016 at 2:43 AM, Ryan Maclear wrote:
> Good Day,
>
> We have a 4 node riak cluster running inside AWS. The riak is riak-kv 2.1.2
> with AAE enabled on Ubuntu 14.04.4 LTS
>
> We are in the process of replacing one node with another using the process
> described here:
>
> http://docs.basho.com/riak/kv/2.1.4/using/cluster-operations/replacing-node/
>
> We have successfully replaced two of the nodes so far but we are having a
> problem with the third. If we look at /var/log/riak/console.log we see the
> start of the hinted handoff, and some time later (sometimes minutes and
> sometimes hours) we see:
>
> 2016-10-31 06:30:40.090 [error]
> <0.19834.2101>@riak_core_handoff_sender:start_fold:272 hinted transfer of
> riak_kv_vnode from 'r...@aew54.miranetworks.net'
> 274031556999544297163190906134303066185487351808 to
> 'r...@aew75.miranetworks.net'
> 274031556999544297163190906134303066185487351808 failed because of TCP recv
> timeout
> 2016-10-31 06:30:40.090 [error]
> <0.187.0>@riak_core_handoff_manager:handle_info:303 An outbound handoff of
> partition riak_kv_vnode 274031556999544297163190906134303066185487351808 was
> terminated for reason: {shutdown,timeout}
>
> So the handoff was terminated due to a tcp timeout. The handoff then starts
> again.
>
> This has been going on for some times (about two weeks now).
>
> The current member status is as follows:
>
> riak-admin member-status
> = Membership
> ==
> Status RingPendingNode
> ---
> leaving 0.0% -- 'r...@aew54.miranetworks.net'
> valid 25.0% -- 'r...@aew59.miranetworks.net'
> valid 25.0% -- 'r...@aew73.miranetworks.net'
> valid 25.0% -- 'r...@aew74.miranetworks.net'
> valid 25.0% -- 'r...@aew75.miranetworks.net'
> ---
> Valid:4 / Leaving:1 / Exiting:0 / Joining:0 / Down:0
>
>
> Here are some questions:
>
> 1. What is the default tcp timeout?
> 2. Is there any way to increase this timeout?
> 3. Is there any way to increase the rate of handoff?
> 4. Are there any other parameters we can tune to try and avoid this?
>
> The output from riak-admin transfers is as follows:
>
> 'r...@aew54.miranetworks.net' waiting to handoff 1 partitions
>
> Active Transfers:
>
> transfer type: hinted
> vnode type: riak_kv_vnode
> partition: 274031556999544297163190906134303066185487351808
> started: 2016-11-01 05:30:47 [2.10 hr ago]
> last update: 2016-11-01 07:36:51 [3.03 s ago]
> total size: 78393086512 bytes
> objects transferred: 11440967
>
> 1513 Objs/s
> riak@aew54.miranetworks.n ===> riak@aew75.miranetworks.n
> et et
> |== | 15%
> 1.53 MB/s
>
>
> Thanks,
> Ryan Maclear
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak TS Agility on handling Petabytes of Data
Hi Rajaa, What's your retention policy? At the moment, TS supports a global TTL. What's your read pattern? Is this a metrics or logging use case, aka can you downsample. Thanks, Alexander On Fri, Nov 18, 2016 at 03:25 wrote: > Dear Team, > > > > As a process of validation, we would like to know certain aspects w.r.t > Riak TS. Let's say per day 100TB real time data we get in our application > which crossed around more than petabytes of data in sometime. > > Would RIAK TS be an Ideal Choice for the above application ground. If so, > 1. How it will handle the data. 2. What is the performance of Reads/Writes > that we can expect. 3. How fast RIAK TS would be compared to MongoDB, > Cassandra & Druid > > > > > > *Thanks & Regards,* > > *Rajaa Krishnamurthy,* > > *IoT Gateway Developer* > > *Mob: +91 9940318287 | E-Mail: rajaa.krishnamur...@bt.com > * > > *TCS WeCare: BTnet Tel : 661 8179 | Tel: +44 (0) 121 311 8179 - WeCare > team can be reached if primary on-call support number is not reachable. > Available 24 X 7* > > > > [image: Blue_Pil] > > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak Recap, November 19th 2016
- Luca Favatella is looking for some insights from the community on how best to store data offline in an Android environment [67]. - Jing Liu is looking for best practices on how to load test Riak [68]. Magnus Kessler asks some more q’s and offers guidance [69]. - Toby Corkindale has concerns about Riak CS logging [70]. Have a fantastical weekend, -Alexander Sicular Solution Architect, Basho @siculars [0] http://basho.com/blog/ [1] http://basho.com/posts/technical/riak-kv-2-2-release-highlights/ [2] http://docs.basho.com/riak/kv/2.2.0/release-notes/ [3] http://docs.basho.com/riak/kv/2.2.0/setup/upgrading/cluster/ [4] https://docs.basho.com/riak/kv/2.2.0/setup/downgrade/ [5] https://github.com/basho/riak-java-client/blob/develop/RELNOTES.md#210 [6] https://github.com/basho/riak-java-client/releases/tag/riak-client-2.1.0 [7] http://search.maven.org/#artifactdetails%7Ccom.basho.riak%7Criak-client%7C2.1.0%7Cjar [8] https://github.com/basho/riak-erlang-client/blob/master/RELNOTES.md [9] https://github.com/basho/riak-erlang-client/releases/tag/2.5.0 [10] https://github.com/basho/riak-python-client/blob/master/RELNOTES.md#261-release [11] https://github.com/basho/riak-python-client/releases/tag/2.6.1 [12] https://pypi.python.org/pypi/riak/2.6.1 [13] https://github.com/basho/riak-php-client/blob/develop/RELNOTES.md [14] https://github.com/basho/riak-php-client/releases/tag/3.2.0 [15] https://packagist.org/packages/basho/riak#V3.2.0 [16] https://github.com/basho/riak-phppb-client/blob/master/RELNOTES.md#120 [17] https://github.com/basho/riak-phppb-client/releases/tag/1.2.0 [18] https://packagist.org/packages/basho/riak-pb#1.2.0 [19] https://github.com/basho/riak-nodejs-client/blob/master/RELNOTES.md [20] https://github.com/basho/riak-nodejs-client/releases/tag/v2.3.0 [21] https://www.npmjs.com/package/basho-riak-client [22] https://github.com/basho/riak-go-client/blob/master/RELNOTES.md [23] https://github.com/basho/riak-go-client/releases/tag/1.8.0 [24] https://github.com/basho/riak-ruby-client/blob/master/RELNOTES.md [25] https://github.com/basho/riak-ruby-client/releases/tag/v2.5.0 [26] https://rubygems.org/gems/riak-client/versions/2.5.0 [27] https://github.com/basho/riak-dotnet-client/blob/develop/RELNOTES.md#v240 [28] https://github.com/basho/riak-dotnet-client/releases/tag/v2.4.0 [29] https://www.nuget.org/packages/RiakClient/2.4.0 [30] https://github.com/erlang-lager/lager [31] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-November/018822.html [32] http://basho.com/products/support/ [33] http://basho.com/products/professional-services/ [34] http://basho.com/contact/ [35] http://bashojobs.theresumator.com [36] http://bashojobs.theresumator.com/apply/zznz3Z/Client-Services-Engineer [37] http://bashojobs.theresumator.com/apply/nVY7Ea/Developer-Advocate-EMEA [38] http://bashojobs.theresumator.com/apply/pA1Nnk/Developer-Advocate-US [39] http://bashojobs.theresumator.com/apply/EPglZt/Solution-Architect [40] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-October/018786.html [41] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-October/018787.html [42] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-November/018794.html [43] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-October/018789.html [44] http://riak-java-client.s3.amazonaws.com/index.html [45] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-November/018792.html [46] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-November/018793.html [47] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-November/018797.html [48] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-November/018798.html [49] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-November/018799.html [50] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-November/018804.html [51] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-November/018802.html [52] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-November/018805.html [53] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-November/018815.html [54] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-November/018806.html [55] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-November/018809.html [56] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-November/018810.html [57] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-November/018811.html [58] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-November/018816.html [59] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-November/018817.html [60] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-November/018819.html [61] http://lists.basho.com/pipermail/riak-users_lists.bash
Re: Riak CS: avoiding RAM overflow and OOM killer
Hi Daniel, How many nodes? -You should be using 5 minimum if you using the default config. There are reasons. How much ram per node? -As you noted, in Riak CS, 1MB file chunks are stored in bitcask. Their key names and some overhead consume memory. How many objects (files)? What is the average file size? -If your size distribution significantly skews < 1MB that means you will have a bunch of files in bitcask eating up ram. Kota was a former Basho engineer who worked on CS... That said, Basho may not support a non standard deployment. -Alexander On Mon, Nov 21, 2016 at 2:45 PM, Daniel Miller wrote: > I found a similar question from over a year ago > (http://lists.basho.com/pipermail/riak-users_lists.basho.com/2015-July/017327.html), > and it sounds like leveldb is the way to go, although possibly not well > tested. Has anything changed with regard to Basho's (or anyone else) > experience with using leveldb backend instead of the mutli backend for CS? > > On Fri, Nov 4, 2016 at 11:48 AM, Daniel Miller wrote: >> >> Hi, >> >> I have a Riak CS cluster up and running, and am anticipating exponential >> growth in the number of key/value pairs over the next few years. From >> reading the documentation and experience, I've concluded that the default >> configuration of CS (with riak_cs_kv_multi_backend) keeps all keys in RAM. >> The OOM killer strikes when Riak uses too much RAM, which is not good for my >> sanity or sleep. Because of the amount of growth I am anticipating, it seems >> unlikely that I can allocate enough RAM to keep up with the load. Disk, on >> the other hand, is less constrained. >> >> A little background on the data set: I have a sparsely accessed key set. >> By that I mean after a key is written, the more time passes with that key >> not being accessed, the less likely it is to be accessed any time soon. At >> any given time, most keys will be dormant. However, any given key _could_ be >> accessed at any time, so should be possible to retrieve it. >> >> I am currently running a smaller cluster (with smaller nodes: less RAM, >> smaller disks) than I expect to use eventually. I am starting to hit some >> growth-related issues that are prompting me to explore more options before >> it becomes a dire situation. >> >> My question: Are there ways to tune Riak (CS) to support this scenario >> gracefully? That is, are there ways to make Riak not load all keys into RAM? >> It looks like leveldb is just what I want, but I'm a little nervous >> switching over to only leveldb when the default/recommended config uses the >> multi backend. >> >> As a stop-gap measure, I enabled swap (with swappiness = 0), which I >> anticipated would kill performance, but was pleasantly surprised to see it >> return to effectively no-swap performance levels after a short period of >> lower performance. I'm guessing this is not a good long-term solution as my >> dataset grows. The problem with using large amounts of swap is that each >> time Riak starts it needs to read all keys into RAM. Long term, as our >> dataset grows, the amount of time needed to read keys into RAM will cause a >> very long restart time (and thus period of unavailability), which could >> endanger availability for a prolonged period if multiple nodes go down at >> once. >> >> Thanks! >> Daniel Miller >> Dimagi, Inc. >> > > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak CS: avoiding RAM overflow and OOM killer
Hi Daniel, Ya, I'm not surprised you're having issues. 4GB ram is woefully underspecd. 😔 🤓Stupid math: 3e7 x 3 (replication) / 9 = 1e7 minimum objects per node ( absolutely more due to obj > 1MB size ) 1e7 x ~400 bytes per obj in ram = 4e9 ram per node just for bitcask. Aka 4 GB. You already hit your limit. We can stop here. Done. End of. ☠️ 🤔But let's continue for funzies😋. Assuming defaults: Default ring_size = 64 / 9 nodes ~ 7 virtual nodes per physical node. Default leveldb ram allocation = 70% Leveldb operates, aka consumes resources including ram, on a vnode basis. It likes to consume ram on the order of 300MB through 2.5GB per vnode, increasing in performance till it caps. Even if you did switch everything to level you'd still be redlined. Bottom line is that bitcask, leveldb and your OS are fighting for ram all day 'ery day😡. Why you hate them and make them fight like that?😩 Not nice! (Trumpisms!)🤓 -Alexander ps. You probably want to bump to 128 ring size. More vnodes equals more parallelism, but also means more resource consumption. You prob want min 8 (v)CPU and 16GB min ram. YMMV, check my math. pps. If you don't want to double your per VM cost (aws ec2, etc) you could add nodes to the cluster. Because Riak uniformly distributes data around the cluster adding nodes increase total resources to the cluster, reduces number of objects allocated to each node. The converse is also true, if you double your node size you could halve your node count. That said, systems like Riak like prefer more nodes. It's just a math game. @siculars http://siculars.posthaven.com Sent from my iRotaryPhone > On Nov 22, 2016, at 08:51, Daniel Miller wrote: > > Hi Alexander, > > Thanks for responding. > > > How many nodes? > > We currently have 9 nodes in our cluster. > > > How much ram per node? > > Each node has 4GB of ram and 4GB of swap. The memory levels (ram + swap) on > each node are currently between 4GB and 5.5GB. > > > How many objects (files)? What is the average file size? > > We currently have >30 million objects, and I analyzed the average object size > before we migrated data into the cluster it was about 4KB/object, with some > objects being much larger (multiple MB). Is there an easy way to get this > information from a running cluster so I can give you more accurate > information? > > >> On Tue, Nov 22, 2016 at 2:42 AM, Alexander Sicular >> wrote: >> Hi Daniel, >> >> How many nodes? >> -You should be using 5 minimum if you using the default config. There >> are reasons. >> >> How much ram per node? >> -As you noted, in Riak CS, 1MB file chunks are stored in bitcask. >> Their key names and some overhead consume memory. >> >> How many objects (files)? What is the average file size? >> -If your size distribution significantly skews < 1MB that means you >> will have a bunch of files in bitcask eating up ram. >> >> Kota was a former Basho engineer who worked on CS... That said, Basho >> may not support a non standard deployment. >> >> -Alexander >> >> On Mon, Nov 21, 2016 at 2:45 PM, Daniel Miller wrote: >> > I found a similar question from over a year ago >> > (http://lists.basho.com/pipermail/riak-users_lists.basho.com/2015-July/017327.html), >> > and it sounds like leveldb is the way to go, although possibly not well >> > tested. Has anything changed with regard to Basho's (or anyone else) >> > experience with using leveldb backend instead of the mutli backend for CS? >> > >> > On Fri, Nov 4, 2016 at 11:48 AM, Daniel Miller wrote: >> >> >> >> Hi, >> >> >> >> I have a Riak CS cluster up and running, and am anticipating exponential >> >> growth in the number of key/value pairs over the next few years. From >> >> reading the documentation and experience, I've concluded that the default >> >> configuration of CS (with riak_cs_kv_multi_backend) keeps all keys in RAM. >> >> The OOM killer strikes when Riak uses too much RAM, which is not good for >> >> my >> >> sanity or sleep. Because of the amount of growth I am anticipating, it >> >> seems >> >> unlikely that I can allocate enough RAM to keep up with the load. Disk, on >> >> the other hand, is less constrained. >> >> >> >> A little background on the data set: I have a sparsely accessed key set. >> >> By that I mean after a key is written, the more time passes with that key >> >> not being accessed, the less likely it is to be accessed any time soon. At >> &
Re: Riak CS: avoiding RAM overflow and OOM killer
Hello DeadZen, Yes, networking interconnect becomes a bigger issue with more nodes in the cluster. A Riak cluster is actually a fully meshed network of erlang virtual machines. Multiple 1/10 gig nics dedicated to inter/intra networking are your friends. That said, we have many customers running many 50+ node clusters. Happy thanksgiving ppl! -Alexander @siculars http://siculars.posthaven.com Sent from my iRotaryPhone > On Nov 23, 2016, at 10:59, DeadZen wrote: > > ok I loled at this. then got worries trump could win a node election. > > anyways. 24gigs per riak server is not a bad safe bet. > Erlang in general is ram heavy. It uses it more effectively then most > languages wrt concurrency, but ram is the fuel for concurrency and buffer > for operations, especially dumb operations involving large orange loud-mouth > objects.. as pointed out you can increase cumulative resources by adding more > physical nodes. and there is a trade off for adding more virtual nodes. > theres also an IPC trade off adding more then a few dozen physical nodes in > the same cluster. Theres also possiby a trade off using riak ts and riak kv > in the same cluster. Nothing but tradeoffs. > >> On Tue, Nov 22, 2016 at 12:29 PM Alexander Sicular >> wrote: >> Hi Daniel, >> >> Ya, I'm not surprised you're having issues. 4GB ram is woefully underspecd. 😔 >> >> 🤓Stupid math: >> >> 3e7 x 3 (replication) / 9 = 1e7 minimum objects per node ( absolutely more >> due to obj > 1MB size ) >> >> 1e7 x ~400 bytes per obj in ram = 4e9 ram per node just for bitcask. Aka 4 >> GB. >> >> You already hit your limit. We can stop here. Done. End of. ☠️ >> >> 🤔But let's continue for funzies😋. >> >> Assuming defaults: >> >> Default ring_size = 64 / 9 nodes ~ 7 virtual nodes per physical node. >> >> Default leveldb ram allocation = 70% >> >> Leveldb operates, aka consumes resources including ram, on a vnode basis. It >> likes to consume ram on the order of 300MB through 2.5GB per vnode, >> increasing in performance till it caps. Even if you did switch everything to >> level you'd still be redlined. >> >> Bottom line is that bitcask, leveldb and your OS are fighting for ram all >> day 'ery day😡. Why you hate them and make them fight like that?😩 Not nice! >> (Trumpisms!)🤓 >> >> -Alexander >> >> ps. You probably want to bump to 128 ring size. More vnodes equals more >> parallelism, but also means more resource consumption. You prob want min 8 >> (v)CPU and 16GB min ram. YMMV, check my math. >> >> pps. If you don't want to double your per VM cost (aws ec2, etc) you could >> add nodes to the cluster. Because Riak uniformly distributes data around the >> cluster adding nodes increase total resources to the cluster, reduces number >> of objects allocated to each node. The converse is also true, if you double >> your node size you could halve your node count. That said, systems like Riak >> like prefer more nodes. It's just a math game. >> >> @siculars >> http://siculars.posthaven.com >> >> Sent from my iRotaryPhone >> >>> On Nov 22, 2016, at 08:51, Daniel Miller wrote: >>> >>> Hi Alexander, >>> >>> Thanks for responding. >>> >>> > How many nodes? >>> >>> We currently have 9 nodes in our cluster. >> >>> >>> >>> > How much ram per node? >> >>> >>> Each node has 4GB of ram and 4GB of swap. The memory levels (ram + swap) on >>> each node are currently between 4GB and 5.5GB. >> >>> >>> >>> > How many objects (files)? What is the average file size? >>> >> >>> We currently have >30 million objects, and I analyzed the average object >>> size before we migrated data into the cluster it was about 4KB/object, with >>> some objects being much larger (multiple MB). Is there an easy way to get >>> this information from a running cluster so I can give you more accurate >>> information? >>> >> >>> >>> On Tue, Nov 22, 2016 at 2:42 AM, Alexander Sicular >>> wrote: >>> Hi Daniel, >>> >>> How many nodes? >>> -You should be using 5 minimum if you using the default config. There >>> are reasons. >>> >>> How much ram per node? >>> -As you noted, in Riak CS, 1MB file chunks are stored in bitcask. >>> Their key names and some overhead consume m
Riak Recap, December 9th, 2016
Hello Folks, Hope you’re all enjoying this lull in the holiday season, it's a great time to catch up on some Riak. As usual, we have some reading material while you’re hopefully staying warm this weekend. Are you working on something Riak related? Let me know and I’ll get your talk, project or blog post in the next Riak Recap. Perhaps your project could be featured on our blog [0]. ## Announcements - New on Basho’s blog, part one of a three part series: “IoT Roundtable Participants Basho, IBM and Raytheon Discuss IoT at the Edge” [1] - Pavel Hardak lays out the open source IoT landscape at QCon SF 2016: “Building Scalable IoT Apps Using OSS Technologies” [2] - John Musser delivered a talk at Apache Spark Summit, Europe 2016 entitled “Lessons Learned Optimizing NoSQL for Apache Spark” [3] - Following up on Lager moving into its own repo, Mark Allen lets us know that he and Andrew Thompson will be hosting office hours on freenode. Take a look here for the details [4] ## Basho for Business We want to help you get the most out of Riak. Basho offers Open Source support [5], professional services [6], such as developer and operations training, and we offer reduced pricing for startups. If you're interested in hearing more, drop us your details [7] and we'll have someone reach out to answer your questions. Hey, you might get to talk to me! ## Jobs at Basho Interested in working on distributed computing related problems or helping our community with theirs? We’re hiring, here are a few of our open positions. Check out the rest [8]. - Client Services Engineer (London) [9]. - Developer Advocate EMEA (London) [10]. (Yes, we are remote friendly.) ## Recently Answered - Daniel Miller gets help regarding RAM overflow issues in Riak CS [11]. - Luca Favatella returns with am update on how best to store data offline in an Android environment [12]. - Konstantin Kalin gets help from Alex Moore [13] on some Java client issues [14]. - Toby Corkindale brings us a Java client shutdown issue [15] with demo code [16] and opens a github ticket for it [17]. Thanks for the report! - Gal Ben Haim wants to use KV buckets in TS [18]. Pavel Hardak, Product Manager for Riak TS, replies [19]. - Henning Verbeek is looking for information around where bucket type data resides and how to access them [20], Shaun McVey replies [21]. - Ricardo Mayerhofer is interested in a percentile() function in Riak TS [22], Pavel lets us know that its on the roadmap [23]. - Jing Liu wants to monitor ports and fd’s associated with Riak [24], Fred Dushin shares some advice [25]. - Arun Rajagopalan is looking for tips on listing keys for small buckets [26]. Although you should generally not list keys, Russell Brown lets us know the backend makes a difference [27]. ## Open Discussions - A R needs help writing data to TS via C# [28], Luke is helping with some of the debug [29]. - Toby Corkindale has concerns about Riak CS logging [30]. - Neo needs some help getting CS up and running from source [31]. - Vladyslav Zakhozhai is wondering how exactly pbc_pool_master_workers is defined [32]. Have a cozy weekend, -Alexander Sicular Solution Architect, Basho @siculars [0] http://basho.com/blog/ [1] http://basho.com/posts/business/iot-roundtable-participants-basho-ibm-and-raytheon-discuss-iot-at-the-edge-part-1-of-3/ [2] http://basho.com/resources/video/ [3] https://www.youtube.com/watch?v=qiglp28QsEU [4] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-December/018865.html [5] http://basho.com/products/support/ [6] http://basho.com/products/professional-services/ [7] http://basho.com/contact/ [8] http://bashojobs.theresumator.com [9] http://bashojobs.theresumator.com/apply/zznz3Z/Client-Services-Engineer [10] http://bashojobs.theresumator.com/apply/nVY7Ea/Developer-Advocate-EMEA [11] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-November/018801.html [12] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-November/018803.html [13] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-December/018861.html [14] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-November/018842.html [15] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-November/018852.html [16] https://gist.github.com/TJC/9a6a174cb1419a7c32e8018c5a495e3d [17] https://github.com/basho/riak-java-client/issues/689 [18] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-December/018860.html [19] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-December/018866.html [20] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-December/018863.html [21] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-December/018864.
Re: Storing Timestamps With Microsecond Resolution in RiakTS
Hi Joe, You could do that. Riak currently supports millisecond resolution. Look at the primary key composition. There are two lines, the first is the partition key and the second is the local key. The local key denotes the sort order and is the actual unique key for that grouping (quanta). You could do something like (syntax?) : Create table test( time timestamp not null, micro int not null, data varchar not null, Primary key( (Quantum(time, 1, m)), time, micro ) ) -Alexander @siculars http://siculars.posthaven.com Sent from my iRotaryPhone > On Dec 20, 2016, at 16:05, Joe Olson wrote: > > Is anyone storing timestamps with microsecond resolution in RiakTS? > > I'm interested in hearing if anyone is doing this, and how they are doing it. > My gut reaction is to have a compound timestamp + integer primary key, with > the microsecond part of the timestamp (least significant digits) going in the > integer field. > > Any guidance would be great! > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak Recap, December 23rd, 2016
Hello New Year's Revelers, This is the last recap of the year and it’s a big one. The dedicated team at Basho delivered yet another TS release, Riak TS 1.5, and it’s packed full of new features and performance enhancements. Many thanks to all the folks at Basho who’ve worked on Riak this year and many thanks to all of you who participate in this community. Looking forward to hearing about how you use Riak in the new year. Are you working on something Riak related? Let me know and I’ll get your talk, project or blog post in the next Riak Recap. Perhaps your project could be featured on our blog [0]. ## Announcements - Riak TS 1.5. New features like LIMIT, ORDER BY, ASC/DESC, BLOB type, DELETE EXPLAIN and many performance enhancements are just some of the features to check out in the documentation [1]. Read the release notes [2] and download [3] at your earliest convenience ;) - Or course, what release would be complete without library updates? - Riak Java Client 2.1.1: Maven [4], Release Notes [5], API Docs [6] - Riak Erlang Client 2.5.1: Tag [7], API Docs: [8] - Riak Python Client 2.7.0: PyPi [9], Tag [10], API Docs [11] - Riak Ruby Client 2.6.0: Rubygems [12], Tag [13], API Docs [14] - Riak Node.js Client 2.4.0: NPM [15], Tag [16], API Docs [17] - Riak .NET Client 2.5.0: Nuget [18], Tag [19], API Docs [20] - Riak PHP Client 3.3.0: Packagist [21], Tag [22], API Docs [23] - Riak PHP-PB Client 1.3.0: Packagist [24], Tag [25], API Docs [26] - Riak Go Client 1.9.0: Tag [27], API Docs [28] - Completing the three part series[29], “IoT Roundtable Participants Basho, IBM and Raytheon Discuss IoT at the Edge” here are parts two [30] and three [31]. - Gianluca Padovani recently spoke at NoSlidesConf [32] on riak_core [33] and will be blogging about the topic in a multipart series, the first of which has already been published, “Create a riak_core application in Elixir (Part 1)” [34], with accompanying code [35]. - Jason Voegele will be giving a talk at CodeMash [36], Jan 12th, on “Dialyzer: Optimistic Type Checking for Erlang and Elixir” [37]. Slides to follow. ## Basho for Business We want to help you get the most out of Riak. Basho offers Open Source support [38], professional services [39], such as developer and operations training, and we offer reduced pricing for startups. If you're interested in hearing more, drop us your details [40] and we'll have someone reach out to answer your questions. Hey, you might get to talk to me! ## Jobs at Basho Interested in working on distributed computing related problems or helping our community with theirs? We’re hiring, here are a few of our open positions. Check out the rest [41]. - Client Services Engineer (London) [42]. - Developer Advocate EMEA (London) [43]. (Yes, we are remote friendly.) ## Recently Answered - Felipe Esteves is interested in creating and listing bucket types [44]. Luke Bakken answers and opens a ticket to clean up the documentation [45]. - Joe Olson is looking for information on storing data in Riak at microsecond resolution [46]. Although Riak currently only supports millisecond resolution, you could work around it [47]. ## Open Discussions - Toby Corkindale needs help closing connections in Riak CS once his high availability HAproxy fails over [48]. - Pulin Gupta can delete data from Riak but it still persists in yokozuna [49]. - A R needs help writing data to TS via C# [50], Luke is helping with some of the debug [51]. - Toby Corkindale has concerns about Riak CS logging [52]. Merry Christmas, Happy Hanukkah, Happy Holidays, have a fantastic New Years and see you all in 2017! -Alexander Sicular Solution Architect, Basho @siculars [0] http://basho.com/blog/ [1] http://docs.basho.com/riak/ts/1.5.0/ [2] http://docs.basho.com/riak/ts/1.5.0/releasenotes/ [3] http://docs.basho.com/riak/ts/1.5.0/downloads/ [4] http://repo1.maven.org/maven2/com/basho/riak/riak-client/2.1.1/ [5] https://github.com/basho/riak-java-client/blob/riak-client-2.1.1/RELNOTES.md#211 [6] http://basho.github.io/riak-java-client/2.1.1/ [7] https://github.com/basho/riak-erlang-client/releases/tag/2.5.1 [8] http://basho.github.io/riak-erlang-client/ [9] https://pypi.python.org/pypi?:action=display&name=riak&version=2.7.0 [10] https://github.com/basho/riak-python-client/releases/tag/2.7.0 [11] http://basho.github.io/riak-python-client/ [12] https://rubygems.org/gems/riak-client/versions/2.6.0 [13] https://github.com/basho/riak-ruby-client/releases/tag/v2.6.0 [14] http://www.rubydoc.info/github/basho/riak-ruby-client/ [15] https://www.npmjs.com/package/basho-riak-client [16] https://github.com/basho/riak-nodejs-client/releases/tag/v2.4.0 [17] http
Re: Is every node share the same metadata ?
Hi Alex, You seem to be referring to bitcask metadata. That metadata is loaded into ram on each node. As you note, the bitcask capacity planner calculates ram required by a cluster to service a certain number of keys. I think where you are confused is that this metadata is not synced across the cluster. It is specific to the data stored specifically on each node. -Alexander Alexander Sicular Solutions Architect Basho Technologies 9175130679 @siculars On Fri, Jan 6, 2017 at 11:31 AM, Alex Feng wrote: > > Hi Riak Users, > > I am a little bit confused about the metadata, it is stored in memory and > synced in every node, so bascially every node share the same metadata, > right ? > But then from the official document, the formula to calcuate the RAM > requirement, the RAM requirement is for whole cluster, for example, based > on the formula below, the requirement is 50G, then if we have 5 nodes, then > each node only needs 10G RAM. Seems the metadata is divided to different > nodes, I must hvae misunderstood something, could someone please to clarify > this, thank you very much. > > > https://docs.basho.com/riak/kv/2.2.0/setup/planning/bitcask-capacity-calc/ > > Approximate RAM Needed for Bitcask = (static bitcask per key overhead + > estimated average bucket+key length in bytes) * estimate total number of > keys * n_val > > > > Br, > Alex > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak Recap, January 6th, 2017
Hello Folks, Welcome to a new year, the first Riak Recap of 2017 and a bunch of Riak related content. Hope you all had a great couple of weeks and are ready to ease into January with a few blog posts while steeping in some new features and updates in the Riak ecosystem. Are you working on something Riak related? Let me know and I’ll get your talk, project or blog post in the next Riak Recap. Perhaps your project could be featured on our blog [0]. ## Announcements - Adam Wray, Basho’s CEO, talks IoT, edge computing and competition with diginomica [1]. - Kait Carter highlights 2016 at Basho Academy and walks us through what’s to come in 2017 [2], for example a brand new course: “Load Balancing a Riak Cluster” [3]. - Chris Mancini takes us through the ins and outs of Riak’s new data type, the fabled HyperLogLog or HLL for short [4]. The post also links a deeper dive on HLL’s by Zeeshan Lakhani (pdf) [5]. You want this data in your base! - Randy Secrist reminded me to let everyone know to check out the latest Riak Explorer which adds Riak TS 1.5 functionality [6]. (And is also pretty cool.) - Riak Spark Connector 1.6.2 brings a couple of fixes, please upgrade [7][8]. - Jason Voegele will be giving a talk at CodeMash [9], Jan 12th, on “Dialyzer: Optimistic Type Checking for Erlang and Elixir” [10]. ## Basho for Business We want to help you get the most out of Riak. Basho offers Open Source support [11], professional services [12], such as developer and operations training, and we offer reduced pricing for startups. If you're interested in hearing more, drop us your details [13] and we'll have someone reach out to answer your questions. Hey, you might get to talk to me! ## Jobs at Basho Interested in working on distributed computing related problems or helping our community with theirs? We’re hiring, here are a few of our open positions. Check out the rest [14]. - Client Services Engineer (London) [15]. - Developer Advocate EMEA (London) [16]. (Yes, we are remote friendly.) ## Recently Answered - Arun Rajagopalan has a question on reaping tombstones in Riak [17]. A few folks join in with some recommendations and a hat tip towards future Riak KV 2.3 functionality that should make this problem go away. - Toby Corkindale gets some architectural help re. closing connections in Riak CS once his high availability HAproxy fails over [18]. - Tomi Takussaari is concerned about secondary index query failure when two nodes in the cluster fail [19]. Magnus Kessler replies with some architectural reasoning behind how 2I queries work [20]. - Ricardo Mayerhofer runs into a misconfiguration when using riak-shell, Riak TS’s repl [21]. - I help Alex Fang out with a question on bitcask metadata and ram consumption [22]. ## Open Discussions - Luca Favatella would like to get notified when CRDT updates are reflected in Solr [23]. Unfortunately there is no atomic way to do that natively, afaik. - Arun Rajagopalan wants to know the status of CS compatibility with Riak KV 2.2.0 [24]. Have a fabulous new year and, most importantly, a great weekend! -Alexander Sicular Solution Architect, Basho @siculars [0] http://basho.com/blog/ [1] http://diginomica.com/2017/01/05/interview-basho-ceo-takes-aim-iot-edge-computing/ [2] http://basho.com/posts/business/basho-academy-new-year-new-content/ [3] https://academy.basho.com [4] http://basho.com/posts/technical/what-in-the-hell-is-hyperloglog/ [5] https://github.com/basho/riak_kv/blob/develop/docs/hll/hll.pdf [6] https://github.com/basho-labs/riak_explorer/releases/tag/1.4.0 [7] https://spark-packages.org/package/basho/spark-riak-connector [8] https://github.com/basho/spark-riak-connector [9] http://www.codemash.org/ [10] http://www.codemash.org/session/dialyzer-optimistic-type-checking-for-erlang-and-elixir/ [11] http://basho.com/products/support/ [12] http://basho.com/products/professional-services/ [13] http://basho.com/contact/ [14] http://bashojobs.theresumator.com [15] http://bashojobs.theresumator.com/apply/zznz3Z/Client-Services-Engineer [16] http://bashojobs.theresumator.com/apply/nVY7Ea/Developer-Advocate-EMEA [17] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-December/01.html [18] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-December/018886.html [19] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2017-January/018899.html [20] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2017-January/018900.html [21] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2017-January/018903.html [22] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2017-January/018907.html [23] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-December/018893.html [24] http://lists.basho.com
Re: Export Riak TS tables to CSV
Hi Ricardo, Riak itself won't do this. Afaik, the client libraries return language specific array like data structures. To actually convert those to csv would be an exercise for the developer. Thankfully most languages have a readily available array to csv library which will basically do it for you. -Alexander On Thu, Jan 12, 2017 at 11:35 Ricardo Mayerhofer wrote: > Hi all! > We're planning to use some Riak TS tables as input to Amazon ML. Amazon ML > requires the input to be in CSV format. > > Is there any easy way to export Riak TS data? What's the recommended > approach? > > Thanks! > > -- > Ricardo Mayerhofer > > > > > ___ > > riak-users mailing list > > riak-users@lists.basho.com > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak: leveldb vs multi backend disk usage
Riak CS stores data chunks in bitcask and the index/metadata file in leveldb. Bitcask, as noted, has no compression. When you force Riak to use level for the data chunks you get compression for that data which may or may not be good for your use case. If it's not good for your use case I believe you can turn level compression off. -Alexander @siculars http://siculars.posthaven.com Sent from my iRotaryPhone > On Jan 27, 2017, at 04:29, Daniel Miller wrote: > > For ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: How many keys inside a bucket ?
Hi, you should consider using Riak TS for this use case. -Alexander @siculars http://siculars.posthaven.com Sent from my iRotaryPhone > On Jan 27, 2017, at 01:54, Alex Feng wrote: > > Hi, > > I am wondering if there are some best practice or recommendation for how many > keys inside a single bucket? > > Let's say I have some sensors reporting data every 5 seconds, I can put all > these data under one single bucket, or I can dynamically generate a bucket > every day. > > My question is, is there any performance difference for "insert", "query" > etc. between them? for example, 1 million records under one bucket and 10k > records under 100 buckets ? > > Thank you. > > Br, > Alex > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: How many keys inside a bucket ?
Hello Alex, As long as each bucket does not have its own properties but rather shares one or a handful of bucket types you should be fine and it wouldn't make a difference. One way to record data temporally, aka in a time series fashion, from a data model perspective is via a pattern called deterministic materialized keys. You basically record the sensor name (known id) and a base time at some granularity into the key name and push all data captured in that time frame into that key. This works well if you expect to record data at regular intervals. If, however, your data is sparse you'll need some additional model to know when you've recorded data. -Alexander @siculars http://siculars.posthaven.com Sent from my iRotaryPhone > On Jan 27, 2017, at 14:32, Alex Feng wrote: > > Hi Alexander, > > Yes, I should consider the possibility of switching to Riak TS. > But I guess the question still valid, does it ? Should I divide millions of > keys to different buckets, does it make any difference in performance, > memory, space ? > > > Br, > Alex > > 2017-01-27 2:50 GMT+01:00 Alexander Sicular : >> Hi, you should consider using Riak TS for this use case. >> >> -Alexander >> >> >> @siculars >> http://siculars.posthaven.com >> >> Sent from my iRotaryPhone >> >> > On Jan 27, 2017, at 01:54, Alex Feng wrote: >> > >> > Hi, >> > >> > I am wondering if there are some best practice or recommendation for how >> > many keys inside a single bucket? >> > >> > Let's say I have some sensors reporting data every 5 seconds, I can put >> > all these data under one single bucket, or I can dynamically generate a >> > bucket every day. >> > >> > My question is, is there any performance difference for "insert", >> > "query" etc. between them? for example, 1 million records under one >> > bucket and 10k records under 100 buckets ? >> > >> > Thank you. >> > >> > Br, >> > Alex >> > ___ >> > riak-users mailing list >> > riak-users@lists.basho.com >> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: [Basho Riak] Fail To Update Document Repeatly With Cluster of 5 Nodes
Speaking of timings:
ring_members : ['riak-node1@64.137.190.244','riak-node2@64.137.247.82',
'riak-node3@64.137.162.64','riak-node4@64.137.161.229',
'riak-node5@64.137.217.73']
Are these nodes in the same local area network?
On Thu, Feb 9, 2017 at 12:49 PM, my hue wrote:
> Dear Russel,
>
> I did the simplest possible with new document and use modify_type to update
> a single register.
> I still meet fail update at some times.
>
> My steps did as follow :
>
> Step 1: Initial a new document Map
> Step 2: Create new map with : riakc_pb_socket:update_type(Pid,
> {BucketType, Bucket}, Key, riakc_map:to_op(Map), []).
> Step 3: Fetch to check result :
> riakc_pb_socket:fetch_type(Pid,{BucketType,Bucket}, Key).
> Step 4: Create Fun for input of modify_type which update only one field of
> map
>
> Fun1 = fun(OldMap) -> riakc_map:update({<<"status_id">>, register}, fun(R)
> -> riakc_register:set(<<"show">>, R) end, OldMap) end.
>
> Fun2 = fun(OldMap) -> riakc_map:update({<<"status_id">>, register}, fun(R)
> -> riakc_register:set(<<"hide">>, R) end, OldMap) end.
>
> Step 5: Update :
>
> riakc_pb_socket:modify_type(Pid, Fun1, {BucketType, Bucket}, Key, []).
>
> Fetch to check :
>
> riakc_pb_socket:fetch_type(Pid,{BucketType,Bucket}, Key).
>
> Step 6: Update:
>
> riakc_pb_socket:modify_type(Pid, Fun2, {BucketType, Bucket}, Key, []).
>
> Fetch to check :
>
> riakc_pb_socket:fetch_type(Pid,{BucketType,Bucket}, Key).
>
>
> For my debug and test, I repeated step 5 and step 6 on one document about
> 20 times.
> And via many documents, I meet weird behaviour that some documents meet fail
> update, and some documents never fail update.
> The first time, I think that cause network, or timeout between nodes and
> this is only random of fail. So I deleted documents with command:
>
> riakc_pb_socket:delete(Pid, {BucketType,Bucket}, Key, []).
>
> Then retest on each document of first test again. And It is very amazing
> that the documents meet fail at first test still meet fail at this second
> test, and the documents passed at first test still pass at this second test.
> Delete all again, and retest and of course get the same result.
>
> After all I make other test case, I used one fail document at all test
> times, and keep all fields except change key to get different documents for
> the debug. And very surprise that I still got some fail and some success,
> although documents are the same field and value except key. Delete and
> retest and still the same result. Documents succeeded will be always
> succeed. And document meet fail will be always failed. I totally do not
> understand root cause till now. And hope that can get support and help from
> the developers of riak. I can tell that my system mostly fail with cluster
> run when faced this issue.
>
> The following is some map documents I used on the test. And I also attached
> the extracted log of each node at one of the fail times together with this
> email. I do not really get meaning of riak log but hope that can help
> developers of riak get something.
>
>
>
> * New Document which meet fail with my steps.
>
> {map,[],
> [{{<<"account_id">>,register},
> {register,<<>>,<<"accountqweraccountqweraccountqwer">>}},
> {{<<"created_by_id">>,register},
> {register,<<>>,<<"accountqweraccountqweraccountqwer">>}},
> {{<<"created_time_dt">>,register},
> {register,<<>>,<<"2017-02-7T23:49:04Z">>}},
> {{<<"currency">>,register}, {register,<<>>,<<"usd">>}},
>
> {{<<"id">>,register},{register,<<>>,<<"menu1234567812345678123456789">>}},
> {{<<"maintain_mode_b">>,register}, {register,<<>>,<<"false">>}},
> {{<<"menu_category_revision_id">>,register},
> {register,<<>>,<<"0-634736bc14e0bd3ed7e3fe0f1ee64443">>}},
> {{<<"name">>,register},{register,<<>>,<<"menutest">>}},
> {{<<"order_id">>,register},{register,<<>>,<<"0">>}},
> {{<<"rest_location_p">>,register},
> {register,<<>>,<<"10.844117421366443,106.63982392275398">>}},
> {{<<"restaurant_id">>,register},
> {register,<<>>,<<"rest848e042b3a0488640981c8a6dc4a8281">>}},
> {{<<"restaurant_status_id">>,register}, {register,<<>>,<<"active">>}},
> {{<<"start_time">>,register},{register,<<>>,<<"dont_use">>}},
> {{<<"status_id">>,register},{register,<<>>,<<"show">>}},
> {{<<"updated_by_id">>,register},
> {register,<<>>,<<"accountqweraccountqweraccountqwer">>}},
> {{<<"updated_time_dt">>,register},
> {register,<<>>,<<"2017-02-7T23:49:04Z">>}}],
> [],undefined}.
>
> Key = <<"menu1234567812345678123456789">>
>
> * New Document which always success with my steps:
>
> {map,[],
> [{{<<"account_id">>,register},
> {register,<<>>,<<"accountqweraccountqweraccountqwer">>}},
> {{<<"created_by_id">>,register},
> {register,<<>>,<<"accountqweraccountqweraccountqwer">>}},
> {{<<"created_time_dt">>,register},
> {register,<<>>,<<"2017-02-7T23:49:04Z">>}},
> {{<<"currency">>,register},{register,<<>>,<<"usd">>}},
>
> {{<<"id">>,re
Re: [Basho Riak] Fail To Update Document Repeatly With Cluster of 5 Nodes
Yes. For a number of reasons a single Riak cluster is not designed to run
over a WAN (Riak EE is specifically designed to connect two or more
separate clusters over a WAN or LAN.)
On Fri, Feb 10, 2017 at 02:40 Nguyễn Văn Nghĩa Em wrote:
> Hi All,
>
> These nodes are not in Local Area Network because our host provider
> doesn't have a local IP. It is only our DEV environments. If there are
> problems with cluster I will setup a local environment to try.
>
> Do you think that it is the reason of our issues ?
>
> Best Regards,
>
>
> 2017-02-10 3:21 GMT+07:00 DeadZen :
>
> Why are they public?
>
> On Thu, Feb 9, 2017 at 3:11 PM, Alexander Sicular
> wrote:
> > Speaking of timings:
> >
> > ring_members : ['riak-node1@64.137.190.244','riak-node2@64.137.247.82',
> > 'riak-node3@64.137.162.64','riak-node4@64.137.161.229',
> > 'riak-node5@64.137.217.73']
> >
> > Are these nodes in the same local area network?
> >
> > On Thu, Feb 9, 2017 at 12:49 PM, my hue
> wrote:
> >> Dear Russel,
> >>
> >> I did the simplest possible with new document and use modify_type to
> update
> >> a single register.
> >> I still meet fail update at some times.
> >>
> >> My steps did as follow :
> >>
> >> Step 1: Initial a new document Map
> >> Step 2: Create new map with : riakc_pb_socket:update_type(Pid,
> >> {BucketType, Bucket}, Key, riakc_map:to_op(Map), []).
> >> Step 3: Fetch to check result :
> >> riakc_pb_socket:fetch_type(Pid,{BucketType,Bucket}, Key).
> >> Step 4: Create Fun for input of modify_type which update only one
> field of
> >> map
> >>
> >> Fun1 = fun(OldMap) -> riakc_map:update({<<"status_id">>, register},
> fun(R)
> >> -> riakc_register:set(<<"show">>, R) end, OldMap) end.
> >>
> >> Fun2 = fun(OldMap) -> riakc_map:update({<<"status_id">>, register},
> fun(R)
> >> -> riakc_register:set(<<"hide">>, R) end, OldMap) end.
> >>
> >> Step 5: Update :
> >>
> >> riakc_pb_socket:modify_type(Pid, Fun1, {BucketType, Bucket}, Key, []).
> >>
> >> Fetch to check :
> >>
> >> riakc_pb_socket:fetch_type(Pid,{BucketType,Bucket}, Key).
> >>
> >> Step 6: Update:
> >>
> >> riakc_pb_socket:modify_type(Pid, Fun2, {BucketType, Bucket}, Key, []).
> >>
> >> Fetch to check :
> >>
> >> riakc_pb_socket:fetch_type(Pid,{BucketType,Bucket}, Key).
> >>
> >>
> >> For my debug and test, I repeated step 5 and step 6 on one document
> about
> >> 20 times.
> >> And via many documents, I meet weird behaviour that some documents meet
> fail
> >> update, and some documents never fail update.
> >> The first time, I think that cause network, or timeout between nodes and
> >> this is only random of fail. So I deleted documents with command:
> >>
> >> riakc_pb_socket:delete(Pid, {BucketType,Bucket}, Key, []).
> >>
> >> Then retest on each document of first test again. And It is very amazing
> >> that the documents meet fail at first test still meet fail at this
> second
> >> test, and the documents passed at first test still pass at this second
> test.
> >> Delete all again, and retest and of course get the same result.
> >>
> >> After all I make other test case, I used one fail document at all test
> >> times, and keep all fields except change key to get different documents
> for
> >> the debug. And very surprise that I still got some fail and some
> success,
> >> although documents are the same field and value except key. Delete and
> >> retest and still the same result. Documents succeeded will be always
> >> succeed. And document meet fail will be always failed. I totally do not
> >> understand root cause till now. And hope that can get support and help
> from
> >> the developers of riak. I can tell that my system mostly fail with
> cluster
> >> run when faced this issue.
> >>
> >> The following is some map documents I used on the test. And I also
> attached
> >> the extracted log of each node at one of the fail times together with
> this
> >> email. I do not really get meaning of riak log but hope that can help
> >> developers of riak get something.
> >>
> >>
> >>
> >> * New Document
Re: Start up problem talking to Riak
Please don't do that. Don't point the internet at your database. Have them communicate amongst each other on internal ips and route the public through a proxy / middleware. -Alexander @siculars http://siculars.posthaven.com Sent from my iRotaryPhone > On Feb 13, 2017, at 04:00, AWS wrote: > > I know that this isn't directly a Riak issue but I am sure that some of you > have met this before and can maybe help me. I am used to Macs and Windows but > have now set up an Ubuntu 14.04LTS server on my home network. I have 5 fixed > IP addresses so the server has its own external address. I have opened port > 8098 on my router to point at the server and checked that ufw isn't running. > I have tested with it running ufw and with 'allow 8098' applied. I still > cannot connect to Riak. On the same computer I get a pong back to a ping so > Riak seems to be OK. > > I have a Riak server running on AWS and had trouble setting that up until I, > eventually, opened all ports. > > Can anyone please suggest some steps that I might take? I need this running > for an Open University course that I am studying. My AWS free server runs out > before the course finishes so I have to get this up and running soon. > Thanks in advance. > David > > Message sent using Winmail Mail Server > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Testing netsplit in Riak
1. Check the erlang vm variable "nettick", I believe. 2. Hinted handoff resource allocation are configurable via config file or at runtime. On Wed, Feb 22, 2017 at 12:07 Andrey Ershov wrote: > Hi, guys! > > I'm testing netsplit in Riak and can not achieve satisfiable behaviour. > I've just two nodes cluster and bucket with the following settings n=3, > w=2, r=2. And I have just a couple of entries. > Basically I have two problems: > 1) After the split, writes on one side of the partition start lagging > hard. It takes more than 1 minute for the first write to be become > successful. I understand that this is related to the process of setting up > backup vnodes in Riak, but is any way to speed up the process? Which > configuration parameters influence that? > 2) More weird problem is after netsplit. "riak-admin transfers" command > immediately reports that there should 5 partition transfers from one node > to another and 5 partition transfers in the opposite direction. But active > transfers output is empty! > I've put a watch on this command and active transfers are always empty. > Finally, it takes several minutes for Riak to finish hinted handoff. > Several minutes just for several keys! > What Riak is doing all this time? Anyway to speed up the process? > 3) The reason why I'm concerned about hinted-handoff speed is because, I > noticed that until this process finishes, I read stale data on both sides > of ex-netsplit. > > > -- > Thanks, > Andrey > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > -- Alexander Sicular Solutions Architect Basho Technologies 9175130679 @siculars ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Testing netsplit in Riak
There's a reason the time is default higher. The larger the network the higher the probability nodes can't speak to each other momentarily. Too low too much gossip and too much flapping. Ymmv. @siculars http://siculars.posthaven.com Sent from my iRotaryPhone > On Feb 22, 2017, at 13:17, Andrey Ershov wrote: > > Alexander, thanks for your reply! > > 1) I've set erlang.distribution.nettick_time to 1 second and writes after > netsplit are very fast now. So this point is resolved. Do you know how this > parameter affects false positive ratio? Riak docs stay that every > nettick_time seconds netkernal will initiate remote processes life-checking. > However, it does not say anything about the mechanism. Do you know how this > failure detector works? > 2) As for hinted handoff, I still can not find any solution. Variables that > I've tried to change: >- vnode_management_timer from 10s to 1s >- transfer_limit from 2 to 100 > But still transfer take about a minute. Any other variables that I should > take a look at? > > 2017-02-22 21:12 GMT+03:00 Alexander Sicular : >> 1. Check the erlang vm variable "nettick", I believe. >> >> 2. Hinted handoff resource allocation are configurable via config file or at >> runtime. >> >>> On Wed, Feb 22, 2017 at 12:07 Andrey Ershov wrote: >>> Hi, guys! >>> >>> I'm testing netsplit in Riak and can not achieve satisfiable behaviour. >>> I've just two nodes cluster and bucket with the following settings n=3, >>> w=2, r=2. And I have just a couple of entries. >>> Basically I have two problems: >>> 1) After the split, writes on one side of the partition start lagging hard. >>> It takes more than 1 minute for the first write to be become successful. I >>> understand that this is related to the process of setting up backup vnodes >>> in Riak, but is any way to speed up the process? Which configuration >>> parameters influence that? >>> 2) More weird problem is after netsplit. "riak-admin transfers" command >>> immediately reports that there should 5 partition transfers from one node >>> to another and 5 partition transfers in the opposite direction. But active >>> transfers output is empty! >>> I've put a watch on this command and active transfers are always empty. >>> Finally, it takes several minutes for Riak to finish hinted handoff. >>> Several minutes just for several keys! >>> What Riak is doing all this time? Anyway to speed up the process? >>> 3) The reason why I'm concerned about hinted-handoff speed is because, I >>> noticed that until this process finishes, I read stale data on both sides >>> of ex-netsplit. >>> >>> >>> -- >>> Thanks, >>> Andrey >>> ___ >>> riak-users mailing list >>> riak-users@lists.basho.com >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> >> -- >> >> >> Alexander Sicular >> Solutions Architect >> Basho Technologies >> 9175130679 >> @siculars > > > > -- > С уважением, > Ершов Андрей > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: AAE Off
Right. AAE does not come for free. It consumes disk, memory and CPU. Depending on your circumstances it may or may not be advantageous for your system. On Tue, Feb 28, 2017 at 11:39 Matthew Von-Maszewski wrote: > Performance gains on write intensive applications. > > > On Feb 28, 2017, at 11:18 AM, al so wrote: > > > > Why would anyone disable AAE in riak 2.x? > > ___ > > riak-users mailing list > > riak-users@lists.basho.com > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > -- Alexander Sicular Solutions Architect Basho Technologies 9175130679 @siculars ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Object not found after successful PUT on S3 API
Hi Daniel,
Riak CS uses multi by default. By default the manifests are stored in leveldb
and the blobs/chunks are stored in bitcask. If you're looking to force
everything to level you should remove multi and use level as the backend
setting. As Luke noted elsewhere, this configuration hasn't been fully tested
and is not supported.
Off the top of my head, take a look at the email Martin (?) sent about his
modified level backend a few weeks ago for reasons why using level for data
chunks may not be the best idea at this time.
Thanks,
Alexander
@siculars
http://siculars.posthaven.com
Sent from my iRotaryPhone
> On Mar 10, 2017, at 10:50, Daniel Miller wrote:
>
> Hi Luke,
>
> Again, thanks for your help. We are currently preparing to move all objects
> into a new cluster using the S3 API. One question on configuration: currently
> I have "storage_backend = leveldb" in my riak.conf. I assume that on the new
> cluster, in addition to using the advanced.config you provided, I also need
> to set "storage_backend = multi" in riak.conf – is that correct?
>
> Referring back to the subject of this thread for a bit, I'm assuming your
> current theory for why the (most recent) object went missing is because we
> have a bad backend configuration. Note that that object went missing weeks
> after it was originally written into riak, and it was successfully retrieved
> many times before it went missing. Is there a way I can query riak to verify
> your theory that the manifest was overwritten? Russel Brown suggested: "I
> wonder if you can get the manifest and then see if any/all of the chunks are
> present?" Would that help to answer the question about why the object went
> missing? Can you provide any hints on how to do that?
>
> While bad configuration may be the cause of this most recent object going
> missing, it does not explain the original two objects that went missing
> immediately after they were PUT. Those original incidents happened when our
> cluster was still using bitcask/mutli backend, so should not have been
> affected by bad configuration.
>
> ~ Daniel
>
>> On Tue, Mar 7, 2017 at 3:58 PM, Luke Bakken wrote:
>> Hi Daniel,
>>
>> Thanks for providing all of that information.
>>
>> You are missing important configuration for riak_kv that can only be
>> provided in an /etc/riak/advanced.config file. Please see the following
>> document, especially the section to which I link here:
>>
>> http://docs.basho.com/riak/cs/2.1.1/cookbooks/configuration/riak-for-cs/#setting-up-the-proper-riak-backend
>>
>> [
>> {riak_kv, [
>> % NOTE: double-check this path for your environment:
>> {add_paths, ["/usr/lib/riak-cs/lib/riak_cs-2.1.1/ebin"]},
>> {storage_backend, riak_cs_kv_multi_backend},
>> {multi_backend_prefix_list, [{<<"0b:">>, be_blocks}]},
>> {multi_backend_default, be_default},
>> {multi_backend, [
>> {be_default, riak_kv_eleveldb_backend, [
>> {data_root, "/opt/data/ecryptfs/riak"}
>> ]},
>> {be_blocks, riak_kv_eleveldb_backend, [
>> {data_root, "/opt/data/ecryptfs/riak_blocks"}
>> ]}
>> ]}
>> ]}
>> ].
>>
>> Your configuration will look like the above. The contents of this file are
>> merged with the contents of /etc/riak/riak.conf to produce the configuration
>> that Riak uses.
>>
>> Notice that I chose riak_kv_eleveldb_backend twice because of the discussion
>> you had previously about RAM usage and bitcask
>> (http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-November/018801.html)
>>
>> In your current configuration, you are not using the expected prefix for the
>> block data. My guess is that on very rare occasions your data happens to
>> overwrite the manifest for a file. You may also have corrupted files at this
>> point without noticing it at all.
>>
>> IMPORTANT: you can't switch from your current configuration to this new one
>> without re-saving all of your data.
>>
>> --
>> Luke Bakken
>> Engineer
>> lbak...@basho.com
>>
>> --
>> Luke Bakken
>> Engineer
>> lbak...@basho.com
>>
>>> On Tue, Mar 7, 2017 at 6:47 AM, Daniel Miller wrote:
>>> Responses inline.
>>>
On Mon, Mar 6, 2017 at 3:04 PM, Luke Bakken wrote:
Hi Daniel,
Two questions:
* Do you happen to have an /etc/riak/app.config file present?
>>>
>>> No.
>>>
>>> Not sure if relevant, but I did notice that /etc/riak-cs/advanced.config
>>> does exist, which contradicts with what I said earlier. This is surprising
>>> to me because I did not create this file. Maybe it was created by the riak
>>> installer? Anyway, the content is:
>>>
>>> $ cat /etc/riak-cs/advanced.config
>>> [
>>> {riak_cs,
>>> [
>>> ]}
>>> ].
>>>
* On one of your Riak nodes, could you please execute the following
commands:
riak attach
rp(application:get_all_env(riak_kv)).
Copy the
Re: Object not found after successful PUT on S3 API
I apologize for the confusion and defer to Luke on all the technical Riak
things.
On Fri, Mar 10, 2017 at 11:16 Luke Bakken wrote:
> Just to clarify ...
>
> What Alexander is suggesting is what Daniel is currently using, and
> what I suspect may be causing Daniel's issues.
>
> If you wish to run a leveldb-only Riak CS cluster, you still *must*
> use the advanced.config file and the riak_cs_kv_multi_backend, and the
> other settings that I mention in my response and in the docs. Notice
> the multi_backend_prefix_list setting, for one thing.
>
> Daniel -
>
> The storage_backend setting in advanced.config will *override*
> storage_backend in riak.conf. If you wish to ensure the riak.conf
> setting is overridden, you may comment it out in that file.
>
> --
> Luke Bakken
> Engineer
> lbak...@basho.com
>
> On Fri, Mar 10, 2017 at 9:08 AM, Alexander Sicular
> wrote:
> >
> > Hi Daniel,
> >
> > Riak CS uses multi by default. By default the manifests are stored in
> leveldb and the blobs/chunks are stored in bitcask. If you're looking to
> force everything to level you should remove multi and use level as the
> backend setting. As Luke noted elsewhere, this configuration hasn't been
> fully tested and is not supported.
> >
> > Off the top of my head, take a look at the email Martin (?) sent about
> his modified level backend a few weeks ago for reasons why using level for
> data chunks may not be the best idea at this time.
> >
> > Thanks,
> > Alexander
> >
> > @siculars
> > http://siculars.posthaven.com
> >
> > Sent from my iRotaryPhone
> >
> > On Mar 10, 2017, at 10:50, Daniel Miller wrote:
> >
> > Hi Luke,
> >
> > Again, thanks for your help. We are currently preparing to move all
> objects into a new cluster using the S3 API. One question on configuration:
> currently I have "storage_backend = leveldb" in my riak.conf. I assume that
> on the new cluster, in addition to using the advanced.config you provided,
> I also need to set "storage_backend = multi" in riak.conf – is that correct?
> >
> > Referring back to the subject of this thread for a bit, I'm assuming
> your current theory for why the (most recent) object went missing is
> because we have a bad backend configuration. Note that that object went
> missing weeks after it was originally written into riak, and it was
> successfully retrieved many times before it went missing. Is there a way I
> can query riak to verify your theory that the manifest was overwritten?
> Russel Brown suggested: "I wonder if you can get the manifest and then see
> if any/all of the chunks are present?" Would that help to answer the
> question about why the object went missing? Can you provide any hints on
> how to do that?
> >
> > While bad configuration may be the cause of this most recent object
> going missing, it does not explain the original two objects that went
> missing immediately after they were PUT. Those original incidents happened
> when our cluster was still using bitcask/mutli backend, so should not have
> been affected by bad configuration.
> >
> > ~ Daniel
> >
> > On Tue, Mar 7, 2017 at 3:58 PM, Luke Bakken wrote:
> >>
> >> Hi Daniel,
> >>
> >> Thanks for providing all of that information.
> >>
> >> You are missing important configuration for riak_kv that can only be
> provided in an /etc/riak/advanced.config file. Please see the following
> document, especially the section to which I link here:
> >>
> >>
> http://docs.basho.com/riak/cs/2.1.1/cookbooks/configuration/riak-for-cs/#setting-up-the-proper-riak-backend
> >>
> >> [
> >> {riak_kv, [
> >> % NOTE: double-check this path for your environment:
> >> {add_paths, ["/usr/lib/riak-cs/lib/riak_cs-2.1.1/ebin"]},
> >> {storage_backend, riak_cs_kv_multi_backend},
> >> {multi_backend_prefix_list, [{<<"0b:">>, be_blocks}]},
> >> {multi_backend_default, be_default},
> >> {multi_backend, [
> >> {be_default, riak_kv_eleveldb_backend, [
> >> {data_root, "/opt/data/ecryptfs/riak"}
> >> ]},
> >> {be_blocks, riak_kv_eleveldb_backend, [
> >> {data_root, "/opt/data/ecryptfs/riak_blocks"}
> >> ]}
> >> ]}
> >> ]}
> >> ].
> >>
> >> Your configuration will look like the above. The contents of this file
> are merged with
Re: Does RiakKV require a lot of memory?
Also, when using default setting Riak will write three copies of your data to your cluster - even if you only have a cluster of one machine. -Alexander @siculars http://siculars.posthaven.com Sent from my iRotaryPhone > On Mar 27, 2017, at 11:25, Luke Bakken wrote: > > Hi Tsutomu, > > You should only use objects up to a maximum of 1MiB in size with Riak > KV. If you wish to store larger objects or files, please use Riak CS. > > -- > Luke Bakken > Engineer > lbak...@basho.com > >> On Sun, Mar 26, 2017 at 5:38 PM, wrote: >> Hello. >> >> The following problem occurred, Please give me on solution. >> >> (Problem) >> I registered a 10mb to 1000mb file. >> Then RiakKV hung up when the batch program was accessing files from 100 MB >> to 600 MB. >> RiakKV hung up due to insufficient memory. >> >> (Question) >> 1.The problem that occurred this time is the effect of data accessed to a >> 20-1000 MB file? >> Or is it another problem? >> >> 2.During RiakKV data update, when you want to access RiakKV data, >> do you need to do something special? >> Do you need a lot of memory? >> >> (Objects) >> file: 3,882,892 >> file size(total): 332.46GB(346,243,500,913byte) >> file objects: >> 1- 10MB = 3,800,000(all) >> 20- 50MB = 40 to 100 >> 100-1000MB = 50 >> >> (Riak Sever) >> OS:Red Hat Enterprise Linux Server release 6.7 (Santiago) >> （Linux patdevsrv02 2.6.32-573.el6.x86_64） >> CPU:Intel(R) Xeon(R) CPU E5640 @2.67GHz * 2 >> Memory:12GB >> Swap:14GB >> Network:1Gbps >> Disk: >> Filesystem Size Used Avail Use% Mounted on >> /dev/sda3 261G 67G 182G 27% / >> tmpfs 5.9G 300K 5.9G 1% /dev/shm >> /dev/sda1 477M 71M 381M 16% /boot >> /dev/sdb1 275G 243G 19G 93% /USR1 >> /dev/sdc1 1.7T 1.5T 76G 96% /USR2 >> /dev/sdd1 1.1T 736G 309G 71% /USR3 <<< Store >> /dev/sde1 1.1T 1.1T 18G 99% /USR4 >> /dev/sdf1 1.4T 1.1T 365G 74% /media/USB-HDD1 >> >> (Riak) >> RiakKV 2.2.0 (riak-2.2.0-1.el6.x86_64.rpm) >> >> (Riak Node) >> 1 Node. >> >> !!!We plan to add two nodes to the cluster at a later date.!!! >> >> (Java) >> Java1.8 (jre-8u121-linux-x64.rpm) >> >> (riak setting) >> /etc/riak/riak.conf >> storage_backend = leveldb >> leveldb.maximum_memory.percent = 50 >> object.size.maximum = 2GB >> listener.http.internal = 0.0.0.0:8098 >> listener.http.internal = 0.0.0.0:8098 >> platform_data_dir = /USR3/riak >> nodename = riak@... >> riak_control = on >> >> !!!Settings other than these remain the default.!!! >> >> (linux setting) >> /etc/security/limits.conf >> * soft nofile 65536 >> * hard nofile 65536 >> >> Thank you. >> >> Tsutomu Wakuda >> >> >> ___ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: RiakTS Query Question
I'm not 100% certain but I do not believe that is the case. Part of the reason for structured data is efficient retrieval. I believe the data is read but only the data selected leaves the leveldb backend, unselected data never leaves leveldb so there's no overhead when passing data from level to Riak or on the network. I defer to the engineering folks working on TS tho. -Alexander @siculars http://siculars.posthaven.com Sent from my iRotaryPhone > On Mar 29, 2017, at 15:08, Joe Olson wrote: > > Suppose I have the following table in RiakTS: > > > CREATE TABLE T1 ( > > idVARCHAR NOT NULL, > > eventtime TIMESTAMP NOT NULL, > > field2 SINT64, > > data BLOB NOT NULL, > > primary key (id, QUANTUM(eventtime, 365, 'd')),id) > > ) > > > Assume the BLOB field is close to the max size for a RiakTS BLOB value (~1MB) > > > Suppose I want to execute the following query: > > > Select id, eventtime, field2 from T1 where ((id = ID1) and (eventtime >= T1 > and eventtime < T2)) > > > I only want the SINT64 field, not the 1MB BLOB. > > > Am I paying for the bandwidth for the RiakTS cluster to pass around > (internally) the 1MB BLOB field just to get the SINT64 field? > > > If so, is there a way to avoid this, besides creating a second table without > the BLOB field? > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Is Riak dead?
In case you haven't seen it... https://www.theregister.co.uk/2017/07/13/will_the_last_person_at_basho_get_the_lights_oh_too_late/ Cheers, Alexander On Thu, Jul 13, 2017 at 1:56 PM, wrote: > Hmmm--- I wonder if anyone has given thought to a conference focused on the > future of open-source Riak? > > > > I'm sure there are many good ideas out there re: roadmap and governance. > > > > It's just too great not to be worth deep thought and prudent action. > > > > All the best, > > > > LRP > > > > > > -Original Message- > From: "Russell Brown" > Sent: Thursday, July 13, 2017 12:40pm > To: "Senthilkumar Peelikkampatti" > Cc: "riak-users" > Subject: Re: Is Riak dead? > > Hi Senthilkumar, > No Riak is not dead. It’s parent company, Basho is dead. And they kept the > keys. There are numerous forks that are being worked on. Hopefully something > that is a canonical, community backed fork will emerge. Past mailing list > posts show strong support from large commercial and govt. organisations. > Riak is open source and will continue to be critical infrastructure for > numerous organisation in its niche for some time. > > There are also a few support organisations springing up, as well as the > existing partners (Erlang Solutions, Trifork etc.) If you’re asking me, this > could be a very good thing for Riak in the medium to long term. It and it’s > users and community were served very badly by Basho these last couple of > years. It’s a little trodden under, but no way dead. Give it a little time > to bounce back. > > Cheers > > Russell > > > On 11 Jul 2017, at 08:15, Senthilkumar Peelikkampatti > wrote: > >> Is Riak dead? No new activities on Riak GitHub repo. >> >> https://github.com/basho/riak_kv/pulse/monthly >> https://github.com/basho/riak/pulse/monthly >> >> Thanks, >> Senthil >> ___ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Is Riak dead?
Abandon hope all ye who enter here. On Thu, Jul 13, 2017 at 3:15 PM, Tom Santero wrote: > RICON: A New Hope > > On Thu, Jul 13, 2017 at 4:00 PM, Russell Brown > wrote: >> >> We have talked about it. Let's do it! >> >> On Jul 13, 2017 7:56 PM, ll...@writersglen.com wrote: >> >> Hmmm--- I wonder if anyone has given thought to a conference focused on >> the future of open-source Riak? >> >> >> >> I'm sure there are many good ideas out there re: roadmap and governance. >> >> >> >> It's just too great not to be worth deep thought and prudent action. >> >> >> >> All the best, >> >> >> >> LRP >> >> >> >> >> >> -Original Message- >> From: "Russell Brown" >> Sent: Thursday, July 13, 2017 12:40pm >> To: "Senthilkumar Peelikkampatti" >> Cc: "riak-users" >> Subject: Re: Is Riak dead? >> >> Hi Senthilkumar, >> No Riak is not dead. It’s parent company, Basho is dead. And they kept the >> keys. There are numerous forks that are being worked on. Hopefully something >> that is a canonical, community backed fork will emerge. Past mailing list >> posts show strong support from large commercial and govt. organisations. >> Riak is open source and will continue to be critical infrastructure for >> numerous organisation in its niche for some time. >> >> There are also a few support organisations springing up, as well as the >> existing partners (Erlang Solutions, Trifork etc.) If you’re asking me, this >> could be a very good thing for Riak in the medium to long term. It and it’s >> users and community were served very badly by Basho these last couple of >> years. It’s a little trodden under, but no way dead. Give it a little time >> to bounce back. >> >> Cheers >> >> Russell >> >> >> On 11 Jul 2017, at 08:15, Senthilkumar Peelikkampatti >> wrote: >> >> > Is Riak dead? No new activities on Riak GitHub repo. >> > >> > https://github.com/basho/riak_kv/pulse/monthly >> > https://github.com/basho/riak/pulse/monthly >> > >> > Thanks, >> > Senthil >> > ___ >> > riak-users mailing list >> > riak-users@lists.basho.com >> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> >> >> ___ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> >> >> >> ___ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> > > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Is Riak dead?
I love your enthusiasm Lloyd. How about starting with a slack channel and take it from there... Things that would kinda need to happen for Riak to grow beyond where it is now: - complete rebranding. Basho is dead. The term "Riak" may be encumbered. - new everything. name. domain. github repo. bla bla bla. Instead of a framework, how can the Riak codebase actually become an Apache project? On Thu, Jul 13, 2017 at 8:06 PM, wrote: > Hi, > > > > If there is enough interest, I'd be happy to do the legwork in Boston and > put up a very modest contribution toward up-front cost with hope of recovery > through registrations. > > > > Boston has conference spaces of all sizes. I would imagine a conference on > the future of open-source Riak would be a fairly intimate group. I'm also > imagining a fairly bare-bones event to minimize cost of registration. > Registration cost should be just enough to cover the venue and nibbles. > > > > Just to kick off brain-storming: > > > > 1. Say two days--- Friday and Saturday > > 2. Friday: Invite short (20 minute) proposals on key topics > > a. Clarifying legal issues > > b. Technical road map > > c. Organizational framework; e.g. ala Apache Foundation or some such > > d. Marketing; e.g. expanding the open-source community > > 3. Saturday (or Saturday morning) > > Discussion sessions with goal of recommendations for moving forward > > > > How to get from here to there: > > > > 1. Is there enough interest on the mailing list? > > 2. How can we pin down an agenda? > > 3. Can we get corporate sponsorships to minimize registration? > > 4. How can we most widely promote the event? > > 5. What else? > > > > All the best, > > > > Lloyd > > > > > > -Original Message- > From: "Russell Brown" > Sent: Thursday, July 13, 2017 4:00pm > To: ll...@writersglen.com > Cc: "riak-users" , "Senthilkumar Peelikkampatti" > , "Russell Brown" > Subject: Re: Is Riak dead? > > We have talked about it. Let's do it! > > On Jul 13, 2017 7:56 PM, ll...@writersglen.com wrote: > > Hmmm--- I wonder if anyone has given thought to a conference focused on the > future of open-source Riak? > > > > I'm sure there are many good ideas out there re: roadmap and governance. > > > > It's just too great not to be worth deep thought and prudent action. > > > > All the best, > > > > LRP > > > > > > -Original Message- > From: "Russell Brown" > Sent: Thursday, July 13, 2017 12:40pm > To: "Senthilkumar Peelikkampatti" > Cc: "riak-users" > Subject: Re: Is Riak dead? > > Hi Senthilkumar, > No Riak is not dead. It’s parent company, Basho is dead. And they kept the > keys. There are numerous forks that are being worked on. Hopefully something > that is a canonical, community backed fork will emerge. Past mailing list > posts show strong support from large commercial and govt. organisations. > Riak is open source and will continue to be critical infrastructure for > numerous organisation in its niche for some time. > > There are also a few support organisations springing up, as well as the > existing partners (Erlang Solutions, Trifork etc.) If you’re asking me, this > could be a very good thing for Riak in the medium to long term. It and it’s > users and community were served very badly by Basho these last couple of > years. It’s a little trodden under, but no way dead. Give it a little time > to bounce back. > > Cheers > > Russell > > > On 11 Jul 2017, at 08:15, Senthilkumar Peelikkampatti > wrote: > >> Is Riak dead? No new activities on Riak GitHub repo. >> >> https://github.com/basho/riak_kv/pulse/monthly >> https://github.com/basho/riak/pulse/monthly >> >> Thanks, >> Senthil >> ___ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Is Riak dead?
> On Jul 14, 2017, at 11:32, Russell Brown wrote: > > What do you mean “encumbered”? Riak is the name of an Apache2 licensed open > source database, so it can continue to be used to describe that apache 2 > licensed database, please don’t spread FUD. You willing to invest time in a project that could be legally encumbered at some point in the future because lawyers? You willing to put that beyond the clown fiesta that is Basho? I don't think so. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Is Riak dead?
I'm not dancing on anyone's grave. Im a big fan of Riak, you know? I'm just pointing out that there is in fact a grave and if there is any hope for a Riak future the project will have to rebrand. I'm sorry, but no company in their right mind would seriously consider using Riak today if they weren't already using Riak. That's today, the story may change tomorrow. > On Fri, Jul 14, 2017 at 5:41 PM, Russell Brown wrote: > >> On 14 Jul 2017, at 23:36, Dave King wrote: >> >> You asked what difference the tm made. The tm invokes clause 6. So yes it >> means moving forward means a new name. That's one of the things to be >> considered. > > Agree, so if we’re sticking to the subject, “Is Riak dead?” the worst case > is: “No, but potentially the name could be some time in the future”, right? > > I asked rhetorically what difference the TM made. I’ve also read the license, > and taken legal advice. This was not in any way a surprise. > >> >> - Peace >> Dave >> >> >> On Fri, Jul 14, 2017 at 4:30 PM, Russell Brown wrote: >> >> On 14 Jul 2017, at 23:28, Dave King wrote: >> >>> No, but I can read. >> >> I read it too. I’m not sure what your point is. A day may come when the name >> has to change? >> >>> >>> - Peace >>> Dave >>> >>> >>> On Fri, Jul 14, 2017 at 4:27 PM, Russell Brown >>> wrote: >>> >>> On 14 Jul 2017, at 23:26, Dave King wrote: >>> >>>> 6. Trademarks. This License does not grant permission to use the trade >>>> names, trademarks, service marks, or product names of the Licensor, except >>>> as required for reasonable and customary use in describing the origin of >>>> the Work and reproducing the content of the NOTICE file. >>>> >>>> So At minimum I read that as there would need to be a new product name, >>>> and the docs for that product could say formerly known as Riak. >>> >>> You are lawyer? >>> >>>> >>>> - Peace >>>> Dave >>>> >>>> On Fri, Jul 14, 2017 at 4:22 PM, Russell Brown >>>> wrote: >>>> >>>> On 14 Jul 2017, at 23:17, Dave King wrote: >>>> >>>>> Looks like Basho has the trademark on Riak. >>>>> http://www.trademarkia.com/riak-77954950.html >>>> >>>> They sure do. And if you look at the Apache2 license you’ll see that it >>>> grants use of the name to identify the source. >>>> >>>> Look, I’m not a lawyer, and I’m guessing you’re not a lawyer. What >>>> difference does the name or trademark make? There are a bunch of active >>>> users of Riak, some of whom have made it clear they plan to continue using >>>> it. Dance on Basho’s grave all you want, but Riak isn’t dead yet. >>>> >>>>> >>>>> - Peace >>>>> Dave >>>>> >>>>> >>>>> On Fri, Jul 14, 2017 at 4:08 PM, Alexander Sicular >>>>> wrote: >>>>> >>>>> >>>>>> On Jul 14, 2017, at 11:32, Russell Brown wrote: >>>>>> >>>>>> What do you mean “encumbered”? Riak is the name of an Apache2 licensed >>>>>> open source database, so it can continue to be used to describe that >>>>>> apache 2 licensed database, please don’t spread FUD. >>>>> >>>>> You willing to invest time in a project that could be legally encumbered >>>>> at some point in the future because lawyers? You willing to put that >>>>> beyond the clown fiesta that is Basho? I don't think so. >>>>> ___ >>>>> riak-users mailing list >>>>> riak-users@lists.basho.com >>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Erlang Port Configuration
What Andrew said… AFAIK, the only thing you can not configure is which ip address the erlang vm listens on. I think it will listen on all available ip addresses (0.0.0.0). Note that this is different from the riak process itself, which can be limited to a specific ip. That's at least what I came up with last I looked into it about a year ago. I would love to hear otherwise. -Alexander Sicular @siculars http://siculars.posterous.com On Nov 30, 2011, at 3:01 PM, Andrew Thompson wrote: > On Wed, Nov 30, 2011 at 01:50:48PM -0600, Paul Gross wrote: >> I'm setting up a new cluster with an iptables firewall, and I'm >> trying to figure out which ports to open. It looks like I need 4369, >> 8099, 8098 (http), and a configurable range. Can I configure the >> range to be a single port, or is there a minimum number of required >> ports? What determines how many ports I need? >> > > http://wiki.basho.com/Network-Security-and-Firewall-Configurations.html > > Andrew > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Secondary Index Map and reduce order and performance
Do you get the results in both cases?
-Alexander Sicular
@siculars
http://siculars.posterous.com
On Nov 30, 2011, at 4:28 PM, Sajithkumar Kizhakkiniyil wrote:
> Hello
> Probably my understanding of M/R might be wrong. But I am getting drastic
> performance difference when running secondary index query on PB with map and
> reduce function in different order.
> If my understanding is correct a reduce phase with
> riak_kv_mapreduce.reduce_identity is needed for secondary index query. I
> added one map phase to get the value instead of the key
>
> But if I send the reduce before the map as you see in the map reduce payload
> JSON the values are return much faster than the other way. In my test it 251
> ms vs 700ms. Anyone can explain this behavior.
>
> Reduce before map (Faster)
> ---
> {"inputs":{"index":"PERFTEST_INDEX_NAME_bin","bucket":"_ITEST_SI_BUCKET","key":"PERFTEST_INDEX_VALUE"},"query":[{"reduce":{"arg":"{reduce_phase_only_1,
>
> true}","module":"riak_kv_mapreduce","language":"erlang","keep":false,"function":"reduce_identity"}},{"map":{"source":"function(value,keyData,arg){
> return [value.values[0].data]; }","language":"javascript","keep":true}}]}
>
> Map before reduce (Slower)
> --
> {"inputs":{"index":"PERFTEST_INDEX_NAME_bin","bucket":"_ITEST_SI_BUCKET","key":"PERFTEST_INDEX_VALUE"},"query":[{"map":{"source":"function(value,keyData,arg){
> return [value.values[0].data];
> }","language":"javascript","keep":true}},{"reduce":{"arg":"{reduce_phase_only_1,
>
> true}","module":"riak_kv_mapreduce","language":"erlang","keep":false,"function":"reduce_identity"}}]}
>
>
> This message is private and confidential. If you have received it in error,
> please notify the sender and remove it from your system.
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Secondary Index Map and reduce order and performance
Do you get the *same* results in both cases?
-Alexander Sicular
@siculars
http://siculars.posterous.com
On Nov 30, 2011, at 4:28 PM, Sajithkumar Kizhakkiniyil wrote:
> Hello
> Probably my understanding of M/R might be wrong. But I am getting drastic
> performance difference when running secondary index query on PB with map and
> reduce function in different order.
> If my understanding is correct a reduce phase with
> riak_kv_mapreduce.reduce_identity is needed for secondary index query. I
> added one map phase to get the value instead of the key
>
> But if I send the reduce before the map as you see in the map reduce payload
> JSON the values are return much faster than the other way. In my test it 251
> ms vs 700ms. Anyone can explain this behavior.
>
> Reduce before map (Faster)
> ---
> {"inputs":{"index":"PERFTEST_INDEX_NAME_bin","bucket":"_ITEST_SI_BUCKET","key":"PERFTEST_INDEX_VALUE"},"query":[{"reduce":{"arg":"{reduce_phase_only_1,
>
> true}","module":"riak_kv_mapreduce","language":"erlang","keep":false,"function":"reduce_identity"}},{"map":{"source":"function(value,keyData,arg){
> return [value.values[0].data]; }","language":"javascript","keep":true}}]}
>
> Map before reduce (Slower)
> --
> {"inputs":{"index":"PERFTEST_INDEX_NAME_bin","bucket":"_ITEST_SI_BUCKET","key":"PERFTEST_INDEX_VALUE"},"query":[{"map":{"source":"function(value,keyData,arg){
> return [value.values[0].data];
> }","language":"javascript","keep":true}},{"reduce":{"arg":"{reduce_phase_only_1,
>
> true}","module":"riak_kv_mapreduce","language":"erlang","keep":false,"function":"reduce_identity"}}]}
>
>
> This message is private and confidential. If you have received it in error,
> please notify the sender and remove it from your system.
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Delete operation not allowed. How to change it?
You can't delete a bucket. Period. You have to iterate over each key in a bucket. Cheers, Alexander @siculars on twitter http://siculars.posterous.com Sent from my iRotaryPhone On Dec 21, 2011, at 0:25, Jammy wrote: > > > Hi~@list, > > I'm new to riak. When I use HTTP API to delete a bucket, I got a response as > following. > HTTP/1.1 405 Method Not Allowed [Server: MochiWeb/1.1 WebMachine/1.9.0 > (someone had painted it blue), Date: Wed, 21 Dec 2011 05:24:33 GMT, > Content-Length: 0, Allow: HEAD, GET, PUT] > > Is there a way to make Delete operation allowed? > > > Thanks > Jammy > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Absolute consistency
When n=3 in a three node cluster, sometimes two of those n are on the same node. @siculars http://siculars.posterous.com Sent from my rotary phone. On Jan 5, 2012 11:07 AM, "Thomas Bakketun" wrote: > Hello, > > Riak is based on eventual consistency concept, but sometimes it's > desirable to have absolute consistency. I though it would be possible to > implement this on top of Riak with the help of pr and wr parameters of > fetch and store, but when testing this I sometimes get worrying results. > > I have at test setup with Riak nodes in three virtual machines running > on my desktop machine. The test bucket has n_val=3 and allow_mult=true. > > On my desktop machine I constantly read from every node like this: > watch curl -v "'http://192.168.100.21:8098/riak/test/foo?pr=quorum'" > > When I pause two of the virtual machines, the remaining one starts to > fail with error message "PR-value unsatisfied: 1/2", just as expected. > But occasionally it will returns 200 OK for about 10 seconds, before it > starts returning "PR-value unsatisfied: 1/2". I can't predictably > reproduce this situation, but it's seems to happen when I resume a > machine that was paused when the other machines where running too. > > Pausing virtual machines to simulate downtime might be a bit > unrealistic. Still it concerns me that Riak can return stale data. Here > is an example: > > First I wrote a value when all nodes where up. Then I pause one of the > nodes and wrote a new value. Then I paused the two remaining nodes and > resume the first node. For a few seconds the first node then returned > 200 OK with the old value when though I supplied pr=quorum. > > I have also been able to do successful writes with w, dw and pw=quorum > with only one machine running. I thing the node also returned 200 OK > when the object was read with pr=quorum right after. > > The tests where done with Riak 1.0.0-1 running on Ubuntu 10.10 64 bit. > > -- > Thomas Bakketun, Senior software developer > T. +47 21 53 69 40 > Copyleft Solutions > www.copyleftsolutions.com > > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Absolute consistency
you're right. there should be better notice about this. I think the number of physical machines needs to be greater than n to keep this from happening. i also think I've seen a specific algorithm that is floating around that defines this behavior but i can't conjure it up at the moment. i also think the recommended default config is four nodes. and if its not it should be. -Alexander Sicular @siculars On Jan 5, 2012, at 2:44 PM, Tim Robinson wrote: > Ouch. > > I'm shocked that is not considered a major bug. At minimum that kind of stuff > should be front and center in their wiki/docs. Here I am thinking n 2 on a 3 > node cluster means I'm covered when in fact I am not. It's the whole reason I > gave Riak consideration. > > Tim > > -Original Message- > From: "Runar Jordahl" > Sent: Thursday, January 5, 2012 12:13pm > To: "Thomas Bakketun" > Cc: riak-users@lists.basho.com > Subject: Re: Absolute consistency > > As Alexander pointed out, N does not mean "a separate PC". There was a > thread about this earlier: > > http://lists.basho.com/pipermail/riak-users_lists.basho.com/2011-February/003316.html > > Kind regards > Runar Jordahl > blog.epigent.com > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > Tim Robinson > > > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Absolute consistency
shocked? dude. chillax. shocked would be like finding out the pope's gots kids. or we didn't land on the moon. or did we? there is so much magic abstraction going on here and just sending this email to you I'm shocked every time the anything actually works. -Alexander Sicular @siculars On Jan 5, 2012, at 2:44 PM, Tim Robinson wrote: > Ouch. > > I'm shocked that is not considered a major bug. At minimum that kind of stuff > should be front and center in their wiki/docs. Here I am thinking n 2 on a 3 > node cluster means I'm covered when in fact I am not. It's the whole reason I > gave Riak consideration. > > Tim > > -Original Message- > From: "Runar Jordahl" > Sent: Thursday, January 5, 2012 12:13pm > To: "Thomas Bakketun" > Cc: riak-users@lists.basho.com > Subject: Re: Absolute consistency > > As Alexander pointed out, N does not mean "a separate PC". There was a > thread about this earlier: > > http://lists.basho.com/pipermail/riak-users_lists.basho.com/2011-February/003316.html > > Kind regards > Runar Jordahl > blog.epigent.com > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > Tim Robinson > > > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: missing writes / inconsistent number of keys read after a bulk write
Not sure how your binding does it but MyBucket.keys() should be calling the
list keys feature of riak via the streaming mode which chunks the key reply
instead of buffering and sending one entire list of keys.
@siculars on twitter
http://siculars.posterous.com
Sent from my iRotaryPhone
On Jan 9, 2012, at 17:39, Karthik K wrote:
> (Changing subject to reflect the problem better and reposting ).
>
> Any idea , based on the configuration inline as to what explains the
> inconsistency number of keys read after a bulk write ( say 1M of payload 1000
> bytes ).
>
> Any appropriate write flush setting , that is missed on the client / server
> side ?
>
>
> --
> Karthik.
>
>
> On Fri, Jan 6, 2012 at 5:54 PM, Karthik K wrote:
> Further,
> ulimit -n is 10K on the box.
>
>
> # tail -100 /var//log/riak/erlang.log.1
>
>
> 17:26:26.960 [info] alarm_handler: {set,{system_memory_high_watermark,[]}}
> /usr/lib/riak/lib/os_mon-2.2.6/priv/bin/memsup: Erlang has closed.
> Erlang has closed
> 17:26:34.443 [info] alarm_handler: {clear,system_memory_high_watermark}
>
>
> So - is there a commit / flush setting that is missed when it comes to high
> volume writes ?
>
>
> On Fri, Jan 6, 2012 at 5:37 PM, Karthik K wrote:
> I am using Riak with LevelDB as the storage engine.
>
> app.config:
>
> {storage_backend, riak_kv_eleveldb_backend},
>
>
> {eleveldb, [
> {data_root, "/var/lib/riak/leveldb"},
> {write_buffer_size, 4194304}, %% 4MB in bytes
> {max_open_files, 50}, %% Maximum number of files open at once per
> partition
> {block_size, 65536}, %% 4K blocks
> {cache_size, 33554432}, %% 32 MB default cache size per-partition
> {verify_checksums, true} %% make sure data is what we expected it
> to be
> ]},
>
>
>
>
> I want to insert a million keys into the store ( into a given bucket ) .
>
> pseudo-code:
> riakClient = RiakFactory.pbcClient();
> myBucket = riakClient.createBucket("myBucket").nVal(1).execute();
> for (int i = 1; i <= 100; ++i) {
> final String key = String.valueOf(i);
> myBucket.store(key, new String(payload)).returnBody(false);
> }
>
>
> after this operation, when I do:
>
>int count = 0;
>for (String key : myBucket.keys() ) {
> ++count;
>}
>return count;
>
> This returns a total of 14K keys, while I was expecting close to 1 million or
> so.
>
> I am using riak-java-client (pbc).
>
> Which setting / missing client code can explain the discrepancy ? Thanks.
>
>
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak in the cloud?
On Jan 18, 2012, at 11:25 AM, David Smith wrote: > In addition, there is no guarantee that multiple > instances of the virtual host are on independent hardware hosts, so > you may also be (unknowingly) trading off availability. > I run my virtual stuff on Linode. When setting up a number of virtual machines in succession there does exist the possibility that they will allocate those vm's to the same physical host. That is why you check diligently. If there is an overlap it is a simple support ticket to get you sorted. -alexander ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: secondary index read with limit?
Afaik, range queries the likes of which you suggest are not possible in riak because riak is an inherently unordered system. That being said, it is a major cause of concern for those looking at riak as a potential solution. One ultimately needs to implement a secondary ordered index elsewhere to achieve desired results. I pair riak with redis, like pairing a sizzling steak with a fine red wine. -Alexander Sicular @siculars On Jan 25, 2012, at 1:08 PM, Michael Radford wrote: > I'm wondering if there's any way to achieve a limited secondary index > read in Riak, i.e., up to N records in some key range. With that > primitive, it seems like it would be possible for the user to > implement things like pagination, incremental joins over multiple > indices, etc. (For efficiency I guess it would have to be something > like "up to N records from each vnode"?) > > Looking at the source code, I think the answer is no, but since range > queries are already implemented using iterators, it doesn't seem like > it would be too hard to add. Can anyone comment on the roadmap for a > feature like this, and/or the possibility of getting a patch accepted? > > Mike > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Optimal cluster configuration?
What backend are you using? Bitcask (default) or leveldb? What is your N val (3 default)? Do you have search enabled? The only change you mentioned would result in a four fold decrease in virtual nodes per physical host. That would be interesting to see how it plays out. I would imagine there would be less on disk fragmentation with fewer vnodes per machine. Often in the end, the culprit is disk io. There are backend specific tweaks that may be helpful. Riak schedules compaction at different thresholds. Updating a bunch of records in short order may be triggering that. Just some thoughts. I'm sure the Basho folk will be able to offer more specific help. Cheers, -Alexander Sicular @siculars On Jan 31, 2012, at 3:03 PM, Sarang Deshpande wrote: > Hi, > I am working on setting up a riak cluster as a KV storage for ~150Million > documents with average document size being 4K. We started using 6 nodes (each > node: 32GB, 8 quad cores) in a cluster with partition size of 256 (planning > to take it down to 64 as next experiment). While the lookup is fast enough, > **bulk** updating/loading the documents in this cluster with 6 clients and > 100 connections each is quite slow (takes hours to update ~30Million docs). > Is there a better cluster configuration that we should try to speed up the > process? > > Thanks, > ~Sarang > > > > > > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: load balancing protocol buffers clients
Haproxy. Using tcp settings for protobuf. @siculars http://siculars.posterous.com Sent from my rotary phone. On Feb 1, 2012 12:12 PM, "Marc Campbell" wrote: > Hi everyone, > > I'm looking to put a load balancer in front of my riak cluster but not > sure which one will work. I'm using both the HTTP interface and the > protocol buffers client. My connections from the PB client are kept open > for a long time, in fact they are pooled in my app and reused. > > Does anyone have any recommendations for a load balancer for the PB client? > > Marc > > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Sanitizing user input for Search
Hello All, Are there limitations as to character sets or other special characters (perhaps a guide or docs) that I should be actively filtering out of user generated searches that will be passed to Riak Search? Yes, standard filtering rules apply, but are there any specific "gotchas" that people have come across when implementing riak search? A rough sketch of my flow: Browser > nodejs > riak search > mapreduce Tia, Alexander @siculars on twitter http://siculars.posterous.com Sent from my iRotaryPhone ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Is riak the right choice to log and retrieve log data?
Hello Heiner, Looks like a very interesting project. Would love to hear how it turns out. Do let us know what you end up deploying. Here are some of my thoughts. Between secondary indexes and/or search you should not have a problem retrieving a "blip page" of data. Certainly do not use the bitcask backend due to in memory key index. You would definitely need to use leveldb. Data bloat. Your 200B will triple on disk because Riak writes ~400B of data for itself per key (not exactly certain of the numbers in the post 1.0 branch). For this reason alone I'm not sure Riak is optimal for collection of large quantities of small values. The sweetspot is in the multi KB range where the overhead becomes some smaller % of total data. Ring state changes late in your collection cycle will result in a significant amount of data rebalancing which will obviously take a while. I would overdeploy more nodes early on and replace them in-place, over time to take advantage of newer hardware. I feel that although a lot of great work has already been done, there is still more work to be done on the rebalancing mechanics and algorithm to make this less of a pain point for larger systems. I would consider evaluating Cassandra or HBase for this project as well. Cheers, Alexander @siculars on twitter http://siculars.posterous.com Sent from my iRotaryPhone On Feb 6, 2012, at 5:22, Heiner Bunjes wrote: > I need a database to log and retrieve sensor data. > > Is riak the right solution for this task and if, how? > If not, which other DB system might be a better fit? > > I know that before the implementation of secondary indices riak probably was > not suitable for this kind of task. I just hope it is now. > > > > The details are as follows: > > > > Glossary > > - Node = A computer on which an instance of the database > is running > > - Blip = one data record send by a sensor > > - Blip page = The sorted list of all blips for a specific sensor > and a specific time range. > > > The scale is as follows: > > (01) 10E6 sensors deliver 1 blip every 100 seconds > -> Insert rate = 10 kiloblip/s > -> Insert rate ~ 315 gigablip/Year > > (02) They have to be stored for ~3 years > -> Size of database = 1 terablip > > (03) Each blip has about 200 bytes > -> Size of database = 200TB > > (04) The system will start with just 10E4 sensors but will > soon increase upto the described volume. > > > The main operations on the data are: > > (05) Add the new blips to the database > (written blips are never changed)! > > (06) Return all blips for sensor X with a timestamp > between timestamp_a and timestamp_b! > With other words: Return a blip page. > > (07) Return all the blips specified in (06) ordered > by timestamp! > > (08) Delete all blips older than Y! > > > Further the following is true: > > (09) Each added blip is clearly (without ambiguity) identifiable by > sensor_id+timestamp. > > (10) 99.9% of the blips are inserted in > chronological order, the rest is not. > > (11) The database system MUST be free and open source. > > (12) The DB SHOULD be easy to administrate. > > (13) All data MUST still be writable and readable while less > then the configurable number N of nodes are down (unexpectedly). > > (14) The mechanisms to distribute the data to the available > nodes SHOULD be handled by the database. > This means that the database SHOULD automatically > redistribute the data when nodes are added or removed. > > > > > Many thanks in advance > Heiner > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak and Node.js
I use riak-js a bunch. -Alexander ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak had overload My cpu
It could be any number of things... You really need to give us some more data How many nodes, what is the ring size, which backends , etc. @siculars http://siculars.posterous.com Sent from my rotary phone. On Feb 10, 2012 12:00 AM, "kser" wrote: > I had a 8 core cpu on server, used to be running lower than 100% of the > CPU, > but recently has increase to 700%-800% and making all my other process > being > slow. > Any one had experience about it and how can i optimize it? > > -- > View this message in context: > http://riak-users.197444.n3.nabble.com/Riak-had-overload-My-cpu-tp3731675p3731675.html > Sent from the Riak Users mailing list archive at Nabble.com. > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Is Riak a good solution for this problem?
Sure, as long as your working set fits in ram or you set up a sharing system in front of your redis cache. -Alexander Sicular @siculars On Feb 13, 2012, at 8:31 AM, Shuhao Wu wrote: > What about caching with redis? That seems to be a popular option as well. > > Shuhao > > > On Sun, Feb 12, 2012 at 3:49 PM, Aphyr wrote: > On 02/12/2012 03:27 AM, Marco Monteiro wrote: > I'm considering Riak for the statistics of a site that is approaching > a billion page views per month. The plan is to log a little > information about each the page view and then to query that data. > > Honestly, I wouldn't use stock Riak for this; the MR times will become > prohibitive over billions of records. However, jrecursive has been solving a > very similar set of problems at Showyou, and wrote Mecha, a solr/leveldb > backend for Riak. Mecha provides fast, distributed querying on top of Riak's > redundancy/distribution. There's a talk at the next Riak Meetup on the 23rd, > and we should be releasing then as well. > > http://www.meetup.com/San-Francisco-Riak-Meetup/events/51287272/ > > --Kyle > > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Search and Query [UPDATES] in RIAK
On Feb 18, 2012, at 9:04 AM, Rune Skou Larsen wrote: > On 18-02-2012 12:09, Rajat Mathur wrote: >> Also what are the methods for implementing where clause in it. > But there are other ways. Consider if you can define your key in a way, so > you get the right resultset from doing a range-search from key1 to key2. You > can also consider storing your data with secondary indeces for looking up on. > It is my understanding, that riak does not (yet) support queries combining > multiple secondary indeces in AND or OR searches. You may not be able to do composite queries like AND or OR via the secondary index feature (2i) but you can do it via standard search. Check: http://wiki.basho.com/Riak-Search---Querying.html#Boolean-Operators-%E2%80%93-AND%2C-OR%2C-NOT http://wiki.basho.com/MapReduce-Search-2i-Comparison.html Best, Alexander___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Windows
I'm sure it would take a while but I'm just gonna drop two reasons that
immediately come to mind. Bitcask and leveldb. File access, io scheduling and
the like are different between *nix and windows. I imagine it would take quite
a while to qualify just those two components for production use in a windows
environment.
On the flip side, I've been on this list a while and I think this is the first
time I've seen interest in running riak on windows. I'm generally curious what
the use case for that would be vs an *nix based deployment.
Salutations and whatnot,
-Alexander Sicular
@siculars
On Feb 20, 2012, at 1:36 AM, Aleksandr Vinokurov wrote:
> Hello, Jared
>
> If it wouldn't take long, can you pinpoint the major obstacles why this lack?
>
> With best regards,
> Aleksandr
>
> On 19 February 2012 19:33, Jared Morrow wrote:
>
> Aleksandr,
>
> We currently do not support building or running Riak on Windows. We do
> support or have successfully been run on most Linux distributions, Mac OS X,
> *BSD, Solaris, and SmartOS.
>
> Thanks,
> Jared
>
>
> On Feb 19, 2012, at 4:46 AM, Aleksandr Vinokurov wrote:
>
>>
>>
>> Hello all!
>>
>> I'm stuck with the quest: why RIAK is not building ``make all'' on windows
>> (with cygwin).
>>
>> samsung@Kryzhovnik /cygdrive/c/erlang/riak-1.0.3
>> $ make all
>> fatal: Not a git repository (or any of the parent directories): .git
>> ./rebar get-deps
>> fatal: Not a git repository (or any of the parent directories): .git
>> fatal: Not a git repository (or any of the parent directories): .git
>> fatal: Not a git repository (or any of the parent directories): .git
>> ==> cluster_info (get-deps)
>> ==> lager (get-deps)
>> ==> poolboy (get-deps)
>> ==> protobuffs (get-deps)
>> ==> basho_stats (get-deps)
>> ==> riak_sysmon (get-deps)
>> ==> mochiweb (get-deps)
>> ==> webmachine (get-deps)
>> ==> riak_core (get-deps)
>> ==> riakc (get-deps)
>> ==> luke (get-deps)
>> ==> erlang_js (get-deps)
>> ==> meck (get-deps)
>> ==> bitcask (get-deps)
>> ==> merge_index (get-deps)
>> ==> ebloom (get-deps)
>> ==> eper (get-deps)
>> ==> eleveldb (get-deps)
>> ==> sext (get-deps)
>> Pulling edown from {git,"git://github.com/esl/edown.git","HEAD"}
>> ERROR: 'get-deps' failed while processing c:/erlang/riak-1.0.3/deps/sext:
>> {'EXIT',{{case_clause,{error,127}},
>> [{rebar_utils,sh,2,[]},
>> {rebar_deps,vcs_client_vsn,3,[]},
>> {rebar_deps,source_engine_avail,2,[]},
>> {rebar_deps,require_source_engine,1,[]},
>> {rebar_deps,use_source,2,[]},
>> {rebar_deps,'-get-deps/2-lc$^0/1-0-',1,[]},
>> {rebar_deps,'get-deps',2,[]},
>> {rebar_core,run_modules,4,[]}]}}
>> Makefile:14: recipe for target `deps' failed
>> make: *** [deps] Error 1
>>
>>
>> Seems I should read about Rebar...
>>
>> I'm following a
>> http://wiki.basho.com/Building-a-Development-Environment.html page, I'm new
>> to Erlang and I'm lack of understanding why RIAK is not running under
>> Windows? Erlang is running, but RIAK is not -- why?
>>
>> --
>> Александр Винокуров
>> +7 (921) 982-21-43
>> @aleksandrvin
>>
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
>
> --
> Александр Винокуров
> +7 (921) 982-21-43
> @aleksandrvin
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak for Messaging Project Question
> > Actuallly, I think my math was faulty, it's actually 800 billion keys, so > that's many more servers than expected. A few billion? Pssh. You know what's cool? A Trillion. A few things jump off the page from the comments on this thread: Static allocation of ring_creation_size. Once you set it you can not change it. Although I'm sure that is on the drawing board. As Jeremiah pointed out, that means you will have a ton of vnodes on each physical machine while your cluster is still small. Data migration. As you add nodes to the cluster some fraction of the data will have to move around the ring. This could be a lot of data in your case. N-val. I'm not entirely sure about this but if you think about how riak does its map/reduce, search and index querying you have to think about how wide a net riak casts to get results. Is querying more nodes good or bad? And how good or bad is it in different cluster sizes. And how does that change with the number of vnodes on each physical node. I'm not sure but I think it should be explored. Frankly, the hardware at play does not really factor into my decision on whether or not to use riak. I think the only real question is how riak as a system accommodates the kind of data you want to collect and the ways in which you want to access it. The physical implementation is simply some scale function in regards to performance. Ie. more money = more performance. Would love to hear where you go with this. Cheers, Alexander___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Questions on configuring public and private ips for riak on ubuntu
this is a "Very Bad" idea. do not expose your riak instance over a public ip address. riak has no internal security mechanism to keep people from doing very bad things to your data, configuration, etc. -Alexander Sicular @siculars On Mar 5, 2012, at 12:43 AM, Tim Robinson wrote: > Hello all, > > I have a few questions on networking configs for riak. > > I have both a public ip and a private ip for each riak node. I want Riak to > communicate over the private ip addresses to take advantage of free > bandwidth, but I would also like the option to interface with riak using the > public ip's if need be (i.e. for testing / demo's etc). > > I'm gathering that the way people to this is by setting up app.config to use > ip "0.0.0.0" to listen for all ip's. I'm also gathering vm.args needs to have > a unique name in the cluster so I would need to use the hostname for the > -name option (i.e. r...@www.fake-node-domain-name-1.com). > > My hosts file would contain: > > 127.0.0.1 localhost.localdomain localhost > x.x.x.xwww.fake-node-domain-name-1.commynode-1 > > > where x.x.x.x is the public ip not the private. > > This is where I start to get lost. > > As it sits, if I attempt to join using the private ip's i will get the > unreachable error - yet I can telnet connect to/from the equivalent nodes. > > So I could add a second IP to the hosts file, but since I need to keep the > public one as well, how is that riak is going to use the private ips for > gissip ring, hinted hand-off, ... etc etc. > > There's obviously some networking basics I am missing. > > Any guidance from those of you who have done this? > > Thanks. > Tim > > > > > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Problems writing objects to an half full bucket
On Mar 6, 2012, at 10:19 AM, David Smith wrote:
> On Mon, Mar 5, 2012 at 9:55 PM, Marco Monteiro wrote:
>
>> I'm using riak-js and the error I get is:
>>
>> { [Error: socket hang up] code: 'ECONNRESET' }
>
> That is a strange error -- are there any corresponding errors in
> server logs? I would have expected a timeout or some such...
That is a standard network error in nodejs that bubbles up the stack. Same sort
of error you get in the redis nodejs library when you cant connect to redis.
It's a connection thing. Riak-js library is set to connect to the default riak
port on localhost. If that has changed or the service is down you'll get that
sort of error.
-alexander
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Simpler, many m/r phases, or complex, fewer m/r phases?
I would probably say complex/fewer MR phases but I guess it would depend on the compute complexity of your functions (in order to take advantage of parallelism/more compute cores). My reasoning is that every time you Map you are reading the full value from disk. More Maps = more disk i/o. Not to mention the erlang to js overhead if you are running js functions. Please report your findings! Best, -Alexander Sicular @siculars On Mar 14, 2012, at 6:16 PM, Jonathan Langevin wrote: > What is better for performance in Riak? > More phases with simpler logic, or less phases with more complex logic? > > For instance, if I want to check 10 different fields of the result objects, > using 10 different functions, should I combine that all down into 1-2 m/r > phases, or run as 10 different m/r phases? > > I would think more phases would suggest that the workload could be > distributed across various nodes more easily, but few phases would mean that > the values wouldn't have to be processed as many times... > > > Jonathan Langevin > Manager, Information Technology > Loom Inc. > Wilmington, NC: (910) 241-0433 - jlange...@loomlearning.com - > www.loomlearning.com - Skype: intel352 > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Substring Search
Um... Have you tried wildcards? Like "basket*"? Cause I'm using that now against Riak Search and it's working for me, although my data set is not large in the slightest. @siculars on twitter http://siculars.posterous.com Sent from my iRotaryPhone On Mar 25, 2012, at 23:56, Hursh Prasad wrote: > Yes I would like a similar setup as well and checked into ElasticSearch, I am > searching on short phrases less then 200 characters I do not want to miss > trailing words in compound words like ball in basketball. > > It seems the only option I have is SQL databases that support regular > expressions (please correct me if I am wrong). > > I think that is unfortunate, I prefer the performant index/querying, > replication ease, and non-impedance of JSON document store capability of > NoSql db's. > > Hursh > > On Sun, Mar 25, 2012 at 10:03 PM, John D. Rowell wrote: > We use ElasticSearch for everything search related, even when the persistence > layer happens in Riak, Redis or MongoDB. If you're using JSON to store your > Riak documents it's just a matter of storing the documents on ElasticSearch > also (also HTTP/JSON, with a REST API) and you get fully clustered, sharded, > cloud-aware, extremely fast Lucene-compatible searches with very little > effort. > > -jd > > Em 25 de março de 2012 17:28, Hursh Prasad > escreveu: > Yeah I am talking Riak Search, I did not see anything or any approaches. > > It seems like there isn't a good NoSQL solution out there. > > On Sun, Mar 25, 2012 at 4:17 PM, Erik Søe Sørensen wrote: > Are you talking about plain Riak, or using secondary indexes, or of Riak > Search? > > The question makes most sense in context of Riak Search. > If the page > http://wiki.basho.com/Riak-Search---Querying.html > is up-to-date, then the answer appears to be 'no' - wildcards (*,?) are > supported, but >"Currently, the wildcard must come at the end of the term in both cases." > i.e. only prefix queries can be used, not queries of arbitrary substrings. > > Trusting that someone will correct me if I'm wrong, > > /Erik > > > Mobile: + 45 2636 1755 | Skype: eriksoesorensen | Twitter: @eriksoe > Trifork A/S | Margrethepladsen 4 | DK- 8000 Aarhus C | Phone : +45 8732 > 8787 | www.trifork.com > > From: riak-users-boun...@lists.basho.com [riak-users-boun...@lists.basho.com] > On Behalf Of Hursh Prasad [hu...@theeggofcolumbus.com] > Sent: 25 March 2012 22:04 > To: riak-users@lists.basho.com > Subject: Re: Welcome to the "riak-users" mailing list > > Does Riak support (or what are the options for) substring search? > > Example "ball" or "bal" in basketball? > > Thanks, > Hursh > > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: [ANN] Riak CS
Awesome. Grats on the release and all the hard work that went into it. 1. I heard differing things when I asked in other forums but I wanted to get an answer here, does Riak CS support range requests? Aka give me bytes 100-200 from this 3MB file. 2. Pricing. Is Riak CS priced like S3 in terms of storage volume, bandwidth and lookups? Thank! -Alexander @siculars on twitter http://siculars.posterous.com Sent from my iRotaryPhone On Mar 27, 2012, at 20:54, Mark Phillips wrote: > Hi All, > > I just wanted to take a second and share some details on a new Riak-based > product that Basho announced today. > > Riak, the open source database, has long served as the basis for our > commercial offerings. Until this morning, RiakEDS [1] was the only member of > that product family, with Multi Data Center Replication being the primary > distinguishing feature between stock Riak and the EDS product. > > Today we introduced something called Riak CS [2], a closed-source product > built on top Riak that provides an S3 API. Some of the main features of Riak > CS: > > * S3 REST API Compatibility > * Multi Tenancy > * Pluggable Auth > * Large Object Support > * Per-Tenant Usage Visibility (for billing and metering) > * Multi Data Center Replication is also part of Riak CS > > Thanks. Let me know if you have any questions. > > Mark > > [1] http://basho.com/products/riak-enterprise/ > [2] http://basho.com/products/riakcs/ > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: [ANN] Riak CS
On Mar 27, 2012, at 9:59 PM, Andrew Thompson wrote: > On Tue, Mar 27, 2012 at 09:55:17PM -0400, Alexander Sicular wrote: >> Awesome. Grats on the release and all the hard work that went into it. >> >> 1. I heard differing things when I asked in other forums but I wanted to get >> an answer here, does Riak CS support range requests? Aka give me bytes >> 100-200 from this 3MB file. > > As Reid Draper said on HN, yes it does support range queries. I got a DM from someone else at Basho that said otherwise. >> 2. Pricing. Is Riak CS priced like S3 in terms of storage volume, bandwidth >> and lookups? > > Pricing is per-node. I believe you can cram as many CPUs, drives and RAM > info a node as you want and pay the same flat per-node price. That's awesome. My own private S3 or someone else will host one for me. Thanks! ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
