Re: system metrics monitoring

2014-12-09 Thread Shane McEwan
G'day! I use Zabbix[1] which will monitor most of those metrics out of the box (and much, much more). Cacti, Munin, Nagios and Collectd are also good alternatives. [1] http://www.zabbix.com/ Shane. On 09/12/14 06:23, pragya jain wrote: hello all! As mentioned at statistics and monitoring

Re: Upgraded riak 1.4.9 is pegging the CPU

2014-06-05 Thread Shane McEwan
On 05/06/14 16:20, Alain Rodriguez wrote: Hi all, I upgraded 1 of 9 riak nodes in a cluster last night from 1.4.0 to 1.4.9. The rest are running 1.4.0. Ever since I am seeing the upgraded node, riak01 consuming a significantly larger percent of CPU and the PUT times on it have gotten

Re: What do you do when Riak freezes?

2014-03-20 Thread Shane McEwan
On 19/03/14 20:56, Michael Dillon wrote: I've run into a problem with Riak freezing completely on one node running on Ubuntu 12.04 LTS on a XEN VM (EC2). If I ssh into the node and run ps ax that shell session also freezes. I also tried another ssh session with netstat -lnp to see if I could

Re: Client interface

2013-12-18 Thread Shane McEwan
On 18/12/13 16:29, Nicholas Wieland wrote: Hi, I'm a riak newbie and I'm planning to use it for my next project. It would be very useful to have a way to see inside buckets graphically (like phpmyadmin), a project similar to riak-control but for data. Does anybody know of something like this?

Re: restarted node tells me about to handoff partitions (changing format to v1)

2013-12-16 Thread Shane McEwan
On 16/12/13 09:56, Simon Effenberg wrote: after an upgrade from 1.3.1 to 1.4.2 I changed the object format from v0 to v1 and began a rolling restart. But restarting the first node shows me some weird outputs: 'riak@10.46.109.201' waiting to handoff 132 partitions this is the restarted node and

Re: max_files_limit and AAE

2013-11-12 Thread Shane McEwan
On 12/11/13 16:08, Dave Brady wrote: Is there command I can use in the CLI that gives the number of open files? sudo ls -1 /proc/`pidof beam.smp`/fd | wc -l You say you have 42,000 .sst files. Does that include AAE? If not, AAE could push you over the edge. Shane.

Re: Riak/LevelDB Memory Usage

2013-09-13 Thread Shane McEwan
as vnodes migrate to and from a given node (no need to reserve 50%) … you just say total memory you want to allocate to Riak and walk away. Matthew On Sep 6, 2013, at 12:10 PM, Shane McEwan sh...@mcewan.id.au wrote: G'day! Our Riak nodes have 48GB of RAM in them. When we installed Riak 1.2.0

Riak/LevelDB Memory Usage

2013-09-06 Thread Shane McEwan
G'day! Our Riak nodes have 48GB of RAM in them. When we installed Riak 1.2.0 on them we tuned the LevelDB settings as per the Parameter Planning section in http://docs.basho.com/riak/1.2.0/tutorials/choosing-a-backend/LevelDB/ {eleveldb, [ {write_buffer_size_min, 31457280},

Re: Riak 1.3.1 Errors

2013-08-07 Thread Shane McEwan
if those messages are something that are safe to ignore. -Jared On Mon, Aug 5, 2013 at 8:43 AM, Shane McEwan sh...@mcewan.id.au wrote: G'day! Nearly every day since upgrading to Riak 1.3.1 I've been seeing the following errors on random nodes: 2013-08-05 01:00:05.775 [error] 0.212.0 Supervisor

Riak 1.3.1 Errors

2013-08-05 Thread Shane McEwan
G'day! Nearly every day since upgrading to Riak 1.3.1 I've been seeing the following errors on random nodes: 2013-08-05 01:00:05.775 [error] 0.212.0 Supervisor riak_pipe_fitting_sup had child undefined started with riak_pipe_fitting:start_link() at 0.26698.882 exit with reason noproc in

Re: Help with local restore for dev enviroment

2013-07-10 Thread Shane McEwan
On 09/07/13 22:24, Mark Wagner wrote: Hey all, I'm new to riak and I'm working on an ETL script that needs to pull data from a riak cluster. My client has sent me a backup from one of their cluster nodes. bitcask data,. rings and config. *snip* At this point I believe I should be able to

Riak 1.3.1 - The Story So Far

2013-06-17 Thread Shane McEwan
G'day! We've been running with 1.3.1 for most of this week. Generally it's been going well. We especially feel happier knowing that Active Anti-Entropy is keeping an eye on things. As we mostly use map reduce queries we rarely triggered any read repairs so it's good that we'll be getting

Riak 1.2.1 Crash During Rolling Upgrade to 1.3.1

2013-06-11 Thread Shane McEwan
G'day! I upgraded our production 4-node Riak cluster from 1.2.1 to 1.3.1 on the weekend. It didn't go as smoothly as expected. After starting Riak on the first upgraded node, node01, I started getting error messages on two as yet unupgraded nodes, node02 and node03: 2013-06-08 21:22:50.596

New Riak 1.3 Status Entries

2013-06-07 Thread Shane McEwan
G'day! I've noticed a few undocumented 'riak-admin status' entries since upgrading to 1.3.1: read_repairs_primary_outofdate_one read_repairs_primary_outofdate_count read_repairs_primary_notfound_one read_repairs_primary_notfound_count leveldb_read_block_error pipeline_active

Re: New Riak 1.3 Status Entries

2013-06-07 Thread Shane McEwan
On 07/06/13 14:22, Brian Shumate wrote: Thanks for the feedback! I've added an issue[0] to our basho_docs repository[1] to get this information into the documentation. Thanks Brian. I keep forgetting about the docs repository. I'll use that in future. I think the `riak-admin aae-status`

Re: Riak 1.3.1 crash when directory used by AAE is full

2013-05-29 Thread Shane McEwan
On 29/05/13 06:50, Mark Phillips wrote: At the very least we should have the expected systems needs for AAE storage documented. There's probably a middle-ground mitigated with documentation in the short term. We're trying to freeze for 1.4 at the moment, but I'll make sure this gets some

Riak 1.3.1 Upgrade Questions

2013-05-24 Thread Shane McEwan
G'day! We're about to start upgrading our Riak clusters from 1.2.1 to 1.3.1 and I have a couple of questions about the upgrade process. Firstly, the new enable_health_checks config option. It's not clear what impact this option will have on a busy cluster. The way I read it, if a

Re: cloned machines issues

2013-04-29 Thread Shane McEwan
On 27/04/13 10:56, Pieter Breed wrote: I'm trying to get two cloned machines to form a new cluster but I'm struggling to get rid of the old cluster config. I went through the same issue when trying to clone our production cluster to use as a staging cluster. The trick is to configure the

Re: Riak vnodes not available

2013-03-05 Thread Shane McEwan
On 05/03/13 11:36, Daniel Iwan wrote: What is the correct approach to make sure nodes are ready and Riak service is fully up and running and can take request? SInce our app depends on Riak we need to somehow make sure Riak is ready before our services start. G'day! We run 256 vnodes on 4

Re: Parameter Planning (eleveldb)

2013-02-05 Thread Shane McEwan
Simon, remember that it's just a maximum limit we're talking about here so there's no harm in setting it to a large number and Riak will just use whatever it needs (probably much less than the limit). Setting low limits on a multi-user machine might make sense so that one user can't kill the

Re: Missing SST File

2013-01-14 Thread Shane McEwan
] http://docs.basho.com/riak/1.2.1/cookbooks/Repairing-KV-Indexes/ [2] https://gist.github.com/2834473 Hope this helps! Shane. On 11/01/13 13:47, Shane McEwan wrote: Thanks Matthew. We're running version 1.2.1. I was actually following the Repair KV Indexes[1] instructions which triggered

Re: Missing SST File

2013-01-11 Thread Shane McEwan
a different sort of repair I can run? [1] http://docs.basho.com/riak/1.2.1/cookbooks/Repairing-KV-Indexes/ On 11/01/13 12:48, Matthew Von-Maszewski wrote: What version of Riak? Likely you need to take the node offline and run repair. Matthew On Jan 11, 2013, at 4:50 AM, Shane McEwan sh

Test Environment

2012-12-20 Thread Shane McEwan
G'day! I'm currently trying to set up a test environment mimicking our production environment by copying the leveldb directory from each production node onto the corresponding test node. By carefully matching the node's partition ownerships to determine where I copy the data I can start up

Re: Failed to create ~p for mapred_queue_dir defaulting to %s: ~p [/data/riak/mr_queue, /tmp/mr_queue, eacces]

2012-12-18 Thread Shane McEwan
On 18/12/12 04:52, David Montgomery wrote: Below is the head of the console.log file. First error is Failed to create ~p for mapred_queue_dir defaulting to %s: ~p [/data/riak/mr_queue,/tmp/mr_queue,eacces] G'day! eacces means there's a permission problem. After mounting the filesystem you

Re: Cluster setup

2012-12-12 Thread Shane McEwan
On 11/12/12 19:11, Kevin Burton wrote: Thank you. I have restarted the server several times and I don't see riak running with 'ps ax'. The configuration is set to come up on reboot so there shouldn't be any other instance of riak around to run. It's not necessarily another Riak instance but

Re: Cluster setup

2012-12-12 Thread Shane McEwan
On 12/12/12 14:15, Kevin Burton wrote: Thank you but this command didn't return anything: OK, perhaps you're using the same port for different functions in Riak? In particular, I see you're using 8099 for your http port but in my setup 8099 is used for handoff_port. Here's the ports I'm

Re: Cluster setup

2012-12-12 Thread Shane McEwan
On 12/12/12 14:32, Kevin Burton wrote: Good catch! Thank you. If I have three machines that take up ports 8098,8099, and 8100 what do you suggest the handoff port should be? When is it used? No problem! Each machine can use the same ports as the other machines. You just can't use the same

Re: Cluster setup

2012-12-12 Thread Shane McEwan
@10.79.110.52' to 'riak@10.79.108.25' -Original Message- From: Shane McEwan [mailto:sh...@mcewan.id.au] Sent: Wednesday, December 12, 2012 8:58 AM To: Kevin Burton Cc: riak-users@lists.basho.com Subject: Re: Cluster setup On 12/12/12 14:32, Kevin Burton wrote: Good catch! Thank you. If I have three

Re: Cluster setup

2012-12-12 Thread Shane McEwan
thought [azureuser@bsicentos1 ~]$ head /etc/riak/vm.args ## Name of the riak node -name riak@10.79.108.25 It doesn't seem that you can join a node to itself. -Original Message- From: Shane McEwan [mailto:sh...@mcewan.id.au] Sent: Wednesday, December 12, 2012 9:26 AM To: Kevin Burton Cc

Re: Cluster setup

2012-12-12 Thread Shane McEwan
will be on distinct nodes Transfers resulting from cluster changes: 32 32 transfers from 'riak@10.79.110.52' to 'riak@10.79.108.25' *I don't see riak@10.79.90.11 mailto:riak@10.79.90.11 in the cluster.* -Original Message- From: Shane McEwan [mailto:sh...@mcewan.id.au] Sent: Wednesday

Re: Cluster setup

2012-12-12 Thread Shane McEwan
' --- Valid:3 / Leaving:0 / Exiting:0 / Joining:0 / Down:0 Does this look right? -Original Message- From: Shane McEwan [mailto:sh...@mcewan.id.au] Sent: Wednesday, December 12, 2012 9:58 AM To: Kevin Burton Cc: riak-users@lists.basho.com Subject: Re: Cluster setup Run the join

Re: Mal Reduce Exception: {phase:0,error:[timeout],input:

2012-12-10 Thread Shane McEwan
G'day David, It looks like you might be hitting this bug: https://github.com/basho/riak_kv/issues/290 It doesn't look like there's an easy fix unless you want to backport the patch into your current system. :-( Shane. On 10/12/12 04:03, David Montgomery wrote: Hi, I am having an

Re: Transfer database to new machines fail

2012-12-07 Thread Shane McEwan
Or this: grep files /proc/`pgrep -u riak beam`/limits Tricky if you can't get Riak to stay up long enough, though. On 07/12/12 09:59, Rapsey wrote: Attach to riak: riak attach run this: os:cmd(ulimit -n). Sergej On Fri, Dec 7, 2012 at 7:45 AM, kser kong_...@hotmail.com

Re: Transfer database to new machines fail

2012-12-07 Thread Shane McEwan
For Ubuntu you need to: Add: * - nofile 1048576 to /etc/security/limits.d/max_open_files.conf (You may need to create this file.) Uncomment or add: sessionrequired pam_limits.so in /etc/pam.d/su On 07/12/12 10:52, kser wrote: When i run this: riak attach return: Node is not

Re: More Migration Questions

2012-11-13 Thread Shane McEwan
be an easier way to spin up a new cluster with new names and IPs but with old data? Shane. On 08/11/12 21:10, Shane McEwan wrote: G'day! Just to add to the list of people asking questions about migrating to 1.2.1 . . . We're about to migrate our 4 node production Riak database from 1.1.1

Re: More Migration Questions

2012-11-13 Thread Shane McEwan
G'day Tom and Matt. Thanks for your suggestions. Even though our two cluster networks are separate it would theoretically be possible to have nodes from the new cluster join the old cluster and migrate data that way. However, we would prefer to leave the old cluster untouched as much as

Re: More Migration Questions

2012-11-09 Thread Shane McEwan
it up and hope for the best. In theory this rolling upgrade should work . . . assuming 1.1.1 reip actually works. I'm really keen to hear from anyone with any suggestions. On 08/11/12 21:10, Shane McEwan wrote: G'day! Just to add to the list of people asking questions about migrating to 1.2.1

Re: Cluster on startup.

2012-11-07 Thread Shane McEwan
If you edit /etc/init.d/riak you should see a line like: su - riak -c $DAEMON $DAEMON_ARGS || return 2 in the do_start function. If you change that line into the following 4 lines: su - riak -c /path/to/your/riak/dev1/bin/riak $DAEMON_ARGS || return 2 su - riak -c

Re: riak memstore clarification on enomem error

2012-10-09 Thread Shane McEwan
G'day Sangeetha. On 09/10/12 07:40, sangeetha.pattabiram...@cognizant.com wrote: Dear Team , I have a 64 GB RAM ,during the Load of 35 million dataset (1.8 GB) it consumes nearly 40-45 GB of RAM durial the startup of the erlang script ,but While trying to load 40 million

Riak Memory Usage Constantly Growing

2012-10-02 Thread Shane McEwan
G'day! Just recently we've noticed memory usage in our Riak cluster constantly increasing. The memory usage reported by the Riak stats memory_total parameter has been less than 100MB for nearly a year but has recently increased to over 1GB. If we restart the cluster memory usage usually

Re: Riak backups

2012-06-28 Thread Shane McEwan
manually into tarball? Thanks in advance, nv. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com -- Shane McEwan, Senior Systems Administrator, Actual Experience Optimising the human