Re: Opscenter Metrics

2014-01-03 Thread Arun Kumar K
Thanks Nick for the pointer !


On Thu, Jan 2, 2014 at 10:31 PM, Nick Bailey n...@datastax.com wrote:

 I believe the answer to your similar question on server fault should
 answer this:


 http://serverfault.com/questions/564107/what-does-opscenters-write-requests-count-shown-as-ops-sec-exactly-mean

 On Tue, Dec 31, 2013 at 12:55 AM, Arun Kumar K arunk...@gmail.com wrote:

 Hi guys,

 I am using YCSB and using thrift based *client.batch_mutate()* call.

 Now say opscenter reports the write requests as say 1000 *operations*/sec
 when a record count is say 1 records.

 OpsCenter API docs say 'Write Requests as requests per second

 1 What does an 'operation or request' mean from opscenter perspective?

 2 Does it mean unique row inserts across all column families ? or  Does
 itmean count of each mutations for a row ?

  Can some one guide me ?


 Thanks,

  Arun





Re: Cassandra 1.1.6 crash without any exception or error in log

2014-01-03 Thread Robert Coli
On Thu, Jan 2, 2014 at 10:03 PM, Nitin Sharma
nitin.sha...@bloomreach.comwrote:

 I would recommend always running cassandra with
  -XX:+HeapDumpOnOutofMemoryError. This dumps out  a *.hprof file if the
 process dies due to OOM


If you do this, be sure to configure the heap dump directory, or ensure
that the running directory of cassandra (often /home/cassandra) is on a
partition with enough free space to hold a dump the size of your heap.

=Rob


nodetool status owns % calculation after upgrade to 2.0.2

2014-01-03 Thread Mullen, Robert
Hello,
I have a multi region cluster with 3 nodes in each data center, ec2 us-east
and and west.  Prior to upgrading to 2.0.2 from 1.2.6, the owns % of each
node was 100%, which made sense because I had a replication factor of 3 for
each data center.  After upgrading to 2.0.2 each node claims to own about
17% of the data now.


:~$ nodetool status
Datacenter: us-west-2
=
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  AddressLoad   Tokens  Owns   Host ID
Rack
UN  10.198.20.51   958.16 KB  256 16.9%
 6a40b500-cff4-4513-b26b-ea33048c1590  usw2c
UN  10.198.18.125  776 KB 256 17.0%
 aa746ed1-288c-414f-8d97-65fc867a5bdd  usw2b
UN  10.198.16.92   1.39 MB256 16.4%
 01989d0b-0f81-411b-a70e-f22f01189542  usw2a
Datacenter: us-east-1
=
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  AddressLoad   Tokens  Owns   Host ID
Rack
UN  10.198.0.249   1.11 MB256 16.3%
 22b30bea-5643-43b5-8d98-6e0eafe4af75  use1b
UN  10.198.4.801.22 MB256 16.4%
 e31aecd5-1eb1-4ddb-85ac-7a4135618b66  use1d
UN  10.198.2.20137.27 MB  256 17.0%
 3253080f-09b6-47a6-9b66-da3d174d1101  use1c

I checked some of the data on each of the nodes in one column family and
the counts are different across the nodes now.   I'm trying to run a
nodetool repair but it's been running for about 6 hours now.

So a couple of questions:
1.  Any idea why the owns % would have changed from 100% to 17% per node
after upgrade?
2. Is there anything else I can do to get the data back in sync between the
nodes other than nodetool repair?

thanks,
Rob


Re: nodetool status owns % calculation after upgrade to 2.0.2

2014-01-03 Thread Robert Coli
On Fri, Jan 3, 2014 at 3:33 PM, Mullen, Robert robert.mul...@pearson.comwrote:

 I have a multi region cluster with 3 nodes in each data center, ec2
 us-east and and west.  Prior to upgrading to 2.0.2 from 1.2.6, the owns %
 of each node was 100%, which made sense because I had a replication factor
 of 3 for each data center.  After upgrading to 2.0.2 each node claims to
 own about 17% of the data now.


First, why would you upgrade to 2.0.2 when higher versions exist?

Second, are you running in production? If so, read this :
https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/


 So a couple of questions:
 1.  Any idea why the owns % would have changed from 100% to 17% per node
 after upgrade?


Because the display of this information has changed repeatedly over the
years, including for bugfixes.

https://issues.apache.org/jira/browse/CASSANDRA-3412
https://issues.apache.org/jira/browse/CASSANDRA-5076
https://issues.apache.org/jira/browse/CASSANDRA-4598
https://issues.apache.org/jira/browse/CASSANDRA-6168
https://issues.apache.org/jira/browse/CASSANDRA-5954

etc.

2. Is there anything else I can do to get the data back in sync between the
 nodes other than nodetool repair?


What do you mean by the counts are different across the nodes now?

It is pretty unlikely that you have lost any data, from what you have
described.

=Rob