Re: Running Cassandra in Integration Tests

[Edward Capriolo]

Checkout https://github.com/edwardcapriolo/farsandra. It falls under the
realm of almost 100% pure java (besides the fact it uses some shell to
launch Cassandra).

On Thu, Oct 6, 2016 at 7:08 PM, Ali Akhtar  wrote:

> Is it possible to create an isolated cassandra instance which is run
> during integration tests and it disappears after tests have finished
> running? Then its recreated the next time tests run (perhaps being
> populated with test data).
>
>  I'm using Java.
>
>
>

Fwd: Counter global logical clock

[Varun Barala]

Hi all Users,

Recently I was restoring some sstables by changing their timestamp.
But in case of counter It's different there are few things I'd like to ask:-

* What is the maximum value of global logical clock? *Long.MAX_VALUE* ?
** what will happen If updates hit this MAX_VALUE value ?*

I read rule of merging of counter columns:-
- global + global = keep the shard with the highest logical clock
- global + local  = keep the global one
- global + remote = keep the global one
- local  + local  = sum counts (and logical clocks)
- local  + remote = keep the local one
- remote + remote = keep the shard with the highest logical clock

So in my case ::



** Can I change to 'set as global and set logical clock to maximum to make
sure it always win when merge' ? * How can we modify a counter column so it
will always win when merge?*

I'm using *ka* sstable version.

Thanking You!!

Regards,
Varun Barala

Re: Running Cassandra in Integration Tests

[Jonathan Haddad]

3.9 falls under the Tick Tock release cycle, which is almost completely
untested in production by experienced operators.  In the cases where it has
been tested, there have been numerous bugs found which I (and I think most
people on this list) consider to be show stoppers.  Additionally, the Tick
Tock release cycle puts the operator in the uncomfortable position of
having to decide between upgrading to a new version with new features
(probably new bugs) or back porting bug fixes from future versions
themselves.There will never be a 3.9.1 release which fixes bugs in 3.9
without adding new features.

https://github.com/apache/cassandra/blob/trunk/NEWS.txt

For new projects I recommend starting with the recently released 3.0.9.

Assuming the project changes it's policy on releases (all signs point to
yes), then by the time 4.0 rolls out a lot of the features which have been
released in the 3.x series will have matured a bit, so it's very possible
4.0 will stabilize faster than the usual 6 months it takes for a major
release.

All that said, there's nothing wrong with doing compatibility & smoke tests
against the latest 3.x release as well as 3.0 and reporting bugs back to
the Apache Cassandra JIRA, I'm sure it would be greatly appreciated.

https://issues.apache.org/jira/secure/Dashboard.jspa

Jon

On Thu, Oct 6, 2016 at 5:35 PM Ali Akhtar  wrote:

> That looks great Andrew, but it doesn't seem to have been committed to for
> 5 months. Is anyone still using cassandra-unit , does it work with latest
> cassandra versions (e.g 3.9)?
>
> On Fri, Oct 7, 2016 at 5:33 AM, Andrew Tolbert <
> andrew.tolb...@datastax.com> wrote:
>
> Hi Ali,
>
> cassandra-unit  might be
> what you are looking for.  It allows you to run an embedded cassandra
> instance along side your tests and has some nice integration with JUnit.
>
> Thanks,
> Andy
>
> On Thu, Oct 6, 2016 at 7:13 PM Ali Akhtar  wrote:
>
> Ok, but that's not what this question is about.
>
> I'm looking for a way to run an embedded cassandra instance which is
> created & destroyed during tests and which doesn't persist any state
> outside the tests.
>
>
> On Fri, Oct 7, 2016 at 5:10 AM, Kant Kodali  wrote:
>
> you dont need to look for cassandra java api to start/stop instance. you
> just need to write a shell script or python or java or any language to
> execute shell commands!
>
>
>
> On Thu, Oct 6, 2016 4:57 PM, Ali Akhtar ali.rac...@gmail.com wrote:
>
> Okay.. but how would I start this instance? Is there a java api to
> programmatically start / destroy an instance during tests?
>
> On Fri, Oct 7, 2016 at 4:56 AM, Kant Kodali  wrote:
>
> sure as long as that isolated instance is treated as separate cluster you
> shouldn't run into any problems.
>
>
>
> On Thu, Oct 6, 2016 4:08 PM, Ali Akhtar ali.rac...@gmail.com wrote:
>
> Is it possible to create an isolated cassandra instance which is run
> during integration tests and it disappears after tests have finished
> running? Then its recreated the next time tests run (perhaps being
> populated with test data).
>
>  I'm using Java.
>
>
>
>
>
>

Re: Running Cassandra in Integration Tests

[Peddi, Praveen]

Our application takes care of creating keyspace and tables as necessary. Our 
tests didn't need to worry about that.

On Oct 6, 2016, at 9:22 PM, Ali Akhtar 
> wrote:

Oh, and how do you generate your tables / keyspaces for the tests (if you do)?

On Fri, Oct 7, 2016 at 6:21 AM, Ali Akhtar 
> wrote:
Peddi,

Thanks, does this start @ localhost, default port? And, mind sharing which 
version of cassandra you use this with? will this work on 3.9, or on 3.7?

Also, why is the config in a temp location - wouldn't it be better to put it in 
src/test/resources and try to specify that?

Thanks.

On Fri, Oct 7, 2016 at 6:17 AM, Peddi, Praveen 
> wrote:
Honestly there isn't much code to share. Here is the code (excluded exception 
handling). For yaml location, we have the classpathpath and write to temp 
location everytime.

System.setProperty("cassandra.config", "file://" + YAML_LOCATION);
cassandraDaemon = new CassandraDaemon();
 cassandraDaemon.init(null);
cassandraDaemon.start();

//stop cassandra after tests are done
cassandraDaemon.stop();

From: Ali Akhtar >
Reply-To: "user@cassandra.apache.org" 
>
Date: Thursday, October 6, 2016 at 8:44 PM

To: "user@cassandra.apache.org" 
>
Subject: Re: Running Cassandra in Integration Tests

Peddi @ Amazon, would you be able to share your code such as in a gist, if 
possible, please?

Also, which version of cassandra are you doing that with?



On Fri, Oct 7, 2016 at 5:41 AM, Peddi, Praveen 
> wrote:
We could not use cassandra unit for licensing reasons so we ended up 
instantiate CassandraDeamon after setting system property of 
cassandra.config={yaml location}. It works fine for our needs.

Praveen

From: Ali Akhtar >
Reply-To: "user@cassandra.apache.org" 
>
Date: Thursday, October 6, 2016 at 8:35 PM
To: "user@cassandra.apache.org" 
>
Subject: Re: Running Cassandra in Integration Tests

That looks great Andrew, but it doesn't seem to have been committed to for 5 
months. Is anyone still using cassandra-unit , does it work with latest 
cassandra versions (e.g 3.9)?

On Fri, Oct 7, 2016 at 5:33 AM, Andrew Tolbert 
> wrote:
Hi Ali,

cassandra-unit might be what you 
are looking for.  It allows you to run an embedded cassandra instance along 
side your tests and has some nice integration with JUnit.

Thanks,
Andy

On Thu, Oct 6, 2016 at 7:13 PM Ali Akhtar 
> wrote:
Ok, but that's not what this question is about.

I'm looking for a way to run an embedded cassandra instance which is created & 
destroyed during tests and which doesn't persist any state outside the tests.


On Fri, Oct 7, 2016 at 5:10 AM, Kant Kodali 
> wrote:
you dont need to look for cassandra java api to start/stop instance. you just 
need to write a shell script or python or java or any language to execute shell 
commands!




On Thu, Oct 6, 2016 4:57 PM, Ali Akhtar 
ali.rac...@gmail.com wrote:
Okay.. but how would I start this instance? Is there a java api to 
programmatically start / destroy an instance during tests?

On Fri, Oct 7, 2016 at 4:56 AM, Kant Kodali 
> wrote:
sure as long as that isolated instance is treated as separate cluster you 
shouldn't run into any problems.




On Thu, Oct 6, 2016 4:08 PM, Ali Akhtar 
ali.rac...@gmail.com wrote:
Is it possible to create an isolated cassandra instance which is run during 
integration tests and it disappears after tests have finished running? Then its 
recreated the next time tests run (perhaps being populated with test data).

 I'm using Java.

Re: Running Cassandra in Integration Tests

[Peddi, Praveen]

Akthar,
We use 2.2.4.

I am not sure if cassandra.config can take a classpath. Feel free to try. It 
was easy enough to copy the resource to temp file location for us.

Praveen

On Oct 6, 2016, at 9:21 PM, Ali Akhtar 
> wrote:

Peddi,

Thanks, does this start @ localhost, default port? And, mind sharing which 
version of cassandra you use this with? will this work on 3.9, or on 3.7?

Also, why is the config in a temp location - wouldn't it be better to put it in 
src/test/resources and try to specify that?

Thanks.

On Fri, Oct 7, 2016 at 6:17 AM, Peddi, Praveen 
> wrote:
Honestly there isn't much code to share. Here is the code (excluded exception 
handling). For yaml location, we have the classpathpath and write to temp 
location everytime.

System.setProperty("cassandra.config", "file://" + YAML_LOCATION);
cassandraDaemon = new CassandraDaemon();
 cassandraDaemon.init(null);
cassandraDaemon.start();

//stop cassandra after tests are done
cassandraDaemon.stop();

From: Ali Akhtar >
Reply-To: "user@cassandra.apache.org" 
>
Date: Thursday, October 6, 2016 at 8:44 PM

To: "user@cassandra.apache.org" 
>
Subject: Re: Running Cassandra in Integration Tests

Peddi @ Amazon, would you be able to share your code such as in a gist, if 
possible, please?

Also, which version of cassandra are you doing that with?



On Fri, Oct 7, 2016 at 5:41 AM, Peddi, Praveen 
> wrote:
We could not use cassandra unit for licensing reasons so we ended up 
instantiate CassandraDeamon after setting system property of 
cassandra.config={yaml location}. It works fine for our needs.

Praveen

From: Ali Akhtar >
Reply-To: "user@cassandra.apache.org" 
>
Date: Thursday, October 6, 2016 at 8:35 PM
To: "user@cassandra.apache.org" 
>
Subject: Re: Running Cassandra in Integration Tests

That looks great Andrew, but it doesn't seem to have been committed to for 5 
months. Is anyone still using cassandra-unit , does it work with latest 
cassandra versions (e.g 3.9)?

On Fri, Oct 7, 2016 at 5:33 AM, Andrew Tolbert 
> wrote:
Hi Ali,

cassandra-unit might be what you 
are looking for.  It allows you to run an embedded cassandra instance along 
side your tests and has some nice integration with JUnit.

Thanks,
Andy

On Thu, Oct 6, 2016 at 7:13 PM Ali Akhtar 
> wrote:
Ok, but that's not what this question is about.

I'm looking for a way to run an embedded cassandra instance which is created & 
destroyed during tests and which doesn't persist any state outside the tests.


On Fri, Oct 7, 2016 at 5:10 AM, Kant Kodali 
> wrote:
you dont need to look for cassandra java api to start/stop instance. you just 
need to write a shell script or python or java or any language to execute shell 
commands!




On Thu, Oct 6, 2016 4:57 PM, Ali Akhtar 
ali.rac...@gmail.com wrote:
Okay.. but how would I start this instance? Is there a java api to 
programmatically start / destroy an instance during tests?

On Fri, Oct 7, 2016 at 4:56 AM, Kant Kodali 
> wrote:
sure as long as that isolated instance is treated as separate cluster you 
shouldn't run into any problems.




On Thu, Oct 6, 2016 4:08 PM, Ali Akhtar 
ali.rac...@gmail.com wrote:
Is it possible to create an isolated cassandra instance which is run during 
integration tests and it disappears after tests have finished running? Then its 
recreated the next time tests run (perhaps being populated with test data).

 I'm using Java.

Re: Running Cassandra in Integration Tests

[Ali Akhtar]

Oh, and how do you generate your tables / keyspaces for the tests (if you
do)?

On Fri, Oct 7, 2016 at 6:21 AM, Ali Akhtar  wrote:

> Peddi,
>
> Thanks, does this start @ localhost, default port? And, mind sharing which
> version of cassandra you use this with? will this work on 3.9, or on 3.7?
>
> Also, why is the config in a temp location - wouldn't it be better to put
> it in src/test/resources and try to specify that?
>
> Thanks.
>
> On Fri, Oct 7, 2016 at 6:17 AM, Peddi, Praveen  wrote:
>
>> Honestly there isn’t much code to share. Here is the code (excluded
>> exception handling). For yaml location, we have the classpathpath and write
>> to temp location everytime.
>>
>> System.setProperty("cassandra.config", "file://" + YAML_LOCATION);
>> cassandraDaemon = new CassandraDaemon();
>>  cassandraDaemon.init(null);
>> cassandraDaemon.start();
>>
>> //stop cassandra after tests are done
>> cassandraDaemon.stop();
>>
>> From: Ali Akhtar 
>> Reply-To: "user@cassandra.apache.org" 
>> Date: Thursday, October 6, 2016 at 8:44 PM
>>
>> To: "user@cassandra.apache.org" 
>> Subject: Re: Running Cassandra in Integration Tests
>>
>> Peddi @ Amazon, would you be able to share your code such as in a gist,
>> if possible, please?
>>
>> Also, which version of cassandra are you doing that with?
>>
>>
>>
>> On Fri, Oct 7, 2016 at 5:41 AM, Peddi, Praveen  wrote:
>>
>>> We could not use cassandra unit for licensing reasons so we ended up
>>> instantiate CassandraDeamon after setting system property of
>>> cassandra.config={yaml location}. It works fine for our needs.
>>>
>>> Praveen
>>>
>>> From: Ali Akhtar 
>>> Reply-To: "user@cassandra.apache.org" 
>>> Date: Thursday, October 6, 2016 at 8:35 PM
>>> To: "user@cassandra.apache.org" 
>>> Subject: Re: Running Cassandra in Integration Tests
>>>
>>> That looks great Andrew, but it doesn't seem to have been committed to
>>> for 5 months. Is anyone still using cassandra-unit , does it work with
>>> latest cassandra versions (e.g 3.9)?
>>>
>>> On Fri, Oct 7, 2016 at 5:33 AM, Andrew Tolbert <
>>> andrew.tolb...@datastax.com> wrote:
>>>
 Hi Ali,

 cassandra-unit  might be
 what you are looking for.  It allows you to run an embedded cassandra
 instance along side your tests and has some nice integration with JUnit.

 Thanks,
 Andy

 On Thu, Oct 6, 2016 at 7:13 PM Ali Akhtar  wrote:

> Ok, but that's not what this question is about.
>
> I'm looking for a way to run an embedded cassandra instance which is
> created & destroyed during tests and which doesn't persist any state
> outside the tests.
>
>
> On Fri, Oct 7, 2016 at 5:10 AM, Kant Kodali  wrote:
>
> you dont need to look for cassandra java api to start/stop instance.
> you just need to write a shell script or python or java or any language to
> execute shell commands!
>
>
>
> On Thu, Oct 6, 2016 4:57 PM, Ali Akhtar ali.rac...@gmail.com wrote:
>
> Okay.. but how would I start this instance? Is there a java api to
> programmatically start / destroy an instance during tests?
>
> On Fri, Oct 7, 2016 at 4:56 AM, Kant Kodali  wrote:
>
> sure as long as that isolated instance is treated as separate cluster
> you shouldn't run into any problems.
>
>
>
> On Thu, Oct 6, 2016 4:08 PM, Ali Akhtar ali.rac...@gmail.com wrote:
>
> Is it possible to create an isolated cassandra instance which is run
> during integration tests and it disappears after tests have finished
> running? Then its recreated the next time tests run (perhaps being
> populated with test data).
>
>  I'm using Java.
>
>
>
>
>
>>>
>>
>

Re: Running Cassandra in Integration Tests

[Ali Akhtar]

Peddi,

Thanks, does this start @ localhost, default port? And, mind sharing which
version of cassandra you use this with? will this work on 3.9, or on 3.7?

Also, why is the config in a temp location - wouldn't it be better to put
it in src/test/resources and try to specify that?

Thanks.

On Fri, Oct 7, 2016 at 6:17 AM, Peddi, Praveen  wrote:

> Honestly there isn’t much code to share. Here is the code (excluded
> exception handling). For yaml location, we have the classpathpath and write
> to temp location everytime.
>
> System.setProperty("cassandra.config", "file://" + YAML_LOCATION);
> cassandraDaemon = new CassandraDaemon();
>  cassandraDaemon.init(null);
> cassandraDaemon.start();
>
> //stop cassandra after tests are done
> cassandraDaemon.stop();
>
> From: Ali Akhtar 
> Reply-To: "user@cassandra.apache.org" 
> Date: Thursday, October 6, 2016 at 8:44 PM
>
> To: "user@cassandra.apache.org" 
> Subject: Re: Running Cassandra in Integration Tests
>
> Peddi @ Amazon, would you be able to share your code such as in a gist, if
> possible, please?
>
> Also, which version of cassandra are you doing that with?
>
>
>
> On Fri, Oct 7, 2016 at 5:41 AM, Peddi, Praveen  wrote:
>
>> We could not use cassandra unit for licensing reasons so we ended up
>> instantiate CassandraDeamon after setting system property of
>> cassandra.config={yaml location}. It works fine for our needs.
>>
>> Praveen
>>
>> From: Ali Akhtar 
>> Reply-To: "user@cassandra.apache.org" 
>> Date: Thursday, October 6, 2016 at 8:35 PM
>> To: "user@cassandra.apache.org" 
>> Subject: Re: Running Cassandra in Integration Tests
>>
>> That looks great Andrew, but it doesn't seem to have been committed to
>> for 5 months. Is anyone still using cassandra-unit , does it work with
>> latest cassandra versions (e.g 3.9)?
>>
>> On Fri, Oct 7, 2016 at 5:33 AM, Andrew Tolbert <
>> andrew.tolb...@datastax.com> wrote:
>>
>>> Hi Ali,
>>>
>>> cassandra-unit  might be
>>> what you are looking for.  It allows you to run an embedded cassandra
>>> instance along side your tests and has some nice integration with JUnit.
>>>
>>> Thanks,
>>> Andy
>>>
>>> On Thu, Oct 6, 2016 at 7:13 PM Ali Akhtar  wrote:
>>>
 Ok, but that's not what this question is about.

 I'm looking for a way to run an embedded cassandra instance which is
 created & destroyed during tests and which doesn't persist any state
 outside the tests.


 On Fri, Oct 7, 2016 at 5:10 AM, Kant Kodali  wrote:

 you dont need to look for cassandra java api to start/stop instance.
 you just need to write a shell script or python or java or any language to
 execute shell commands!



 On Thu, Oct 6, 2016 4:57 PM, Ali Akhtar ali.rac...@gmail.com wrote:

 Okay.. but how would I start this instance? Is there a java api to
 programmatically start / destroy an instance during tests?

 On Fri, Oct 7, 2016 at 4:56 AM, Kant Kodali  wrote:

 sure as long as that isolated instance is treated as separate cluster
 you shouldn't run into any problems.



 On Thu, Oct 6, 2016 4:08 PM, Ali Akhtar ali.rac...@gmail.com wrote:

 Is it possible to create an isolated cassandra instance which is run
 during integration tests and it disappears after tests have finished
 running? Then its recreated the next time tests run (perhaps being
 populated with test data).

  I'm using Java.





>>
>

Re: Running Cassandra in Integration Tests

[Peddi, Praveen]

Honestly there isn't much code to share. Here is the code (excluded exception 
handling). For yaml location, we have the classpathpath and write to temp 
location everytime.

System.setProperty("cassandra.config", "file://" + YAML_LOCATION);
cassandraDaemon = new CassandraDaemon();
 cassandraDaemon.init(null);
cassandraDaemon.start();

//stop cassandra after tests are done
cassandraDaemon.stop();

From: Ali Akhtar >
Reply-To: "user@cassandra.apache.org" 
>
Date: Thursday, October 6, 2016 at 8:44 PM
To: "user@cassandra.apache.org" 
>
Subject: Re: Running Cassandra in Integration Tests

Peddi @ Amazon, would you be able to share your code such as in a gist, if 
possible, please?

Also, which version of cassandra are you doing that with?



On Fri, Oct 7, 2016 at 5:41 AM, Peddi, Praveen 
> wrote:
We could not use cassandra unit for licensing reasons so we ended up 
instantiate CassandraDeamon after setting system property of 
cassandra.config={yaml location}. It works fine for our needs.

Praveen

From: Ali Akhtar >
Reply-To: "user@cassandra.apache.org" 
>
Date: Thursday, October 6, 2016 at 8:35 PM
To: "user@cassandra.apache.org" 
>
Subject: Re: Running Cassandra in Integration Tests

That looks great Andrew, but it doesn't seem to have been committed to for 5 
months. Is anyone still using cassandra-unit , does it work with latest 
cassandra versions (e.g 3.9)?

On Fri, Oct 7, 2016 at 5:33 AM, Andrew Tolbert 
> wrote:
Hi Ali,

cassandra-unit might be what you 
are looking for.  It allows you to run an embedded cassandra instance along 
side your tests and has some nice integration with JUnit.

Thanks,
Andy

On Thu, Oct 6, 2016 at 7:13 PM Ali Akhtar 
> wrote:
Ok, but that's not what this question is about.

I'm looking for a way to run an embedded cassandra instance which is created & 
destroyed during tests and which doesn't persist any state outside the tests.


On Fri, Oct 7, 2016 at 5:10 AM, Kant Kodali 
> wrote:
you dont need to look for cassandra java api to start/stop instance. you just 
need to write a shell script or python or java or any language to execute shell 
commands!




On Thu, Oct 6, 2016 4:57 PM, Ali Akhtar 
ali.rac...@gmail.com wrote:
Okay.. but how would I start this instance? Is there a java api to 
programmatically start / destroy an instance during tests?

On Fri, Oct 7, 2016 at 4:56 AM, Kant Kodali 
> wrote:
sure as long as that isolated instance is treated as separate cluster you 
shouldn't run into any problems.




On Thu, Oct 6, 2016 4:08 PM, Ali Akhtar 
ali.rac...@gmail.com wrote:
Is it possible to create an isolated cassandra instance which is run during 
integration tests and it disappears after tests have finished running? Then its 
recreated the next time tests run (perhaps being populated with test data).

 I'm using Java.

Re: Running Cassandra in Integration Tests

[Ali Akhtar]

Peddi @ Amazon, would you be able to share your code such as in a gist, if
possible, please?

Also, which version of cassandra are you doing that with?



On Fri, Oct 7, 2016 at 5:41 AM, Peddi, Praveen  wrote:

> We could not use cassandra unit for licensing reasons so we ended up
> instantiate CassandraDeamon after setting system property of
> cassandra.config={yaml location}. It works fine for our needs.
>
> Praveen
>
> From: Ali Akhtar 
> Reply-To: "user@cassandra.apache.org" 
> Date: Thursday, October 6, 2016 at 8:35 PM
> To: "user@cassandra.apache.org" 
> Subject: Re: Running Cassandra in Integration Tests
>
> That looks great Andrew, but it doesn't seem to have been committed to for
> 5 months. Is anyone still using cassandra-unit , does it work with latest
> cassandra versions (e.g 3.9)?
>
> On Fri, Oct 7, 2016 at 5:33 AM, Andrew Tolbert <
> andrew.tolb...@datastax.com> wrote:
>
>> Hi Ali,
>>
>> cassandra-unit  might be
>> what you are looking for.  It allows you to run an embedded cassandra
>> instance along side your tests and has some nice integration with JUnit.
>>
>> Thanks,
>> Andy
>>
>> On Thu, Oct 6, 2016 at 7:13 PM Ali Akhtar  wrote:
>>
>>> Ok, but that's not what this question is about.
>>>
>>> I'm looking for a way to run an embedded cassandra instance which is
>>> created & destroyed during tests and which doesn't persist any state
>>> outside the tests.
>>>
>>>
>>> On Fri, Oct 7, 2016 at 5:10 AM, Kant Kodali  wrote:
>>>
>>> you dont need to look for cassandra java api to start/stop instance. you
>>> just need to write a shell script or python or java or any language to
>>> execute shell commands!
>>>
>>>
>>>
>>> On Thu, Oct 6, 2016 4:57 PM, Ali Akhtar ali.rac...@gmail.com wrote:
>>>
>>> Okay.. but how would I start this instance? Is there a java api to
>>> programmatically start / destroy an instance during tests?
>>>
>>> On Fri, Oct 7, 2016 at 4:56 AM, Kant Kodali  wrote:
>>>
>>> sure as long as that isolated instance is treated as separate cluster
>>> you shouldn't run into any problems.
>>>
>>>
>>>
>>> On Thu, Oct 6, 2016 4:08 PM, Ali Akhtar ali.rac...@gmail.com wrote:
>>>
>>> Is it possible to create an isolated cassandra instance which is run
>>> during integration tests and it disappears after tests have finished
>>> running? Then its recreated the next time tests run (perhaps being
>>> populated with test data).
>>>
>>>  I'm using Java.
>>>
>>>
>>>
>>>
>>>
>

Re: Running Cassandra in Integration Tests

[Kant Kodali]

embedded cassandra instance? you mean dont persist any data to disk? If so I
dont see that being possible at this time. prior to running your test code you
just run shell script and after test truncate everything and bring down the
instance. one way or other this needs to happen underneath so I dont see how you
could bypass that whichever way you may choose.
 





On Thu, Oct 6, 2016 5:35 PM, Ali Akhtar ali.rac...@gmail.com
wrote:
That looks great Andrew, but it doesn't seem to have been committed to for 5
months. Is anyone still using cassandra-unit , does it work with latest
cassandra versions (e.g 3.9)?
On Fri, Oct 7, 2016 at 5:33 AM, Andrew Tolbert  
wrote:
Hi Ali,
cassandra-unit might be what you are looking for.  It allows you to run an
embedded cassandra instance along side your tests and has some nice integration
with JUnit.
Thanks,Andy
On Thu, Oct 6, 2016 at 7:13 PM Ali Akhtar  wrote:
Ok, but that's not what this question is about.
I'm looking for a way to run an embedded cassandra instance which is created &
destroyed during tests and which doesn't persist any state outside the tests.

On Fri, Oct 7, 2016 at 5:10 AM, Kant Kodali   wrote:
you dont need to look for cassandra java api to start/stop instance. you just
need to write a shell script or python or java or any language to execute shell
commands!

 





On Thu, Oct 6, 2016 4:57 PM, Ali Akhtar ali.rac...@gmail.com
wrote:
Okay.. but how would I start this instance? Is there a java api to
programmatically start / destroy an instance during tests?
On Fri, Oct 7, 2016 at 4:56 AM, Kant Kodali   wrote:
sure as long as that isolated instance is treated as separate cluster you
shouldn't run into any problems.

 





On Thu, Oct 6, 2016 4:08 PM, Ali Akhtar ali.rac...@gmail.com
wrote:
Is it possible to create an isolated cassandra instance which is run during
integration tests and it disappears after tests have finished running? Then its
recreated the next time tests run (perhaps being populated with test data).
I'm using Java.

Re: Running Cassandra in Integration Tests

[Peddi, Praveen]

We could not use cassandra unit for licensing reasons so we ended up 
instantiate CassandraDeamon after setting system property of 
cassandra.config={yaml location}. It works fine for our needs.

Praveen

From: Ali Akhtar >
Reply-To: "user@cassandra.apache.org" 
>
Date: Thursday, October 6, 2016 at 8:35 PM
To: "user@cassandra.apache.org" 
>
Subject: Re: Running Cassandra in Integration Tests

That looks great Andrew, but it doesn't seem to have been committed to for 5 
months. Is anyone still using cassandra-unit , does it work with latest 
cassandra versions (e.g 3.9)?

On Fri, Oct 7, 2016 at 5:33 AM, Andrew Tolbert 
> wrote:
Hi Ali,

cassandra-unit might be what you 
are looking for.  It allows you to run an embedded cassandra instance along 
side your tests and has some nice integration with JUnit.

Thanks,
Andy

On Thu, Oct 6, 2016 at 7:13 PM Ali Akhtar 
> wrote:
Ok, but that's not what this question is about.

I'm looking for a way to run an embedded cassandra instance which is created & 
destroyed during tests and which doesn't persist any state outside the tests.


On Fri, Oct 7, 2016 at 5:10 AM, Kant Kodali 
> wrote:
you dont need to look for cassandra java api to start/stop instance. you just 
need to write a shell script or python or java or any language to execute shell 
commands!




On Thu, Oct 6, 2016 4:57 PM, Ali Akhtar 
ali.rac...@gmail.com wrote:
Okay.. but how would I start this instance? Is there a java api to 
programmatically start / destroy an instance during tests?

On Fri, Oct 7, 2016 at 4:56 AM, Kant Kodali 
> wrote:
sure as long as that isolated instance is treated as separate cluster you 
shouldn't run into any problems.




On Thu, Oct 6, 2016 4:08 PM, Ali Akhtar 
ali.rac...@gmail.com wrote:
Is it possible to create an isolated cassandra instance which is run during 
integration tests and it disappears after tests have finished running? Then its 
recreated the next time tests run (perhaps being populated with test data).

 I'm using Java.

Re: Running Cassandra in Integration Tests

[Ali Akhtar]

That looks great Andrew, but it doesn't seem to have been committed to for
5 months. Is anyone still using cassandra-unit , does it work with latest
cassandra versions (e.g 3.9)?

On Fri, Oct 7, 2016 at 5:33 AM, Andrew Tolbert 
wrote:

> Hi Ali,
>
> cassandra-unit  might be
> what you are looking for.  It allows you to run an embedded cassandra
> instance along side your tests and has some nice integration with JUnit.
>
> Thanks,
> Andy
>
> On Thu, Oct 6, 2016 at 7:13 PM Ali Akhtar  wrote:
>
>> Ok, but that's not what this question is about.
>>
>> I'm looking for a way to run an embedded cassandra instance which is
>> created & destroyed during tests and which doesn't persist any state
>> outside the tests.
>>
>>
>> On Fri, Oct 7, 2016 at 5:10 AM, Kant Kodali  wrote:
>>
>> you dont need to look for cassandra java api to start/stop instance. you
>> just need to write a shell script or python or java or any language to
>> execute shell commands!
>>
>>
>>
>> On Thu, Oct 6, 2016 4:57 PM, Ali Akhtar ali.rac...@gmail.com wrote:
>>
>> Okay.. but how would I start this instance? Is there a java api to
>> programmatically start / destroy an instance during tests?
>>
>> On Fri, Oct 7, 2016 at 4:56 AM, Kant Kodali  wrote:
>>
>> sure as long as that isolated instance is treated as separate cluster you
>> shouldn't run into any problems.
>>
>>
>>
>> On Thu, Oct 6, 2016 4:08 PM, Ali Akhtar ali.rac...@gmail.com wrote:
>>
>> Is it possible to create an isolated cassandra instance which is run
>> during integration tests and it disappears after tests have finished
>> running? Then its recreated the next time tests run (perhaps being
>> populated with test data).
>>
>>  I'm using Java.
>>
>>
>>
>>
>>

Re: Running Cassandra in Integration Tests

[Andrew Tolbert]

Hi Ali,

cassandra-unit  might be what
you are looking for.  It allows you to run an embedded cassandra instance
along side your tests and has some nice integration with JUnit.

Thanks,
Andy

On Thu, Oct 6, 2016 at 7:13 PM Ali Akhtar  wrote:

> Ok, but that's not what this question is about.
>
> I'm looking for a way to run an embedded cassandra instance which is
> created & destroyed during tests and which doesn't persist any state
> outside the tests.
>
>
> On Fri, Oct 7, 2016 at 5:10 AM, Kant Kodali  wrote:
>
> you dont need to look for cassandra java api to start/stop instance. you
> just need to write a shell script or python or java or any language to
> execute shell commands!
>
>
>
> On Thu, Oct 6, 2016 4:57 PM, Ali Akhtar ali.rac...@gmail.com wrote:
>
> Okay.. but how would I start this instance? Is there a java api to
> programmatically start / destroy an instance during tests?
>
> On Fri, Oct 7, 2016 at 4:56 AM, Kant Kodali  wrote:
>
> sure as long as that isolated instance is treated as separate cluster you
> shouldn't run into any problems.
>
>
>
> On Thu, Oct 6, 2016 4:08 PM, Ali Akhtar ali.rac...@gmail.com wrote:
>
> Is it possible to create an isolated cassandra instance which is run
> during integration tests and it disappears after tests have finished
> running? Then its recreated the next time tests run (perhaps being
> populated with test data).
>
>  I'm using Java.
>
>
>
>
>

Re: Running Cassandra in Integration Tests

[Ali Akhtar]

Ok, but that's not what this question is about.

I'm looking for a way to run an embedded cassandra instance which is
created & destroyed during tests and which doesn't persist any state
outside the tests.


On Fri, Oct 7, 2016 at 5:10 AM, Kant Kodali  wrote:

> you dont need to look for cassandra java api to start/stop instance. you
> just need to write a shell script or python or java or any language to
> execute shell commands!
>
>
>
> On Thu, Oct 6, 2016 4:57 PM, Ali Akhtar ali.rac...@gmail.com wrote:
>
>> Okay.. but how would I start this instance? Is there a java api to
>> programmatically start / destroy an instance during tests?
>>
>> On Fri, Oct 7, 2016 at 4:56 AM, Kant Kodali  wrote:
>>
>> sure as long as that isolated instance is treated as separate cluster you
>> shouldn't run into any problems.
>>
>>
>>
>> On Thu, Oct 6, 2016 4:08 PM, Ali Akhtar ali.rac...@gmail.com wrote:
>>
>> Is it possible to create an isolated cassandra instance which is run
>> during integration tests and it disappears after tests have finished
>> running? Then its recreated the next time tests run (perhaps being
>> populated with test data).
>>
>>  I'm using Java.
>>
>>
>>
>>

Re: Running Cassandra in Integration Tests

[Ali Akhtar]

Okay.. but how would I start this instance? Is there a java api to
programmatically start / destroy an instance during tests?

On Fri, Oct 7, 2016 at 4:56 AM, Kant Kodali  wrote:

> sure as long as that isolated instance is treated as separate cluster you
> shouldn't run into any problems.
>
>
>
> On Thu, Oct 6, 2016 4:08 PM, Ali Akhtar ali.rac...@gmail.com wrote:
>
>> Is it possible to create an isolated cassandra instance which is run
>> during integration tests and it disappears after tests have finished
>> running? Then its recreated the next time tests run (perhaps being
>> populated with test data).
>>
>>  I'm using Java.
>>
>>
>>

Re: Running Cassandra in Integration Tests

[Kant Kodali]

sure as long as that isolated instance is treated as separate cluster you
shouldn't run into any problems.






On Thu, Oct 6, 2016 4:08 PM, Ali Akhtar ali.rac...@gmail.com
wrote:
Is it possible to create an isolated cassandra instance which is run during
integration tests and it disappears after tests have finished running? Then its
recreated the next time tests run (perhaps being populated with test data).
I'm using Java.

Running Cassandra in Integration Tests

[Ali Akhtar]

Is it possible to create an isolated cassandra instance which is run during
integration tests and it disappears after tests have finished running? Then
its recreated the next time tests run (perhaps being populated with test
data).

 I'm using Java.

RE: Rationale for using Hazelcast in front of Cassandra?

[KARR, DAVID]

Clearly, with “traditional” RDBMSs, you tend to put a cache “close to” the 
client.  However, I was under the impression that Cassandra nodes could be 
positioned “close to” their clients, and Cassandra has its own cache (I 
believe), so how effective would it be to put a cache in front of a cache?

From: Dorian Hoxha [mailto:dorian.ho...@gmail.com]
Sent: Thursday, October 06, 2016 2:52 PM
To: user@cassandra.apache.org
Subject: Re: Rationale for using Hazelcast in front of Cassandra?

Maybe when you can have very hot keys that can give trouble to your 
3(replication) cassandra nodes ?
Example: why does facebook use memcache ? They certainly have things 
distributed on thousands of servers.

On Thu, Oct 6, 2016 at 11:40 PM, KARR, DAVID 
> wrote:
I've seen use cases that briefly describe using Hazelcast as a "front-end" for 
Cassandra, perhaps as a cache.  This seems counterintuitive to me.  Can someone 
describe to me when this kind of architecture might make sense?

Re: Rationale for using Hazelcast in front of Cassandra?

[Dorian Hoxha]

Maybe when you can have very hot keys that can give trouble to your
3(replication) cassandra nodes ?
Example: why does facebook use memcache ? They certainly have things
distributed on thousands of servers.

On Thu, Oct 6, 2016 at 11:40 PM, KARR, DAVID  wrote:

> I've seen use cases that briefly describe using Hazelcast as a "front-end"
> for Cassandra, perhaps as a cache.  This seems counterintuitive to me.  Can
> someone describe to me when this kind of architecture might make sense?
>

Rationale for using Hazelcast in front of Cassandra?

[KARR, DAVID]

I've seen use cases that briefly describe using Hazelcast as a "front-end" for 
Cassandra, perhaps as a cache.  This seems counterintuitive to me.  Can someone 
describe to me when this kind of architecture might make sense?

Re: Tombstoned error and then OOM

[kurt Greaves]

you'll still need to query all the data even if it's secondary indexed.

On 4 October 2016 at 17:13, INDRANIL BASU  wrote:

> The query has a where clause on a column which is a secondary index in the
> column family.
> E.g
> select * from test_schema.test_cf where status = 0;
> Here the status is integer column which is indexed.
>
> -- IB
>
> --
> *From:* kurt Greaves 
> *To:* user@cassandra.apache.org; INDRANIL BASU 
> *Sent:* Tuesday, 4 October 2016 10:38 PM
> *Subject:* Re: Tombstoned error and then OOM
>
> This sounds like you're running a query that consumes a lot of memory. Are
> you by chance querying a very large partition or not bounding your query?
>
> I'd also recommend upgrading to 2.1.15, 2.1.0 is very old and has quite a
> few bugs.
>
> On 3 October 2016 at 17:08, INDRANIL BASU  wrote:
>
> Hello All,
>
>
>
> I am getting the below error repeatedly in the system log of C* 2.1.0
>
> WARN  [SharedPool-Worker-64] 2016-09-27 00:43:35,835
> SliceQueryFilter.java:236 - Read 0 live and 1923 tombstoned cells in
> test_schema.test_cf.test_cf_ col1_idx (see tombstone_warn_threshold). 5000
> columns was requested, slices=[-], delInfo={deletedAt=-
> 9223372036854775808, localDeletion=2147483647}
>
> After that NullPointer Exception and finally OOM
>
> ERROR [CompactionExecutor:6287] 2016-09-29 22:09:13,546
> CassandraDaemon.java:166 - Exception in thread Thread[CompactionExecutor:
> 6287,1,main]
> java.lang. NullPointerException: null
> at org.apache.cassandra.service. CacheService$
> KeyCacheSerializer.serialize( CacheService.java:475)
> ~[apache-cassandra-2.1.0.jar: 2.1.0]
> at org.apache.cassandra.service. CacheService$
> KeyCacheSerializer.serialize( CacheService.java:463)
> ~[apache-cassandra-2.1.0.jar: 2.1.0]
> at org.apache.cassandra.cache. AutoSavingCache$Writer.
> saveCache(AutoSavingCache. java:225) ~[apache-cassandra-2.1.0.jar: 2.1.0]
> at org.apache.cassandra.db. compaction.CompactionManager$
> 11.run(CompactionManager.java: 1061) ~[apache-cassandra-2.1.0.jar: 2.1.0]
> at java.util.concurrent. Executors$RunnableAdapter. call(Unknown
> Source) ~[na:1.7.0_80]
> at java.util.concurrent. FutureTask.run(Unknown Source)
> ~[na:1.7.0_80]
> at java.util.concurrent. ThreadPoolExecutor.runWorker( Unknown
> Source) [na:1.7.0_80]
> at java.util.concurrent. ThreadPoolExecutor$Worker.run( Unknown
> Source) [na:1.7.0_80]
> at java.lang.Thread.run(Unknown Source) [na:1.7.0_80]
> ERROR [CompactionExecutor:9712] 2016-10-01 10:09:13,871
> CassandraDaemon.java:166 - Exception in thread Thread[CompactionExecutor:
> 9712,1,main]
> java.lang. NullPointerException: null
> ERROR [CompactionExecutor:10070] 2016-10-01 14:09:14,154
> CassandraDaemon.java:166 - Exception in thread Thread[CompactionExecutor:
> 10070,1,main]
> java.lang. NullPointerException: null
> ERROR [CompactionExecutor:10413] 2016-10-01 18:09:14,265
> CassandraDaemon.java:166 - Exception in thread Thread[CompactionExecutor:
> 10413,1,main]
> java.lang. NullPointerException: null
> ERROR [MemtableFlushWriter:2396] 2016-10-01 20:28:27,425
> CassandraDaemon.java:166 - Exception in thread Thread[MemtableFlushWriter:
> 2396,5,main]
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method) ~[na:1.7.0_80]
> at java.lang.Thread.start(Unknown Source) ~[na:1.7.0_80]
> at java.util.concurrent. ThreadPoolExecutor.addWorker( Unknown
> Source) ~[na:1.7.0_80]
> at java.util.concurrent. ThreadPoolExecutor.
> processWorkerExit(Unknown Source) ~[na:1.7.0_80]
> at java.util.concurrent. ThreadPoolExecutor.runWorker( Unknown
> Source) ~[na:1.7.0_80]
> at java.util.concurrent. ThreadPoolExecutor$Worker.run( Unknown
> Source) ~[na:1.7.0_80]
> at java.lang.Thread.run(Unknown Source) ~[na:1.7.0_80]
>
> -- IB
>
>
>
>
>
>
>
>
> --
> Kurt Greaves
> k...@instaclustr.com
> www.instaclustr.com
>
>
>


-- 
Kurt Greaves
k...@instaclustr.com
www.instaclustr.com

Re: Nodetool rebuild question

[Jeff Jirsa]

Read repairs (both foreground/blocking due to consistency level requirements 
and background/nonblocking due to table option/probability) will go memTable -> 
flush -> sstable.

 

 

 

From: Anubhav Kale 
Reply-To: "user@cassandra.apache.org" 
Date: Thursday, October 6, 2016 at 11:50 AM
To: "user@cassandra.apache.org" 
Subject: RE: Nodetool rebuild question

 

Sure. 

 

When a read repair happens, does it go via the memtable -> SS Table route OR 
does the source node send SS Table tmp files directly to inconsistent replica ?

 

From: Jeff Jirsa [mailto:jeff.ji...@crowdstrike.com] 
Sent: Wednesday, October 5, 2016 2:20 PM
To: user@cassandra.apache.org
Subject: Re: Nodetool rebuild question

 

If you set RF to 0, you can ignore my second sentence/paragraph. The third 
still applies.

 

 

From: Anubhav Kale 
Reply-To: "user@cassandra.apache.org" 
Date: Wednesday, October 5, 2016 at 1:56 PM
To: "user@cassandra.apache.org" 
Subject: RE: Nodetool rebuild question

 

Thanks. 

 

We always set RF to 0 and then “removenode” all nodes in the DC that we want to 
decom. So, I highly doubt that is the problem. Plus, #SSTables on a given node 
on average is ~2000 (we have 140 nodes in one ring and two rings overall).

 

From: Jeff Jirsa [mailto:jeff.ji...@crowdstrike.com] 
Sent: Wednesday, October 5, 2016 1:44 PM
To: user@cassandra.apache.org
Subject: Re: Nodetool rebuild question

 

Both of your statements are true.

 

During your decom, you likely streamed LOTs of sstables to the remaining nodes 
(especially true if you didn’t drop the replication factor to 0 for the DC you 
decommissioned). Since those tens of thousands of sstables take a while to 
compact, if you then rebuild (or bootstrap) before compaction is done, you’ll 
get a LOT of extra sstables.

 

This is one of the reasons that people with large clusters don’t use vnodes – 
if you needed to bootstrap ~100 more nodes into a cluster, you’d have to wait 
potentially a day or more per node to compact away the leftovers before 
bootstrapping the next, which is prohibitive at scale. 

 

-  Jeff

 

From: Anubhav Kale 
Reply-To: "user@cassandra.apache.org" 
Date: Wednesday, October 5, 2016 at 1:34 PM
To: "user@cassandra.apache.org" 
Subject: Nodetool rebuild question

 

Hello,

 

As part of rebuild, I noticed that the destination node gets -tmp- files from 
other nodes. Are following statements correct ?

 

1.   The files are written to disk without going through memtables.

2.   Regular compactors eventually compact them to bring down # SSTables to 
a reasonable number.

 

We have noticed that the destination node has created > 40K *Data* files in 
first hour of streaming itself. We have not seen such pattern before, so trying 
to understand what could have changed. (We do use Vnodes and We haven’t 
increased # nodes recently, but have decomm-ed a DC). 

 

Thanks much !


CONFIDENTIALITY NOTE: This e-mail and any attachments are confidential and may 
be legally privileged. If you are not the intended recipient, do not disclose, 
copy, distribute, or use this email or any attachments. If you have received 
this in error please let the sender know and then delete the email and all 
attachments.


CONFIDENTIALITY NOTE: This e-mail and any attachments are confidential and may 
be legally privileged. If you are not the intended recipient, do not disclose, 
copy, distribute, or use this email or any attachments. If you have received 
this in error please let the sender know and then delete the email and all 
attachments.


CONFIDENTIALITY NOTE: This e-mail and any attachments are confidential and may 
be legally privileged. If you are not the intended recipient, do not disclose, 
copy, distribute, or use this email or any attachments. If you have received 
this in error please let the sender know and then delete the email and all 
attachments.


smime.p7s
Description: S/MIME cryptographic signature

RE: Nodetool rebuild question

[Anubhav Kale]

Sure.

When a read repair happens, does it go via the memtable -> SS Table route OR 
does the source node send SS Table tmp files directly to inconsistent replica ?

From: Jeff Jirsa [mailto:jeff.ji...@crowdstrike.com]
Sent: Wednesday, October 5, 2016 2:20 PM
To: user@cassandra.apache.org
Subject: Re: Nodetool rebuild question

If you set RF to 0, you can ignore my second sentence/paragraph. The third 
still applies.


From: Anubhav Kale 
>
Reply-To: "user@cassandra.apache.org" 
>
Date: Wednesday, October 5, 2016 at 1:56 PM
To: "user@cassandra.apache.org" 
>
Subject: RE: Nodetool rebuild question

Thanks.

We always set RF to 0 and then “removenode” all nodes in the DC that we want to 
decom. So, I highly doubt that is the problem. Plus, #SSTables on a given node 
on average is ~2000 (we have 140 nodes in one ring and two rings overall).

From: Jeff Jirsa [mailto:jeff.ji...@crowdstrike.com]
Sent: Wednesday, October 5, 2016 1:44 PM
To: user@cassandra.apache.org
Subject: Re: Nodetool rebuild question

Both of your statements are true.

During your decom, you likely streamed LOTs of sstables to the remaining nodes 
(especially true if you didn’t drop the replication factor to 0 for the DC you 
decommissioned). Since those tens of thousands of sstables take a while to 
compact, if you then rebuild (or bootstrap) before compaction is done, you’ll 
get a LOT of extra sstables.

This is one of the reasons that people with large clusters don’t use vnodes – 
if you needed to bootstrap ~100 more nodes into a cluster, you’d have to wait 
potentially a day or more per node to compact away the leftovers before 
bootstrapping the next, which is prohibitive at scale.


-  Jeff

From: Anubhav Kale 
>
Reply-To: "user@cassandra.apache.org" 
>
Date: Wednesday, October 5, 2016 at 1:34 PM
To: "user@cassandra.apache.org" 
>
Subject: Nodetool rebuild question

Hello,

As part of rebuild, I noticed that the destination node gets -tmp- files from 
other nodes. Are following statements correct ?


1.   The files are written to disk without going through memtables.

2.   Regular compactors eventually compact them to bring down # SSTables to 
a reasonable number.

We have noticed that the destination node has created > 40K *Data* files in 
first hour of streaming itself. We have not seen such pattern before, so trying 
to understand what could have changed. (We do use Vnodes and We haven’t 
increased # nodes recently, but have decomm-ed a DC).

Thanks much !

CONFIDENTIALITY NOTE: This e-mail and any attachments are confidential and may 
be legally privileged. If you are not the intended recipient, do not disclose, 
copy, distribute, or use this email or any attachments. If you have received 
this in error please let the sender know and then delete the email and all 
attachments.

CONFIDENTIALITY NOTE: This e-mail and any attachments are confidential and may 
be legally privileged. If you are not the intended recipient, do not disclose, 
copy, distribute, or use this email or any attachments. If you have received 
this in error please let the sender know and then delete the email and all 
attachments.

Re: Partition Key - Wide rows?

[Saladi Naidu]

It depends on Partition/Primary key design. In order to execute all 3 queries, 
Partition Key is Org id and others are Clustering keys. if there are many org's 
it will be ok, but if it is one org then a single partition  will hold all the 
data and its not good Naidu Saladi 
 

On Thursday, October 6, 2016 12:14 PM, Ali Akhtar  
wrote:
 

 Thanks, Phil.
1- In my use-case, its probably okay to partition all the org data together. 
This is for a b2b enterprise SaaS application, the customers will be 
organizations.
So it is probably okay to store each org's data next to each other, right?
2- I'm thinking of having the primary key be: (org_id, team_id, project_id, 
issue_id). 
In the above case, will there be a skinny row per issue, or a wide row per org 
/ team / project?
3- Just to double check, with the above primary key, can I still query using 
just the org_id, org + team id, and org + team + project id?
4- If I wanted to refer to a particular issue, it looks like I'd need to send 
all 4 parameters. That may be problematic. Is there a better way of modeling 
this data?


On Thu, Oct 6, 2016 at 9:30 PM, Philip Persad  wrote:



1) No.  Your first 3 queries will work but not the last one (get issue by id).  
In Cassandra when you query you must include every preceding portion of the 
primary key.

2) 64 bytes (16 * 4), or somewhat more if storing as strings?  I don't think 
that's something I'd worry too much about.

3) Depends on how you build your partition key.  If partition key is (org id), 
then you get one partition per org (probably bad depending on your dataset).  
If partition key is (org id, team id, project id) then you will have one 
partition per project which is probably fine ( again, depending on your 
dataset).

Cheers,

-PhilFrom: Ali Akhtar
Sent: ‎2016-‎10-‎06 9:04 AM
To: user@cassandra.apache.org
Subject: Partition Key - Wide rows?

Heya,
I'm designing some tables, where data needs to be stored in the following 
hierarchy:
Organization -> Team -> Project -> Issues
I need to be able to retrieve issues:
- For the whole org - using org id- For a team (org id + team id)- For a 
project (org id + team id + project id)- If possible, by using just the issue id
I'm considering using all 4 ids as the primary key. The first 3 will use UUIDs, 
except issue id which will be an alphanumeric string, unique per project.
1) Will this setup allow using all 4 query scenarios?2) Will this make the 
primary key really long, 3 UUIDs + similar length'd issue id?3) Will this store 
issues as skinny rows, or wide rows? If an org has a lot of teams, which have a 
lot of projects, which have a lot of issues, etc, could I have issues w/ 
running out of the column limit of wide rows?4) Is there a better way of 
achieving this scenario?







   

Re: Partition Key - Wide rows?

[Jonathan Haddad]

>  In my use-case, its probably okay to partition all the org data together.


Maybe, maybe not.  Cassandra doesn't handle really big partitions very well
right now.  If you've got more than 100MB of data per org, you're better
off breaking it up (by project or team) and doing multiple queries to
stitch the data together client side.



On Thu, Oct 6, 2016 at 10:14 AM Ali Akhtar  wrote:

> Thanks, Phil.
>
> 1- In my use-case, its probably okay to partition all the org data
> together. This is for a b2b enterprise SaaS application, the customers will
> be organizations.
>
> So it is probably okay to store each org's data next to each other, right?
>
> 2- I'm thinking of having the primary key be: (org_id, team_id,
> project_id, issue_id).
>
> In the above case, will there be a skinny row per issue, or a wide row per
> org / team / project?
>
> 3- Just to double check, with the above primary key, can I still query
> using just the org_id, org + team id, and org + team + project id?
>
> 4- If I wanted to refer to a particular issue, it looks like I'd need to
> send all 4 parameters. That may be problematic. Is there a better way of
> modeling this data?
>
>
>
> On Thu, Oct 6, 2016 at 9:30 PM, Philip Persad 
> wrote:
>
>
>
> 1) No.  Your first 3 queries will work but not the last one (get issue by
> id).  In Cassandra when you query you must include every preceding portion
> of the primary key.
>
> 2) 64 bytes (16 * 4), or somewhat more if storing as strings?  I don't
> think that's something I'd worry too much about.
>
> 3) Depends on how you build your partition key.  If partition key is (org
> id), then you get one partition per org (probably bad depending on your
> dataset).  If partition key is (org id, team id, project id) then you will
> have one partition per project which is probably fine ( again, depending on
> your dataset).
>
> Cheers,
>
> -Phil
> --
> From: Ali Akhtar 
> Sent: ‎2016-‎10-‎06 9:04 AM
> To: user@cassandra.apache.org
> Subject: Partition Key - Wide rows?
>
> Heya,
>
> I'm designing some tables, where data needs to be stored in the following
> hierarchy:
>
> Organization -> Team -> Project -> Issues
>
> I need to be able to retrieve issues:
>
> - For the whole org - using org id
> - For a team (org id + team id)
> - For a project (org id + team id + project id)
> - If possible, by using just the issue id
>
> I'm considering using all 4 ids as the primary key. The first 3 will use
> UUIDs, except issue id which will be an alphanumeric string, unique per
> project.
>
> 1) Will this setup allow using all 4 query scenarios?
> 2) Will this make the primary key really long, 3 UUIDs + similar length'd
> issue id?
> 3) Will this store issues as skinny rows, or wide rows? If an org has a
> lot of teams, which have a lot of projects, which have a lot of issues,
> etc, could I have issues w/ running out of the column limit of wide rows?
> 4) Is there a better way of achieving this scenario?
>
>
>
>
>
>

Re: Partition Key - Wide rows?

[Ali Akhtar]

Thanks, Phil.

1- In my use-case, its probably okay to partition all the org data
together. This is for a b2b enterprise SaaS application, the customers will
be organizations.

So it is probably okay to store each org's data next to each other, right?

2- I'm thinking of having the primary key be: (org_id, team_id, project_id,
issue_id).

In the above case, will there be a skinny row per issue, or a wide row per
org / team / project?

3- Just to double check, with the above primary key, can I still query
using just the org_id, org + team id, and org + team + project id?

4- If I wanted to refer to a particular issue, it looks like I'd need to
send all 4 parameters. That may be problematic. Is there a better way of
modeling this data?



On Thu, Oct 6, 2016 at 9:30 PM, Philip Persad 
wrote:

>
>
> 1) No.  Your first 3 queries will work but not the last one (get issue by
> id).  In Cassandra when you query you must include every preceding portion
> of the primary key.
>
> 2) 64 bytes (16 * 4), or somewhat more if storing as strings?  I don't
> think that's something I'd worry too much about.
>
> 3) Depends on how you build your partition key.  If partition key is (org
> id), then you get one partition per org (probably bad depending on your
> dataset).  If partition key is (org id, team id, project id) then you will
> have one partition per project which is probably fine ( again, depending on
> your dataset).
>
> Cheers,
>
> -Phil
> --
> From: Ali Akhtar 
> Sent: ‎2016-‎10-‎06 9:04 AM
> To: user@cassandra.apache.org
> Subject: Partition Key - Wide rows?
>
> Heya,
>
> I'm designing some tables, where data needs to be stored in the following
> hierarchy:
>
> Organization -> Team -> Project -> Issues
>
> I need to be able to retrieve issues:
>
> - For the whole org - using org id
> - For a team (org id + team id)
> - For a project (org id + team id + project id)
> - If possible, by using just the issue id
>
> I'm considering using all 4 ids as the primary key. The first 3 will use
> UUIDs, except issue id which will be an alphanumeric string, unique per
> project.
>
> 1) Will this setup allow using all 4 query scenarios?
> 2) Will this make the primary key really long, 3 UUIDs + similar length'd
> issue id?
> 3) Will this store issues as skinny rows, or wide rows? If an org has a
> lot of teams, which have a lot of projects, which have a lot of issues,
> etc, could I have issues w/ running out of the column limit of wide rows?
> 4) Is there a better way of achieving this scenario?
>
>
>
>
>

Partition Key - Wide rows?

[Ali Akhtar]

Heya,

I'm designing some tables, where data needs to be stored in the following
hierarchy:

Organization -> Team -> Project -> Issues

I need to be able to retrieve issues:

- For the whole org - using org id
- For a team (org id + team id)
- For a project (org id + team id + project id)
- If possible, by using just the issue id

I'm considering using all 4 ids as the primary key. The first 3 will use
UUIDs, except issue id which will be an alphanumeric string, unique per
project.

1) Will this setup allow using all 4 query scenarios?
2) Will this make the primary key really long, 3 UUIDs + similar length'd
issue id?
3) Will this store issues as skinny rows, or wide rows? If an org has a lot
of teams, which have a lot of projects, which have a lot of issues, etc,
could I have issues w/ running out of the column limit of wide rows?
4) Is there a better way of achieving this scenario?

Re: dtests jolokia fails to attach

[Benjamin Roth]

That did the trick! Thanks. I also recognized that one down in the
contribution.md
I guess it would be also a good idea to show this notice if jolokia fails
to attach.

Thanks guys!

2016-10-06 14:12 GMT+02:00 Marcus Eriksson :

> It is this: "-XX:+PerfDisableSharedMem" - in your dtest you need to do
> "remove_perf_disable_shared_mem(node1)" before starting the node
>
> /Marcus
>
> On Thu, Oct 6, 2016 at 8:30 AM, Benjamin Roth 
> wrote:
>
>> Maybe additional information, this is the CS command line for ccm node1:
>>
>> br   20376  3.2  8.6 2331136 708308 pts/5  Sl   06:10   0:30 java
>> -Xloggc:/home/br/.ccm/test/node1/logs/gc.log -ea
>> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42
>> -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103
>> -XX:+AlwaysPreTouch -XX:-UseBiasedLocking -XX:+UseTLAB -XX:+ResizeTLAB
>> -XX:+UseNUMA -XX:+PerfDisableSharedMem -Djava.net.preferIPv4Stack=true
>> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
>> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
>> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
>> -XX:CMSWaitDuration=1 -XX:+CMSParallelInitialMarkEnabled
>> -XX:+CMSEdenChunksRecordAlways -XX:+CMSClassUnloadingEnabled
>> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC
>> -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
>> -XX:+PrintPromotionFailure -XX:+UseGCLogFileRotation
>> -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M -Xms500M -Xmx500M -Xmn50M
>> -XX:+UseCondCardMark -XX:CompileCommandFile=/home/b
>> r/.ccm/test/node1/conf/hotspot_compiler 
>> -javaagent:/home/br/repos/cassandra/lib/jamm-0.3.0.jar
>> -Dcassandra.jmx.local.port=7100 
>> -Dcom.sun.management.jmxremote.authenticate=false
>> -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password
>> -Djava.library.path=/home/br/repos/cassandra/lib/sigar-bin
>> -Dcassandra.migration_task_wait_in_seconds=6
>> -Dcassandra.libjemalloc=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1
>> -Dlogback.configurationFile=logback.xml -Dcassandra.logdir=/var/log/cassandra
>> -Dcassandra.storagedir=/home/br/repos/cassandra/data
>> -Dcassandra-pidfile=/home/br/.ccm/test/node1/cassandra.pid -cp
>> /home/br/.ccm/test/node1/conf:/home/br/repos/cassandra/build
>> /classes/main:/home/br/repos/cassandra/build/classes/thrift
>> :/home/br/repos/cassandra/lib/HdrHistogram-2.1.9.jar:/home/
>> br/repos/cassandra/lib/ST4-4.0.8.jar:/home/br/repos/
>> cassandra/lib/airline-0.6.jar:/home/br/repos/cassandra/lib/
>> antlr-runtime-3.5.2.jar:/home/br/repos/cassandra/lib/asm-5.
>> 0.4.jar:/home/br/repos/cassandra/lib/caffeine-2.2.6.
>> jar:/home/br/repos/cassandra/lib/cassandra-driver-core-3.0.
>> 1-shaded.jar:/home/br/repos/cassandra/lib/commons-cli-1.1.ja
>> r:/home/br/repos/cassandra/lib/commons-codec-1.2.jar:/home/
>> br/repos/cassandra/lib/commons-lang3-3.1.jar:/home/br/repos/
>> cassandra/lib/commons-math3-3.2.jar:/home/br/repos/
>> cassandra/lib/compress-lzf-0.8.4.jar:/home/br/repos/
>> cassandra/lib/concurrent-trees-2.4.0.jar:/home/br/
>> repos/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/
>> home/br/repos/cassandra/lib/disruptor-3.0.1.jar:/home/br/
>> repos/cassandra/lib/ecj-4.4.2.jar:/home/br/repos/cassandra/
>> lib/guava-18.0.jar:/home/br/repos/cassandra/lib/high-
>> scale-lib-1.0.6.jar:/home/br/repos/cassandra/lib/hppc-0.5.
>> 4.jar:/home/br/repos/cassandra/lib/jackson-core-
>> asl-1.9.2.jar:/home/br/repos/cassandra/lib/jackson-mapper-
>> asl-1.9.2.jar:/home/br/repos/cassandra/lib/jamm-0.3.0.jar:/
>> home/br/repos/cassandra/lib/javax.inject.jar:/home/br/
>> repos/cassandra/lib/jbcrypt-0.3m.jar:/home/br/repos/
>> cassandra/lib/jcl-over-slf4j-1.7.7.jar:/home/br/repos/
>> cassandra/lib/jctools-core-1.2.1.jar:/home/br/repos/cassand
>> ra/lib/jflex-1.6.0.jar:/home/br/repos/cassandra/lib/jna-4.
>> 0.0.jar:/home/br/repos/cassandra/lib/joda-time-2.4.jar:/
>> home/br/repos/cassandra/lib/json-simple-1.1.jar:/home/br/
>> repos/cassandra/lib/libthrift-0.9.2.jar:/home/br/repos/
>> cassandra/lib/log4j-over-slf4j-1.7.7.jar:/home/br/repos
>> /cassandra/lib/logback-classic-1.1.3.jar:/home/br/repos/
>> cassandra/lib/logback-core-1.1.3.jar:/home/br/repos/cassand
>> ra/lib/lz4-1.3.0.jar:/home/br/repos/cassandra/lib/metrics-
>> core-3.1.0.jar:/home/br/repos/cassandra/lib/metrics-jvm-3.1.
>> 0.jar:/home/br/repos/cassandra/lib/metrics-logback-
>> 3.1.0.jar:/home/br/repos/cassandra/lib/netty-all-4.0.
>> 39.Final.jar:/home/br/repos/cassandra/lib/ohc-core-0.4.4.
>> jar:/home/br/repos/cassandra/lib/ohc-core-j8-0.4.4.jar:/
>> home/br/repos/cassandra/lib/primitive-1.0.jar:/home/br/
>> repos/cassandra/lib/reporter-config-base-3.0.0.jar:/home/
>> br/repos/cassandra/lib/reporter-config3-3.0.0.jar:/
>> home/br/repos/cassandra/lib/sigar-1.6.4.jar:/home/br/
>> repos/cassandra/lib/slf4j-api-1.7.7.jar:/home/br/repos/
>> cassandra/lib/snakeyaml-1.11.jar:/home/br/repos/cassandra/
>> 

Re: system_distributed.repair_history table

[Chris Lohfink]

small reminder that unless you have autosnapshot to false in
cassandra.yaml, you will need to clear snapshot (nodetool
clearsnapshot system_distributed) to actually delete the sstables

On Thu, Oct 6, 2016 at 9:25 AM, Saladi Naidu  wrote:

> Thanks for the response. It makes sense to periodically truncate as it is
> only for debugging purposes
>
> Naidu Saladi
>
>
> On Wednesday, October 5, 2016 8:03 PM, Chris Lohfink 
> wrote:
>
>
> The only current solution is to truncate it periodically. I opened
> https://issues.apache.org/jira/browse/CASSANDRA-12701 about it if
> interested in following
>
> On Wed, Oct 5, 2016 at 4:23 PM, Saladi Naidu 
> wrote:
>
> We are seeing following warnings in system.log,  As *compaction_large_
> partition_warning_threshold_mb*   in cassandra.yaml file is as default
> value 100, we are seeing these warnings
>
> 110:WARN  [CompactionExecutor:91798] 2016-10-05 00:54:05,554
> BigTableWriter.java:184 - Writing large partition
> system_distributed/repair_ history:gccatmer:mer_admin_job (115943239 bytes)
>
> 111:WARN  [CompactionExecutor:91798] 2016-10-05 00:54:13,303 
> BigTableWriter.java:184 - Writing large partition system_distributed/repair_ 
> history:gcconfigsrvcks:user_ activation (163926097 bytes)
>
>
> When I looked at the table definition it is partitioned by keyspace and 
> cloumnfamily, under this partition, repair history is maintained. When I 
> looked at the count of rows in this partition, most of the paritions have 
> >200,000 rows and these will keep growing because of the partition strategy 
> right. There is no TTL on this so any idea what is the solution for reducing 
> partition size.
>
>
> I also looked at size_estimates table for this column family and found that 
> the mean partition size for each range is 50,610,179 which is very large 
> compared to any other tables.
>
>
>
>
>

Re: system_distributed.repair_history table

[Saladi Naidu]

Thanks for the response. It makes sense to periodically truncate as it is only 
for debugging purposes Naidu Saladi 
 

On Wednesday, October 5, 2016 8:03 PM, Chris Lohfink  
wrote:
 

 The only current solution is to truncate it periodically. I opened 
https://issues.apache.org/jira/browse/CASSANDRA-12701 about it if interested in 
following
On Wed, Oct 5, 2016 at 4:23 PM, Saladi Naidu  wrote:

We are seeing following warnings in system.log,  As compaction_large_ 
partition_warning_threshold_mb   in cassandra.yaml file is as default value 
100, we are seeing these warnings
110:WARN  [CompactionExecutor:91798] 2016-10-05 00:54:05,554 
BigTableWriter.java:184 - Writing large partition system_distributed/repair_ 
history:gccatmer:mer_admin_job (115943239 bytes)
111:WARN  [CompactionExecutor:91798] 2016-10-05 00:54:13,303 
BigTableWriter.java:184 - Writing large partition system_distributed/repair_ 
history:gcconfigsrvcks:user_ activation (163926097 bytes)
When I looked at the table definition it is partitioned by keyspace and 
cloumnfamily, under this partition, repair history is maintained. When I looked 
at the count of rows in this partition, most of the paritions have >200,000 
rows and these will keep growing because of the partition strategy right. There 
is no TTL on this so any idea what is the solution for reducing partition size. 

I also looked at size_estimates table for this column family and found that the 
mean partition size for each range is 50,610,179 which is very large compared 
to any other tables. 



   

Re: Little question

[Alain RODRIGUEZ]

Hi Rubén,

Good luck with your adoptive Cassandra cluster :-).

Some thoughts:

You are struggling with balancing your cluster, there is a lot of
documentation about how the Cassandra Architecture work.

Some insight to help you understand what your setup is while reading.

- You are *not* using vnodes (1 token range per node)
- You are using Random Partitioner.

About your current issue, I'll use an OODA loop as the content is quite
long, it helps me structuring my ideas, improve completeness, and it will
hopefully make it easier to read on your side. It's up to you to follow
this plan or not (Decide / Act) parts.

Observe

When using 5 nodes, *no* vnodes and random partitioner the token
(initial_token in cassandra.yaml) in use should be:

- 0
- 34028236692093846346337460743176821145
- 68056473384187692692674921486353642290
- 102084710076281539039012382229530463435
- 136112946768375385385349842972707284580

For 6 nodes:

- 0
- 28356863910078205288614550619314017621
*- 56713727820156410577229101238628035242*
- 85070591730234615865843651857942052863
- 113427455640312821154458202477256070484
- 141784319550391026443072753096570088105

Using http://www.geroba.com/cassandra/cassandra-token-calculator/, but
there are many tools like this one, just pick one :-).

Orient

So it looks like your setup is built to run with 6 nodes and as of now only
have 5.

Your cluster is then *not* well balanced. This cluster where a node
(60.206) was added recently and an other node has been removed recently,
probably sitting on Token '56713727820156410577229101238628035242'. When
removing this node the balance was broken.

10.128.60.106 datacenter1 rack1 Up Normal 262.12 GB 33.33%
> 85070591730234615865843651857942052863


This node is holding a percentage of the data twice bigger than initially
planned. There is a replication factor of 3, so 2 other nodes are having
more data than expected as well. This cluster needs to be balanced to avoid
making 3 of those nodes being more loaded (size and throughput) than the
other 2. Cassandra can live with it, not sure about your server. If using
the same hardware everywhere, it makes sense to try balancing it.

Other notes:

- Using 'nodetool status ' instead of 'nodetool ring' or
'nodetool status' is a good habit you might want to take as RF and
ownership is defined at the Keyspace level. I believe this has no impacts
in the current issue.

- As you added a new node recently, some old nodes might be still holding
data they are not owning anymore. To address this you need to run a
nodetool cleanup on *all* the nodes *excepted* the last one you added.

This latter point combined with the imbalances would explain that the new
node owns a different amount of data.

Decide

So I tend to agree with you that some data should be cleaned:

Is that difference data that is not cleaned up, such as TTL-expired cell or
> tombstoned data?


But I would say that nodes are holding data out of their primary and
replicated ranges. We need to get rid of this extra data that is no longer
used anyway.

We also want to balance the cluster if using the same hardware on all the
nodes.

Act

Here is what I would probably do

First fix the balance, this means you need to either:

- "move" (nodetool move) nodes around in order to have a well balanced 5
node cluster (you might want to read more about it if going this way).

*OR*

- add a 6th node with
'initial_token: 56713727820156410577229101238628035242'

As 'nodetool move' is quite an heavy operation involving a few nodes and a
lot of streaming, to be performed on each node except one, I would
recommend adding a node, even more if you are still learning about
Cassandra.

Once you're happy with the cluster balance, run "nodetool cleanup" on all
the nodes. It's a local operation that can be run simultaneously on many /
all the nodes, as long as there are resources available, as it is a bit I/O
and CPU intensive.

Then check again the balance (nodetool status ). Due to the
compacton state you can have discrepancies but it should be far better.

C*heers,

---
Alain Rodriguez - @arodream - al...@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com



2016-10-04 15:57 GMT+02:00 Ruben Cardenal :

> Hi,
>
> We've inherited quite a big amazon infrastructure from a company we've
> purchased. It's has an ancient and obsolete implementation of services,
> being the worst (and more expensive) of all of them a 5 cluster of
> Cassandra (RF=3). I'm new to Cassandra, and yes, I'm doing my way
> throughout docs.
>
> I was told that Amazon asked them a few months ago to reboot one of their
> servers (it had been turned on for so long that Amazon had to make some
> changes and needed it rebooted), so they had to add a new node to the
> cluster. If you query nodetool as of now, it shows:
>
> $ nodetool ring
> Note: Ownership information does not include topology, please specify a
> keyspace.
> 

Re: dtests jolokia fails to attach

[Marcus Eriksson]

It is this: "-XX:+PerfDisableSharedMem" - in your dtest you need to do
"remove_perf_disable_shared_mem(node1)" before starting the node

/Marcus

On Thu, Oct 6, 2016 at 8:30 AM, Benjamin Roth 
wrote:

> Maybe additional information, this is the CS command line for ccm node1:
>
> br   20376  3.2  8.6 2331136 708308 pts/5  Sl   06:10   0:30 java
> -Xloggc:/home/br/.ccm/test/node1/logs/gc.log -ea -XX:+UseThreadPriorities
> -XX:ThreadPriorityPolicy=42 -XX:+HeapDumpOnOutOfMemoryError -Xss256k
> -XX:StringTableSize=103 -XX:+AlwaysPreTouch -XX:-UseBiasedLocking
> -XX:+UseTLAB -XX:+ResizeTLAB -XX:+UseNUMA -XX:+PerfDisableSharedMem
> -Djava.net.preferIPv4Stack=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
> -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
> -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 
> -XX:+UseCMSInitiatingOccupancyOnly
> -XX:CMSWaitDuration=1 -XX:+CMSParallelInitialMarkEnabled
> -XX:+CMSEdenChunksRecordAlways -XX:+CMSClassUnloadingEnabled
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC
> -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintPromotionFailure -XX:+UseGCLogFileRotation
> -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M -Xms500M -Xmx500M -Xmn50M
> -XX:+UseCondCardMark 
> -XX:CompileCommandFile=/home/br/.ccm/test/node1/conf/hotspot_compiler
> -javaagent:/home/br/repos/cassandra/lib/jamm-0.3.0.jar
> -Dcassandra.jmx.local.port=7100 
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password
> -Djava.library.path=/home/br/repos/cassandra/lib/sigar-bin
> -Dcassandra.migration_task_wait_in_seconds=6 -Dcassandra.libjemalloc=/usr/
> lib/x86_64-linux-gnu/libjemalloc.so.1 -Dlogback.configurationFile=logback.xml
> -Dcassandra.logdir=/var/log/cassandra 
> -Dcassandra.storagedir=/home/br/repos/cassandra/data
> -Dcassandra-pidfile=/home/br/.ccm/test/node1/cassandra.pid -cp
> /home/br/.ccm/test/node1/conf:/home/br/repos/cassandra/
> build/classes/main:/home/br/repos/cassandra/build/classes/
> thrift:/home/br/repos/cassandra/lib/HdrHistogram-2.1.9.jar:/home/br/repos/
> cassandra/lib/ST4-4.0.8.jar:/home/br/repos/cassandra/lib/
> airline-0.6.jar:/home/br/repos/cassandra/lib/antlr-
> runtime-3.5.2.jar:/home/br/repos/cassandra/lib/asm-5.0.4.
> jar:/home/br/repos/cassandra/lib/caffeine-2.2.6.jar:/home/
> br/repos/cassandra/lib/cassandra-driver-core-3.0.1-
> shaded.jar:/home/br/repos/cassandra/lib/commons-cli-1.1.
> jar:/home/br/repos/cassandra/lib/commons-codec-1.2.jar:/
> home/br/repos/cassandra/lib/commons-lang3-3.1.jar:/home/
> br/repos/cassandra/lib/commons-math3-3.2.jar:/home/br/repos/cassandra/lib/
> compress-lzf-0.8.4.jar:/home/br/repos/cassandra/lib/
> concurrent-trees-2.4.0.jar:/home/br/repos/cassandra/lib/
> concurrentlinkedhashmap-lru-1.4.jar:/home/br/repos/
> cassandra/lib/disruptor-3.0.1.jar:/home/br/repos/cassandra/
> lib/ecj-4.4.2.jar:/home/br/repos/cassandra/lib/guava-18.
> 0.jar:/home/br/repos/cassandra/lib/high-scale-lib-
> 1.0.6.jar:/home/br/repos/cassandra/lib/hppc-0.5.4.jar:/
> home/br/repos/cassandra/lib/jackson-core-asl-1.9.2.jar:/
> home/br/repos/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/
> home/br/repos/cassandra/lib/jamm-0.3.0.jar:/home/br/repos/
> cassandra/lib/javax.inject.jar:/home/br/repos/cassandra/
> lib/jbcrypt-0.3m.jar:/home/br/repos/cassandra/lib/jcl-over-
> slf4j-1.7.7.jar:/home/br/repos/cassandra/lib/jctools-
> core-1.2.1.jar:/home/br/repos/cassandra/lib/jflex-1.6.0.jar:
> /home/br/repos/cassandra/lib/jna-4.0.0.jar:/home/br/repos/
> cassandra/lib/joda-time-2.4.jar:/home/br/repos/cassandra/
> lib/json-simple-1.1.jar:/home/br/repos/cassandra/lib/
> libthrift-0.9.2.jar:/home/br/repos/cassandra/lib/log4j-
> over-slf4j-1.7.7.jar:/home/br/repos/cassandra/lib/logback-
> classic-1.1.3.jar:/home/br/repos/cassandra/lib/logback-
> core-1.1.3.jar:/home/br/repos/cassandra/lib/lz4-1.3.0.jar:/
> home/br/repos/cassandra/lib/metrics-core-3.1.0.jar:/home/
> br/repos/cassandra/lib/metrics-jvm-3.1.0.jar:/home/br/repos/cassandra/lib/
> metrics-logback-3.1.0.jar:/home/br/repos/cassandra/lib/
> netty-all-4.0.39.Final.jar:/home/br/repos/cassandra/lib/
> ohc-core-0.4.4.jar:/home/br/repos/cassandra/lib/ohc-core-
> j8-0.4.4.jar:/home/br/repos/cassandra/lib/primitive-1.0.
> jar:/home/br/repos/cassandra/lib/reporter-config-base-3.0.
> 0.jar:/home/br/repos/cassandra/lib/reporter-config3-3.0.0.jar:/home/br/
> repos/cassandra/lib/sigar-1.6.4.jar:/home/br/repos/
> cassandra/lib/slf4j-api-1.7.7.jar:/home/br/repos/cassandra/
> lib/snakeyaml-1.11.jar:/home/br/repos/cassandra/lib/snappy-
> java-1.1.1.7.jar:/home/br/repos/cassandra/lib/snowball-
> stemmer-1.3.0.581.1.jar:/home/br/repos/cassandra/lib/stream-
> 2.5.2.jar:/home/br/repos/cassandra/lib/thrift-server-0.
> 3.7.jar:/home/br/repos/cassandra/lib/jsr223/*/*.jar
> -Dcassandra.join_ring=True -Dcassandra.logdir=/home/br/.ccm/test/node1/logs
> -Dcassandra.boot_without_jna=true 

Re: dtests jolokia fails to attach

[Paulo Motta]

I had this problem before but can't remember the root cause, but I think it
was related to conflicting JVMs on the same machine. Can you check if you
have more than one JVM installed and try to define JAVA_HOME if it's not
defined?

Maybe this is related: https://github.com/rhuss/jolokia/issues/189

2016-10-06 3:30 GMT-03:00 Benjamin Roth :

> Maybe additional information, this is the CS command line for ccm node1:
>
> br   20376  3.2  8.6 2331136 708308 pts/5  Sl   06:10   0:30 java
> -Xloggc:/home/br/.ccm/test/node1/logs/gc.log -ea -XX:+UseThreadPriorities
> -XX:ThreadPriorityPolicy=42 -XX:+HeapDumpOnOutOfMemoryError -Xss256k
> -XX:StringTableSize=103 -XX:+AlwaysPreTouch -XX:-UseBiasedLocking
> -XX:+UseTLAB -XX:+ResizeTLAB -XX:+UseNUMA -XX:+PerfDisableSharedMem
> -Djava.net.preferIPv4Stack=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
> -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
> -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 
> -XX:+UseCMSInitiatingOccupancyOnly
> -XX:CMSWaitDuration=1 -XX:+CMSParallelInitialMarkEnabled
> -XX:+CMSEdenChunksRecordAlways -XX:+CMSClassUnloadingEnabled
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC
> -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintPromotionFailure -XX:+UseGCLogFileRotation
> -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M -Xms500M -Xmx500M -Xmn50M
> -XX:+UseCondCardMark 
> -XX:CompileCommandFile=/home/br/.ccm/test/node1/conf/hotspot_compiler
> -javaagent:/home/br/repos/cassandra/lib/jamm-0.3.0.jar
> -Dcassandra.jmx.local.port=7100 
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password
> -Djava.library.path=/home/br/repos/cassandra/lib/sigar-bin
> -Dcassandra.migration_task_wait_in_seconds=6 -Dcassandra.libjemalloc=/usr/
> lib/x86_64-linux-gnu/libjemalloc.so.1 -Dlogback.configurationFile=logback.xml
> -Dcassandra.logdir=/var/log/cassandra 
> -Dcassandra.storagedir=/home/br/repos/cassandra/data
> -Dcassandra-pidfile=/home/br/.ccm/test/node1/cassandra.pid -cp
> /home/br/.ccm/test/node1/conf:/home/br/repos/cassandra/
> build/classes/main:/home/br/repos/cassandra/build/classes/
> thrift:/home/br/repos/cassandra/lib/HdrHistogram-2.1.9.jar:/home/br/repos/
> cassandra/lib/ST4-4.0.8.jar:/home/br/repos/cassandra/lib/
> airline-0.6.jar:/home/br/repos/cassandra/lib/antlr-
> runtime-3.5.2.jar:/home/br/repos/cassandra/lib/asm-5.0.4.
> jar:/home/br/repos/cassandra/lib/caffeine-2.2.6.jar:/home/
> br/repos/cassandra/lib/cassandra-driver-core-3.0.1-
> shaded.jar:/home/br/repos/cassandra/lib/commons-cli-1.1.
> jar:/home/br/repos/cassandra/lib/commons-codec-1.2.jar:/
> home/br/repos/cassandra/lib/commons-lang3-3.1.jar:/home/
> br/repos/cassandra/lib/commons-math3-3.2.jar:/home/br/repos/cassandra/lib/
> compress-lzf-0.8.4.jar:/home/br/repos/cassandra/lib/
> concurrent-trees-2.4.0.jar:/home/br/repos/cassandra/lib/
> concurrentlinkedhashmap-lru-1.4.jar:/home/br/repos/
> cassandra/lib/disruptor-3.0.1.jar:/home/br/repos/cassandra/
> lib/ecj-4.4.2.jar:/home/br/repos/cassandra/lib/guava-18.
> 0.jar:/home/br/repos/cassandra/lib/high-scale-lib-
> 1.0.6.jar:/home/br/repos/cassandra/lib/hppc-0.5.4.jar:/
> home/br/repos/cassandra/lib/jackson-core-asl-1.9.2.jar:/
> home/br/repos/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/
> home/br/repos/cassandra/lib/jamm-0.3.0.jar:/home/br/repos/
> cassandra/lib/javax.inject.jar:/home/br/repos/cassandra/
> lib/jbcrypt-0.3m.jar:/home/br/repos/cassandra/lib/jcl-over-
> slf4j-1.7.7.jar:/home/br/repos/cassandra/lib/jctools-
> core-1.2.1.jar:/home/br/repos/cassandra/lib/jflex-1.6.0.jar:
> /home/br/repos/cassandra/lib/jna-4.0.0.jar:/home/br/repos/
> cassandra/lib/joda-time-2.4.jar:/home/br/repos/cassandra/
> lib/json-simple-1.1.jar:/home/br/repos/cassandra/lib/
> libthrift-0.9.2.jar:/home/br/repos/cassandra/lib/log4j-
> over-slf4j-1.7.7.jar:/home/br/repos/cassandra/lib/logback-
> classic-1.1.3.jar:/home/br/repos/cassandra/lib/logback-
> core-1.1.3.jar:/home/br/repos/cassandra/lib/lz4-1.3.0.jar:/
> home/br/repos/cassandra/lib/metrics-core-3.1.0.jar:/home/
> br/repos/cassandra/lib/metrics-jvm-3.1.0.jar:/home/br/repos/cassandra/lib/
> metrics-logback-3.1.0.jar:/home/br/repos/cassandra/lib/
> netty-all-4.0.39.Final.jar:/home/br/repos/cassandra/lib/
> ohc-core-0.4.4.jar:/home/br/repos/cassandra/lib/ohc-core-
> j8-0.4.4.jar:/home/br/repos/cassandra/lib/primitive-1.0.
> jar:/home/br/repos/cassandra/lib/reporter-config-base-3.0.
> 0.jar:/home/br/repos/cassandra/lib/reporter-config3-3.0.0.jar:/home/br/
> repos/cassandra/lib/sigar-1.6.4.jar:/home/br/repos/
> cassandra/lib/slf4j-api-1.7.7.jar:/home/br/repos/cassandra/
> lib/snakeyaml-1.11.jar:/home/br/repos/cassandra/lib/snappy-
> java-1.1.1.7.jar:/home/br/repos/cassandra/lib/snowball-
> stemmer-1.3.0.581.1.jar:/home/br/repos/cassandra/lib/stream-
> 2.5.2.jar:/home/br/repos/cassandra/lib/thrift-server-0.
> 

Re: dtests jolokia fails to attach

[Benjamin Roth]

Maybe additional information, this is the CS command line for ccm node1:

br   20376  3.2  8.6 2331136 708308 pts/5  Sl   06:10   0:30 java
-Xloggc:/home/br/.ccm/test/node1/logs/gc.log -ea -XX:+UseThreadPriorities
-XX:ThreadPriorityPolicy=42 -XX:+HeapDumpOnOutOfMemoryError -Xss256k
-XX:StringTableSize=103 -XX:+AlwaysPreTouch -XX:-UseBiasedLocking
-XX:+UseTLAB -XX:+ResizeTLAB -XX:+UseNUMA -XX:+PerfDisableSharedMem
-Djava.net.preferIPv4Stack=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly -XX:CMSWaitDuration=1
-XX:+CMSParallelInitialMarkEnabled -XX:+CMSEdenChunksRecordAlways
-XX:+CMSClassUnloadingEnabled -XX:+PrintGCDetails -XX:+PrintGCDateStamps
-XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution
-XX:+PrintGCApplicationStoppedTime -XX:+PrintPromotionFailure
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M
-Xms500M -Xmx500M -Xmn50M -XX:+UseCondCardMark
-XX:CompileCommandFile=/home/br/.ccm/test/node1/conf/hotspot_compiler
-javaagent:/home/br/repos/cassandra/lib/jamm-0.3.0.jar
-Dcassandra.jmx.local.port=7100
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password
-Djava.library.path=/home/br/repos/cassandra/lib/sigar-bin
-Dcassandra.migration_task_wait_in_seconds=6
-Dcassandra.libjemalloc=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1
-Dlogback.configurationFile=logback.xml
-Dcassandra.logdir=/var/log/cassandra
-Dcassandra.storagedir=/home/br/repos/cassandra/data
-Dcassandra-pidfile=/home/br/.ccm/test/node1/cassandra.pid -cp
/home/br/.ccm/test/node1/conf:/home/br/repos/cassandra/build/classes/main:/home/br/repos/cassandra/build/classes/thrift:/home/br/repos/cassandra/lib/HdrHistogram-2.1.9.jar:/home/br/repos/cassandra/lib/ST4-4.0.8.jar:/home/br/repos/cassandra/lib/airline-0.6.jar:/home/br/repos/cassandra/lib/antlr-runtime-3.5.2.jar:/home/br/repos/cassandra/lib/asm-5.0.4.jar:/home/br/repos/cassandra/lib/caffeine-2.2.6.jar:/home/br/repos/cassandra/lib/cassandra-driver-core-3.0.1-shaded.jar:/home/br/repos/cassandra/lib/commons-cli-1.1.jar:/home/br/repos/cassandra/lib/commons-codec-1.2.jar:/home/br/repos/cassandra/lib/commons-lang3-3.1.jar:/home/br/repos/cassandra/lib/commons-math3-3.2.jar:/home/br/repos/cassandra/lib/compress-lzf-0.8.4.jar:/home/br/repos/cassandra/lib/concurrent-trees-2.4.0.jar:/home/br/repos/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/home/br/repos/cassandra/lib/disruptor-3.0.1.jar:/home/br/repos/cassandra/lib/ecj-4.4.2.jar:/home/br/repos/cassandra/lib/guava-18.0.jar:/home/br/repos/cassandra/lib/high-scale-lib-1.0.6.jar:/home/br/repos/cassandra/lib/hppc-0.5.4.jar:/home/br/repos/cassandra/lib/jackson-core-asl-1.9.2.jar:/home/br/repos/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/home/br/repos/cassandra/lib/jamm-0.3.0.jar:/home/br/repos/cassandra/lib/javax.inject.jar:/home/br/repos/cassandra/lib/jbcrypt-0.3m.jar:/home/br/repos/cassandra/lib/jcl-over-slf4j-1.7.7.jar:/home/br/repos/cassandra/lib/jctools-core-1.2.1.jar:/home/br/repos/cassandra/lib/jflex-1.6.0.jar:/home/br/repos/cassandra/lib/jna-4.0.0.jar:/home/br/repos/cassandra/lib/joda-time-2.4.jar:/home/br/repos/cassandra/lib/json-simple-1.1.jar:/home/br/repos/cassandra/lib/libthrift-0.9.2.jar:/home/br/repos/cassandra/lib/log4j-over-slf4j-1.7.7.jar:/home/br/repos/cassandra/lib/logback-classic-1.1.3.jar:/home/br/repos/cassandra/lib/logback-core-1.1.3.jar:/home/br/repos/cassandra/lib/lz4-1.3.0.jar:/home/br/repos/cassandra/lib/metrics-core-3.1.0.jar:/home/br/repos/cassandra/lib/metrics-jvm-3.1.0.jar:/home/br/repos/cassandra/lib/metrics-logback-3.1.0.jar:/home/br/repos/cassandra/lib/netty-all-4.0.39.Final.jar:/home/br/repos/cassandra/lib/ohc-core-0.4.4.jar:/home/br/repos/cassandra/lib/ohc-core-j8-0.4.4.jar:/home/br/repos/cassandra/lib/primitive-1.0.jar:/home/br/repos/cassandra/lib/reporter-config-base-3.0.0.jar:/home/br/repos/cassandra/lib/reporter-config3-3.0.0.jar:/home/br/repos/cassandra/lib/sigar-1.6.4.jar:/home/br/repos/cassandra/lib/slf4j-api-1.7.7.jar:/home/br/repos/cassandra/lib/snakeyaml-1.11.jar:/home/br/repos/cassandra/lib/snappy-java-1.1.1.7.jar:/home/br/repos/cassandra/lib/snowball-stemmer-1.3.0.581.1.jar:/home/br/repos/cassandra/lib/stream-2.5.2.jar:/home/br/repos/cassandra/lib/thrift-server-0.3.7.jar:/home/br/repos/cassandra/lib/jsr223/*/*.jar
-Dcassandra.join_ring=True -Dcassandra.logdir=/home/br/.ccm/test/node1/logs
-Dcassandra.boot_without_jna=true
org.apache.cassandra.service.CassandraDaemon

Java version:
br@dev1:~/repos/cassandra-dtest$ java -version
java version "1.8.0_102"
Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)

dtests jolokia fails to attach

[Benjamin Roth]

Since some days I have the problem that I cannot run a dtest as jolokia
fails to attach to the process.
It worked some days ago. I tried both on MacOS + Linux with dtest master
and pip'ed requirements.

dtest output is:
Failed to start jolokia agent (command was:
/usr/lib/jvm/oracle-java8-jdk-amd64/bin/java -cp
/usr/lib/jvm/oracle-java8-jdk-amd64/lib/tools.jar:lib/jolokia-jvm-1.2.3-agent.jar
org.jolokia.jvmagent.client.AgentLauncher --host 127.0.0.1 start 19810):
Command '('/usr/lib/jvm/oracle-java8-jdk-amd64/bin/java', '-cp',
'/usr/lib/jvm/oracle-java8-jdk-amd64/lib/tools.jar:lib/jolokia-jvm-1.2.3-agent.jar',
'org.jolokia.jvmagent.client.AgentLauncher', '--host', '127.0.0.1',
'start', '19810')' returned non-zero exit status 1
Exit status was: 1
Output was: Illegal Argument (command: start) : Cannot attach to process-ID
19810.
See --help for possible reasons.

When I try it manually by starting a ccm instance, nodes come up, JMX port
is open, PID matches, user is the same:

br@dev1:~$ lsof -i:9042
COMMAND   PID USER   FD   TYPEDEVICE SIZE/OFF NODE NAME
java20376   br  133u  IPv4 124151851  0t0  TCP localhost:9042
(LISTEN)
java20385   br  124u  IPv4 124152856  0t0  TCP 127.0.0.3:9042
(LISTEN)
java20394   br  124u  IPv4 124151896  0t0  TCP 127.0.0.2:9042
(LISTEN)
br@dev1:~/repos/cassandra-dtest$
/usr/lib/jvm/oracle-java8-jdk-amd64/bin/java -cp
/usr/lib/jvm/oracle-java8-jdk-amd64/lib/tools.jar:lib/jolokia-jvm-1.2.3-agent.jar
org.jolokia.jvmagent.client.AgentLauncher --host 127.0.0.1 start 20376
Illegal Argument (command: start) : Cannot attach to process-ID 20376.
See --help for possible reasons.
br@dev1:~/repos/cassandra-dtest$ lsof -i:7100
COMMAND   PID USER   FD   TYPEDEVICE SIZE/OFF NODE NAME
java20376   br   72u  IPv4 124151830  0t0  TCP *:font-service
(LISTEN)

Any ideas? I just want to be able to close CASSANDRA-12689

-- 
Benjamin Roth
Prokurist

Jaumo GmbH · www.jaumo.com
Wehrstraße 46 · 73035 Göppingen · Germany
Phone +49 7161 304880-6 · Fax +49 7161 304880-1
AG Ulm · HRB 731058 · Managing Director: Jens Kammerer