Weird replace_address issue in 1.2

[Carlos Alonso]

Hi guys.

Tonight one of our 3 nodes cluster has died. This is Cassandra 1.2. and RF
= 3

After bringing up a new node and starting it with -Dreplace_address of the
dead one different things are happening.

On the new node nodetool status is showing the expected ring (him along
with the other two working nodes), but his status is UN, when I would
expect it to be UJ, because is joining, right?

Also in this same node, the logs are not showing anything related to
received streams, but streams are being received, the used disk space is
growing and nodetool netstats shows progress.

In this same node, this logs messages are continuously appearing:

INFO [GossipStage:1] 2015-10-29 13:23:10,719 Gossiper.java (line 843) Node
/ is now part of the cluster
 INFO [GossipStage:1] 2015-10-29 13:23:10,721 Gossiper.java (line 809)
InetAddress / is now UP
 WARN [GossipStage:1] 2015-10-29 13:23:10,723 StorageService.java (line
1469) Not updating token metadata for / because I am
replacing it
 INFO [GossipStage:1] 2015-10-29 13:23:10,723 StorageService.java (line
1567) Nodes / and / have the same token
115915760983105627952720478187817787338.  Ignoring /
 INFO [GossipTasks:1] 2015-10-29 13:23:41,350 Gossiper.java (line 622)
FatClient / has been silent for 3ms, removing from gossip

Now switching to the old working nodes, nodetool status shows the old ring,
with the failed node as DN, but the new node doesn't appear in any of them.

Streams are flowing from one of them to the newcomer and both netstats and
logs show it.

Gossipinfo in the new node shows the two working nodes as normal and
himself as hibernating and the other nodes show the same but also the dead
node as normal too.

Is that nodetool status/gossipinfo disagreement normal and the logs of the
newcomer? Any experience with this?

Regards

Carlos Alonso | Software Engineer | @calonso 

Error Code

[Eduardo Alfaia]

Hi Guys,

Does anyone know what error code in cassandra is?

Error decoding response from Cassandra. opcode: 0008;

Thanks

Re: Oracle TIMESTAMP(9) equivalent in Cassandra

[Clint Martin]

Generating the time uuid on the server side via the now() function also
makes the operation non idempotent. This may not be a huge problem for your
application but it is something to keep in mind.

Clint
On Oct 29, 2015 9:01 AM, "Kai Wang"  wrote:

> If you want the timestamp to be generated on the C* side, you need to sync
> clocks among nodes to the nanosecond precision first. That alone might be
> hard or impossible already. I think the safe bet is to generate the
> timestamp on the client side. But depending on your data volume, if data
> comes from multiple clients you still need to sync clocks among them.
>
>
> On Thu, Oct 29, 2015 at 7:57 AM,  wrote:
>
>> Hi Doan,
>>
>>
>>
>> Is the timeBased() method available in Java driver similar to now() function
>> in cqlsh. Does both provide identical results.
>>
>>
>>
>> Also, the preference is to generate values during record insertion from
>> database side, rather than client side. Something similar to SYSTIMESTAMP
>> in Oracle.
>>
>>
>>
>> Regards, Chandra Sekar KR
>>
>> *From:* DuyHai Doan [mailto:doanduy...@gmail.com]
>> *Sent:* 29/10/2015 5:13 PM
>> *To:* user@cassandra.apache.org
>> *Subject:* Re: Oracle TIMESTAMP(9) equivalent in Cassandra
>>
>>
>>
>> You can use TimeUUID data type and provide the value yourself from client
>> side.
>>
>>
>>
>> The Java driver offers an utility class
>> com.datastax.driver.core.utils.UUIDs and the method timeBased() to generate
>> the TimeUUID.
>>
>>
>>
>>  The precision is only guaranteed up to 100 nano seconds. So you can have
>> possibly 10k distincts values for 1 millsec. For your requirement of 20k
>> per sec, it should be enough.
>>
>>
>>
>> On Thu, Oct 29, 2015 at 12:10 PM,  wrote:
>>
>> Hi,
>>
>>
>>
>> Oracle Timestamp data type supports fractional seconds (upto 9 digits, 6
>> is default). What is the Cassandra equivalent data type for Oracle
>> TimeStamp nanosecond precision.
>>
>>
>>
>> This is required for determining the order of insertion of record where
>> the number of records inserted per sec is close to 20K. Is TIMEUUID an
>> alternate functionality which can determine the order of record insertion
>> in Cassandra ?
>>
>>
>>
>> Regards, Chandra Sekar KR
>>
>> The information contained in this electronic message and any attachments
>> to this message are intended for the exclusive use of the addressee(s) and
>> may contain proprietary, confidential or privileged information. If you are
>> not the intended recipient, you should not disseminate, distribute or copy
>> this e-mail. Please notify the sender immediately and destroy all copies of
>> this message and any attachments. WARNING: Computer viruses can be
>> transmitted via email. The recipient should check this email and any
>> attachments for the presence of viruses. The company accepts no liability
>> for any damage caused by any virus transmitted by this email.
>> www.wipro.com
>>
>>
>> The information contained in this electronic message and any attachments
>> to this message are intended for the exclusive use of the addressee(s) and
>> may contain proprietary, confidential or privileged information. If you are
>> not the intended recipient, you should not disseminate, distribute or copy
>> this e-mail. Please notify the sender immediately and destroy all copies of
>> this message and any attachments. WARNING: Computer viruses can be
>> transmitted via email. The recipient should check this email and any
>> attachments for the presence of viruses. The company accepts no liability
>> for any damage caused by any virus transmitted by this email.
>> www.wipro.com
>>
>
>

Re: Error Code

[Kai Wang]

https://github.com/datastax/python-driver/blob/75ddc514617304797626cc69957eb6008695be1e/cassandra/connection.py#L573

Is your error message complete?

On Thu, Oct 29, 2015 at 9:45 AM, Eduardo Alfaia 
wrote:

> Hi Guys,
>
> Does anyone know what error code in cassandra is?
>
> Error decoding response from Cassandra. opcode: 0008;
>
> Thanks
>

Re: Error Code

[Eduardo Alfaia]

yes, but what does it mean?
> On 29 Oct 2015, at 15:18, Kai Wang  wrote:
> 
> https://github.com/datastax/python-driver/blob/75ddc514617304797626cc69957eb6008695be1e/cassandra/connection.py#L573
>  
> 
> 
> Is your error message complete?
> 
> On Thu, Oct 29, 2015 at 9:45 AM, Eduardo Alfaia  > wrote:
> Hi Guys,
> 
> Does anyone know what error code in cassandra is?
> 
> Error decoding response from Cassandra. opcode: 0008;
> 
> Thanks
> 

RE: Oracle TIMESTAMP(9) equivalent in Cassandra

[chandrasekar.krc]

Hi Doan,

Is the timeBased() method available in Java driver similar to now() function in 
cqlsh. Does both provide identical results.

Also, the preference is to generate values during record insertion from 
database side, rather than client side. Something similar to SYSTIMESTAMP in 
Oracle.

Regards, Chandra Sekar KR
From: DuyHai Doan [mailto:doanduy...@gmail.com]
Sent: 29/10/2015 5:13 PM
To: user@cassandra.apache.org
Subject: Re: Oracle TIMESTAMP(9) equivalent in Cassandra

You can use TimeUUID data type and provide the value yourself from client side.

The Java driver offers an utility class com.datastax.driver.core.utils.UUIDs 
and the method timeBased() to generate the TimeUUID.

 The precision is only guaranteed up to 100 nano seconds. So you can have 
possibly 10k distincts values for 1 millsec. For your requirement of 20k per 
sec, it should be enough.

On Thu, Oct 29, 2015 at 12:10 PM, 
> wrote:
Hi,

Oracle Timestamp data type supports fractional seconds (upto 9 digits, 6 is 
default). What is the Cassandra equivalent data type for Oracle TimeStamp 
nanosecond precision.

This is required for determining the order of insertion of record where the 
number of records inserted per sec is close to 20K. Is TIMEUUID an alternate 
functionality which can determine the order of record insertion in Cassandra ?

Regards, Chandra Sekar KR
The information contained in this electronic message and any attachments to 
this message are intended for the exclusive use of the addressee(s) and may 
contain proprietary, confidential or privileged information. If you are not the 
intended recipient, you should not disseminate, distribute or copy this e-mail. 
Please notify the sender immediately and destroy all copies of this message and 
any attachments. WARNING: Computer viruses can be transmitted via email. The 
recipient should check this email and any attachments for the presence of 
viruses. The company accepts no liability for any damage caused by any virus 
transmitted by this email. www.wipro.com

The information contained in this electronic message and any attachments to 
this message are intended for the exclusive use of the addressee(s) and may 
contain proprietary, confidential or privileged information. If you are not the 
intended recipient, you should not disseminate, distribute or copy this e-mail. 
Please notify the sender immediately and destroy all copies of this message and 
any attachments. WARNING: Computer viruses can be transmitted via email. The 
recipient should check this email and any attachments for the presence of 
viruses. The company accepts no liability for any damage caused by any virus 
transmitted by this email. www.wipro.com

Re: Oracle TIMESTAMP(9) equivalent in Cassandra

[Jon Haddad]

Keep in mind that in a distributed environment you probably have so much 
variance that nanosecond precision is pointless.  Even google notes that in the 
paper, Dapper, a Large-Scale Distributed Systems Tracing Infrastructure 
[http://research.google.com/pubs/pub36356.html 
]


> On Oct 29, 2015, at 11:42 AM, DuyHai Doan  wrote:
> 
> You can use TimeUUID data type and provide the value yourself from client 
> side.
> 
> The Java driver offers an utility class com.datastax.driver.core.utils.UUIDs 
> and the method timeBased() to generate the TimeUUID.
> 
>  The precision is only guaranteed up to 100 nano seconds. So you can have 
> possibly 10k distincts values for 1 millsec. For your requirement of 20k per 
> sec, it should be enough. 
> 
> On Thu, Oct 29, 2015 at 12:10 PM,  > wrote:
> Hi,
> 
>  
> 
> Oracle Timestamp data type supports fractional seconds (upto 9 digits, 6 is 
> default). What is the Cassandra equivalent data type for Oracle TimeStamp 
> nanosecond precision.
> 
>  
> 
> This is required for determining the order of insertion of record where the 
> number of records inserted per sec is close to 20K. Is TIMEUUID an alternate 
> functionality which can determine the order of record insertion in Cassandra ?
> 
>  
> 
> Regards, Chandra Sekar KR
> 
> The information contained in this electronic message and any attachments to 
> this message are intended for the exclusive use of the addressee(s) and may 
> contain proprietary, confidential or privileged information. If you are not 
> the intended recipient, you should not disseminate, distribute or copy this 
> e-mail. Please notify the sender immediately and destroy all copies of this 
> message and any attachments. WARNING: Computer viruses can be transmitted via 
> email. The recipient should check this email and any attachments for the 
> presence of viruses. The company accepts no liability for any damage caused 
> by any virus transmitted by this email. www.wipro.com 

Re: Oracle TIMESTAMP(9) equivalent in Cassandra

[Kai Wang]

If you want the timestamp to be generated on the C* side, you need to sync
clocks among nodes to the nanosecond precision first. That alone might be
hard or impossible already. I think the safe bet is to generate the
timestamp on the client side. But depending on your data volume, if data
comes from multiple clients you still need to sync clocks among them.


On Thu, Oct 29, 2015 at 7:57 AM,  wrote:

> Hi Doan,
>
>
>
> Is the timeBased() method available in Java driver similar to now() function
> in cqlsh. Does both provide identical results.
>
>
>
> Also, the preference is to generate values during record insertion from
> database side, rather than client side. Something similar to SYSTIMESTAMP
> in Oracle.
>
>
>
> Regards, Chandra Sekar KR
>
> *From:* DuyHai Doan [mailto:doanduy...@gmail.com]
> *Sent:* 29/10/2015 5:13 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Oracle TIMESTAMP(9) equivalent in Cassandra
>
>
>
> You can use TimeUUID data type and provide the value yourself from client
> side.
>
>
>
> The Java driver offers an utility class
> com.datastax.driver.core.utils.UUIDs and the method timeBased() to generate
> the TimeUUID.
>
>
>
>  The precision is only guaranteed up to 100 nano seconds. So you can have
> possibly 10k distincts values for 1 millsec. For your requirement of 20k
> per sec, it should be enough.
>
>
>
> On Thu, Oct 29, 2015 at 12:10 PM,  wrote:
>
> Hi,
>
>
>
> Oracle Timestamp data type supports fractional seconds (upto 9 digits, 6
> is default). What is the Cassandra equivalent data type for Oracle
> TimeStamp nanosecond precision.
>
>
>
> This is required for determining the order of insertion of record where
> the number of records inserted per sec is close to 20K. Is TIMEUUID an
> alternate functionality which can determine the order of record insertion
> in Cassandra ?
>
>
>
> Regards, Chandra Sekar KR
>
> The information contained in this electronic message and any attachments
> to this message are intended for the exclusive use of the addressee(s) and
> may contain proprietary, confidential or privileged information. If you are
> not the intended recipient, you should not disseminate, distribute or copy
> this e-mail. Please notify the sender immediately and destroy all copies of
> this message and any attachments. WARNING: Computer viruses can be
> transmitted via email. The recipient should check this email and any
> attachments for the presence of viruses. The company accepts no liability
> for any damage caused by any virus transmitted by this email.
> www.wipro.com
>
>
> The information contained in this electronic message and any attachments
> to this message are intended for the exclusive use of the addressee(s) and
> may contain proprietary, confidential or privileged information. If you are
> not the intended recipient, you should not disseminate, distribute or copy
> this e-mail. Please notify the sender immediately and destroy all copies of
> this message and any attachments. WARNING: Computer viruses can be
> transmitted via email. The recipient should check this email and any
> attachments for the presence of viruses. The company accepts no liability
> for any damage caused by any virus transmitted by this email.
> www.wipro.com
>

Re: Error Code

[Eduardo Alfaia]

I am using a python driver from DataStax. Cassandra driver 2.7.2
> On 29 Oct 2015, at 15:26, Chris Lohfink  wrote:
> 
> It means a response (opcode 8) message couldn't be decoded. What driver are 
> you using? What version? What version of C*?
> 
> Chris
> 
> On Thu, Oct 29, 2015 at 9:19 AM, Eduardo Alfaia  > wrote:
> yes, but what does it mean?
> 
>> On 29 Oct 2015, at 15:18, Kai Wang > > wrote:
>> 
>> https://github.com/datastax/python-driver/blob/75ddc514617304797626cc69957eb6008695be1e/cassandra/connection.py#L573
>>  
>> 
>> 
>> Is your error message complete?
>> 
>> On Thu, Oct 29, 2015 at 9:45 AM, Eduardo Alfaia > > wrote:
>> Hi Guys,
>> 
>> Does anyone know what error code in cassandra is?
>> 
>> Error decoding response from Cassandra. opcode: 0008;
>> 
>> Thanks
>> 
> 
> 

Re: memtable flush size with LCS

[Ken Hancock]

Or if you're doing a high volume of writes, then your flushed file size may
be completely determined by other CFs that have consumed the commitlog
size, forcing any memtables whose commitlog is being delete to be forced to
disk.

On Wed, Oct 28, 2015 at 2:51 PM, Jeff Jirsa 
wrote:

> It’s worth mentioning that initial flushed file size is typically
> determined by memtable_cleanup_threshold and the memtable space options
> (memtable_heap_space_in_mb, memtable_offheap_space_in_mb, depending on
> memtable_allocation_type)
>
>
>
> From: Nate McCall
> Reply-To: "user@cassandra.apache.org"
> Date: Wednesday, October 28, 2015 at 11:45 AM
> To: Cassandra Users
> Subject: Re: memtable flush size with LCS
>
>
>  do you mean that this property is ignored at memtable flush time, and so
>> memtables are already allowed to be much larger than sstable_size_in_mb?
>>
>
> Yes, 'sstable_size_in_mb' plays no part in the flush process. Flushing is
> based on solely on runtime activity and the file size is determined by
> whatever was in the memtable at that time.
>
>
>
> --
> -
> Nate McCall
> Austin, TX
> @zznate
>
> Co-Founder & Sr. Technical Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>

Re: Error Code

[Eduardo Alfaia]

Hi Tyler, in fact I am having a problem like this:

https://groups.google.com/a/lists.datastax.com/forum/#!searchin/python-driver-user/Unable$20to$20connect$20to$20any$20servers/python-driver-user/F_pjeclCJdY/b9E5jCreNacJ

The error code was shown me cause this error.

> On 29 Oct 2015, at 16:27, Tyler Hobbs  wrote:
> 
> That means the driver could not decode a Result message from Cassandra.  Can 
> you post the query that's failing along with your schema for that table to 
> the Python driver mailing list?  Here's a link: 
> https://groups.google.com/a/lists.datastax.com/forum/#!forum/python-driver-user
>  
> 
> 
> On Thu, Oct 29, 2015 at 9:43 AM, Eduardo Alfaia  > wrote:
> I am using a python driver from DataStax. Cassandra driver 2.7.2
> 
>> On 29 Oct 2015, at 15:26, Chris Lohfink > > wrote:
>> 
>> It means a response (opcode 8) message couldn't be decoded. What driver are 
>> you using? What version? What version of C*?
>> 
>> Chris
>> 
>> On Thu, Oct 29, 2015 at 9:19 AM, Eduardo Alfaia > > wrote:
>> yes, but what does it mean?
>> 
>>> On 29 Oct 2015, at 15:18, Kai Wang >> > wrote:
>>> 
>>> https://github.com/datastax/python-driver/blob/75ddc514617304797626cc69957eb6008695be1e/cassandra/connection.py#L573
>>>  
>>> 
>>> 
>>> Is your error message complete?
>>> 
>>> On Thu, Oct 29, 2015 at 9:45 AM, Eduardo Alfaia >> > wrote:
>>> Hi Guys,
>>> 
>>> Does anyone know what error code in cassandra is?
>>> 
>>> Error decoding response from Cassandra. opcode: 0008;
>>> 
>>> Thanks
>>> 
>> 
>> 
> 
> 
> 
> 
> -- 
> Tyler Hobbs
> DataStax 

Re: Error Connecting to Cassandra

[Eduardo Alfaia]

Hi Adam

I changed the timeout to 10 sec, but It happened the same error.

> On 29 Oct 2015, at 16:13, Adam Holmberg  wrote:
> 
> It's timing out at the default of two seconds while trying to query and build 
> the schema metadata. You can raise this timeout:
> http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.control_connection_timeout
>  
> 
> 
> Adam
> 
> On Thu, Oct 29, 2015 at 10:06 AM, Eduardo Alfaia  > wrote:
> I have gotten the log from the application.
> 
> 
> 
>> On 29 Oct 2015, at 06:11, Jeff Jirsa > > wrote:
>> 
>> The cassandra system.log would be more useful
>> 
>> When Cassandra starts rejecting or dropping tcp connections, try to connect 
>> using cqlsh, and check the logs for indication that it’s failing. 
>> 
>> From: Eduardo Alfaia
>> Reply-To: "user@cassandra.apache.org "
>> Date: Wednesday, October 28, 2015 at 5:09 PM
>> To: "user@cassandra.apache.org "
>> Subject: Re: Error Connecting to Cassandra
>> 
>> I am using cassandra standalone, the machine is up, I was monitoring the 
>> cassandra seeing the system.log but I didn’t see anything wrong.
>> I’ve captured the flow of packets using wireshark and I’ve seen that the 
>> cassandra server is reset the connection with the client. I am sending the 
>> python app which I am using.
>> 
> 
> 
> 

Re: Error Connecting to Cassandra

[Eduardo Alfaia]

I have gotten the log from the application.{\rtf1\ansi\ansicpg1252\cocoartf1404\cocoasubrtf130
{\fonttbl\f0\fswiss\fcharset0 Helvetica;}
{\colortbl;\red255\green255\blue255;}
\paperw11900\paperh16840\margl1440\margr1440\vieww10800\viewh8400\viewkind0
\pard\tx566\tx1133\tx1700\tx2267\tx2834\tx3401\tx3968\tx4535\tx5102\tx5669\tx6236\tx6803\pardirnatural\partightenfactor0

\f0\fs24 \cf0 Running 1 time\
\pard\tx566\tx1133\tx1700\tx2267\tx2834\tx3401\tx3968\tx4535\tx5102\tx5669\tx6236\tx6803\pardirnatural\partightenfactor0
\cf0 \
workspace_patrizio/python-driver$ python example_core.py\
2015-10-29 16:02:56,899 [DEBUG] cassandra.cluster: Connecting to cluster, contact points: ['10.20.255.191']; protocol version: 4\
2015-10-29 16:02:56,900 [DEBUG] cassandra.pool: Host 10.20.255.191 is now marked up\
2015-10-29 16:02:56,901 [DEBUG] cassandra.cluster: [control connection] Opening new connection to 10.20.255.191\
2015-10-29 16:02:56,903 [DEBUG] cassandra.connection: Not sending options message for new connection(43106448) to 10.20.255.191 because compression is disabled and a cql version was not specified\
2015-10-29 16:02:56,904 [DEBUG] cassandra.connection: Sending StartupMessage on \
2015-10-29 16:02:56,904 [DEBUG] cassandra.connection: Sent StartupMessage on \
2015-10-29 16:02:56,905 [DEBUG] cassandra.io.libevreactor: Starting libev event loop\
2015-10-29 16:02:56,958 [ERROR] cassandra.connection: Closing connection  due to protocol error: code=000a [Protocol error] message="Invalid or unsupported protocol version: 4"\
2015-10-29 16:02:56,959 [DEBUG] cassandra.connection: Defuncting connection (43106448) to 10.20.255.191:\
None\
2015-10-29 16:02:56,959 [DEBUG] cassandra.io.libevreactor: Closing connection (43106448) to 10.20.255.191\
2015-10-29 16:02:56,960 [DEBUG] cassandra.io.libevreactor: Closed socket to 10.20.255.191\
2015-10-29 16:02:56,961 [DEBUG] cassandra.io.libevreactor: All Connections currently closed, event loop ended\
2015-10-29 16:02:56,970 [WARNING] cassandra.cluster: Downgrading core protocol version from 4 to 3 for 10.20.255.191\
2015-10-29 16:02:56,972 [DEBUG] cassandra.connection: Not sending options message for new connection(43107216) to 10.20.255.191 because compression is disabled and a cql version was not specified\
2015-10-29 16:02:56,973 [DEBUG] cassandra.connection: Sending StartupMessage on \
2015-10-29 16:02:56,974 [DEBUG] cassandra.connection: Sent StartupMessage on \
2015-10-29 16:02:56,974 [DEBUG] cassandra.io.libevreactor: Starting libev event loop\
2015-10-29 16:02:56,991 [DEBUG] cassandra.connection: Got ReadyMessage on new connection (43107216) from 10.20.255.191\
2015-10-29 16:02:56,992 [DEBUG] cassandra.cluster: [control connection] Established new connection , registering watchers and refreshing schema and topology\
2015-10-29 16:02:57,034 [DEBUG] cassandra.cluster: [control connection] Refreshing node list and token map using preloaded results\
2015-10-29 16:02:57,035 [INFO] cassandra.policies: Using datacenter 'datacenter1' for DCAwareRoundRobinPolicy (via host '10.20.255.191'); if incorrect, please specify a local_dc to the constructor, or limit contact points to local cluster nodes\
2015-10-29 16:02:57,036 [DEBUG] cassandra.cluster: [control connection] Finished fetching ring info\
2015-10-29 16:02:57,036 [DEBUG] cassandra.cluster: [control connection] Rebuilding token map due to topology changes\
2015-10-29 16:02:57,117 [DEBUG] cassandra.cluster: [control connection] user functions table not found\
2015-10-29 16:02:57,118 [DEBUG] cassandra.cluster: [control connection] user aggregates table not found\
2015-10-29 16:02:57,118 [DEBUG] cassandra.cluster: [control connection] Fetched schema, rebuilding metadata\
2015-10-29 16:02:57,156 [DEBUG] cassandra.cluster: Control connection created\
2015-10-29 16:02:57,159 [DEBUG] cassandra.pool: Initializing connection for host 10.20.255.191\
2015-10-29 16:02:57,162 [DEBUG] cassandra.connection: Not sending options message for new connection(43534032) to 10.20.255.191 because compression is disabled and a cql version was not specified\
2015-10-29 16:02:57,163 [DEBUG] cassandra.connection: Sending StartupMessage on \
2015-10-29 16:02:57,163 [DEBUG] cassandra.connection: Sent StartupMessage on \
2015-10-29 16:02:57,174 [DEBUG] cassandra.connection: Got ReadyMessage on new connection (43534032) from 10.20.255.191\
2015-10-29 16:02:57,180 [DEBUG] cassandra.pool: Finished initializing connection for host 10.20.255.191\
2015-10-29 16:02:57,181 [DEBUG] cassandra.cluster: Added pool for host 10.20.255.191 to session\
2015-10-29 16:02:57,181 [INFO] root: EDUARDOCOSTAALFAIA\

Re: Error Code

[Tyler Hobbs]

That means the driver could not decode a Result message from Cassandra.
Can you post the query that's failing along with your schema for that table
to the Python driver mailing list?  Here's a link:
https://groups.google.com/a/lists.datastax.com/forum/#!forum/python-driver-user

On Thu, Oct 29, 2015 at 9:43 AM, Eduardo Alfaia 
wrote:

> I am using a python driver from DataStax. Cassandra driver 2.7.2
>
> On 29 Oct 2015, at 15:26, Chris Lohfink  wrote:
>
> It means a response (opcode 8) message couldn't be decoded. What driver
> are you using? What version? What version of C*?
>
> Chris
>
> On Thu, Oct 29, 2015 at 9:19 AM, Eduardo Alfaia 
> wrote:
>
>> yes, but what does it mean?
>>
>> On 29 Oct 2015, at 15:18, Kai Wang  wrote:
>>
>>
>> https://github.com/datastax/python-driver/blob/75ddc514617304797626cc69957eb6008695be1e/cassandra/connection.py#L573
>>
>> Is your error message complete?
>>
>> On Thu, Oct 29, 2015 at 9:45 AM, Eduardo Alfaia > > wrote:
>>
>>> Hi Guys,
>>>
>>> Does anyone know what error code in cassandra is?
>>>
>>> Error decoding response from Cassandra. opcode: 0008;
>>>
>>> Thanks
>>>
>>
>>
>>
>
>


-- 
Tyler Hobbs
DataStax 

Re: Error Code

[Chris Lohfink]

It means a response (opcode 8) message couldn't be decoded. What driver are
you using? What version? What version of C*?

Chris

On Thu, Oct 29, 2015 at 9:19 AM, Eduardo Alfaia 
wrote:

> yes, but what does it mean?
>
> On 29 Oct 2015, at 15:18, Kai Wang  wrote:
>
>
> https://github.com/datastax/python-driver/blob/75ddc514617304797626cc69957eb6008695be1e/cassandra/connection.py#L573
>
> Is your error message complete?
>
> On Thu, Oct 29, 2015 at 9:45 AM, Eduardo Alfaia 
> wrote:
>
>> Hi Guys,
>>
>> Does anyone know what error code in cassandra is?
>>
>> Error decoding response from Cassandra. opcode: 0008;
>>
>> Thanks
>>
>
>
>

Re: Re : Data restore to a new cluster

[Luigi Tagliamonte]

+1 also interested about an official documentation about this.


On Mon, Oct 26, 2015 at 5:12 PM, sai krishnam raju potturi <
pskraj...@gmail.com> wrote:

> hi;
>we are working on a data backup and restore procedure to a new cluster.
> We are following the datastax documentation. It mentions a step
>
> "Restore the SSTable files snapshotted from the old cluster onto the new
> cluster using the same directories"
>
>
> http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_snapshot_restore_new_cluster.html
>
> Could not find a mention about  "SCHEMA" creation. Could somebody shed
> some light on this. At what point do we create the "SCHEMA", of required.
>
>
> thanks
> Sai
>



-- 
Luigi
---
“The only way to get smarter is by playing a smarter opponent.”

Re: Error Connecting to Cassandra

[Adam Holmberg]

It's timing out at the default of two seconds while trying to query and
build the schema metadata. You can raise this timeout:
http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.control_connection_timeout

Adam

On Thu, Oct 29, 2015 at 10:06 AM, Eduardo Alfaia 
wrote:

> I have gotten the log from the application.
>
>
>
> On 29 Oct 2015, at 06:11, Jeff Jirsa  wrote:
>
> The cassandra system.log would be more useful
>
> When Cassandra starts rejecting or dropping tcp connections, try to
> connect using cqlsh, and check the logs for indication that it’s failing.
>
> From: Eduardo Alfaia
> Reply-To: "user@cassandra.apache.org"
> Date: Wednesday, October 28, 2015 at 5:09 PM
> To: "user@cassandra.apache.org"
> Subject: Re: Error Connecting to Cassandra
>
> I am using cassandra standalone, the machine is up, I was monitoring the
> cassandra seeing the system.log but I didn’t see anything wrong.
> I’ve captured the flow of packets using wireshark and I’ve seen that the
> cassandra server is reset the connection with the client. I am sending the
> python app which I am using.
>
>
>
>

Re: Oracle TIMESTAMP(9) equivalent in Cassandra

[Jonathan Haddad]

My point is about the difficulty in having perfect clocks in a distributed
system. If nanosecond precision isn't happening at Google scale, it's
unlikely to be happening anywhere. The fact that dapper was written in the
context of tracing is irrelevant.
On Thu, Oct 29, 2015 at 7:27 PM Brice Dutheil 
wrote:

> Additionally if the time uuid is generated client side, make sure the
> boxes that will perform the write hava correct ntp/ptp configuration.
>
> @John Haddad
>
> Keep in mind that in a distributed environment you probably have so much
> variance that nanosecond precision is pointless. Even google notes that in
> the paper, Dapper, a Large-Scale Distributed Systems Tracing Infrastructure
> [http://research.google.com/pubs/pub36356.html]
>
> I agree with your statement about variance. Though I just like to mention
> Dapper is about *tracing* query/code, more generally it’s about about the
> execution overhead of tracing, which is a bit different that just
> *timestamping*.
> ​
>
> -- Brice
>
> On Thu, Oct 29, 2015 at 2:45 PM, Clint Martin <
> clintlmar...@coolfiretechnologies.com> wrote:
>
>> Generating the time uuid on the server side via the now() function also
>> makes the operation non idempotent. This may not be a huge problem for your
>> application but it is something to keep in mind.
>>
>> Clint
>> On Oct 29, 2015 9:01 AM, "Kai Wang"  wrote:
>>
>>> If you want the timestamp to be generated on the C* side, you need to
>>> sync clocks among nodes to the nanosecond precision first. That alone might
>>> be hard or impossible already. I think the safe bet is to generate the
>>> timestamp on the client side. But depending on your data volume, if data
>>> comes from multiple clients you still need to sync clocks among them.
>>>
>>>
>>> On Thu, Oct 29, 2015 at 7:57 AM,  wrote:
>>>
 Hi Doan,



 Is the timeBased() method available in Java driver similar to now() 
 function
 in cqlsh. Does both provide identical results.



 Also, the preference is to generate values during record insertion from
 database side, rather than client side. Something similar to SYSTIMESTAMP
 in Oracle.



 Regards, Chandra Sekar KR

 *From:* DuyHai Doan [mailto:doanduy...@gmail.com]
 *Sent:* 29/10/2015 5:13 PM
 *To:* user@cassandra.apache.org
 *Subject:* Re: Oracle TIMESTAMP(9) equivalent in Cassandra



 You can use TimeUUID data type and provide the value yourself from
 client side.



 The Java driver offers an utility class
 com.datastax.driver.core.utils.UUIDs and the method timeBased() to generate
 the TimeUUID.



  The precision is only guaranteed up to 100 nano seconds. So you can
 have possibly 10k distincts values for 1 millsec. For your requirement of
 20k per sec, it should be enough.



 On Thu, Oct 29, 2015 at 12:10 PM,  wrote:

 Hi,



 Oracle Timestamp data type supports fractional seconds (upto 9 digits,
 6 is default). What is the Cassandra equivalent data type for Oracle
 TimeStamp nanosecond precision.



 This is required for determining the order of insertion of record where
 the number of records inserted per sec is close to 20K. Is TIMEUUID an
 alternate functionality which can determine the order of record insertion
 in Cassandra ?



 Regards, Chandra Sekar KR

 The information contained in this electronic message and any
 attachments to this message are intended for the exclusive use of the
 addressee(s) and may contain proprietary, confidential or privileged
 information. If you are not the intended recipient, you should not
 disseminate, distribute or copy this e-mail. Please notify the sender
 immediately and destroy all copies of this message and any attachments.
 WARNING: Computer viruses can be transmitted via email. The recipient
 should check this email and any attachments for the presence of viruses.
 The company accepts no liability for any damage caused by any virus
 transmitted by this email. www.wipro.com


 The information contained in this electronic message and any
 attachments to this message are intended for the exclusive use of the
 addressee(s) and may contain proprietary, confidential or privileged
 information. If you are not the intended recipient, you should not
 disseminate, distribute or copy this e-mail. Please notify the sender
 immediately and destroy all copies of this message and any attachments.
 WARNING: Computer viruses can be transmitted via email. The recipient
 should check this email and any attachments for the presence of viruses.
 The company accepts no liability for any damage caused by any virus
 transmitted by this email. 

Re: Oracle TIMESTAMP(9) equivalent in Cassandra

[Brice Dutheil]

Additionally if the time uuid is generated client side, make sure the boxes
that will perform the write hava correct ntp/ptp configuration.

@John Haddad

Keep in mind that in a distributed environment you probably have so much
variance that nanosecond precision is pointless. Even google notes that in
the paper, Dapper, a Large-Scale Distributed Systems Tracing Infrastructure
[http://research.google.com/pubs/pub36356.html]

I agree with your statement about variance. Though I just like to mention
Dapper is about *tracing* query/code, more generally it’s about about the
execution overhead of tracing, which is a bit different that just
*timestamping*.
​

-- Brice

On Thu, Oct 29, 2015 at 2:45 PM, Clint Martin <
clintlmar...@coolfiretechnologies.com> wrote:

> Generating the time uuid on the server side via the now() function also
> makes the operation non idempotent. This may not be a huge problem for your
> application but it is something to keep in mind.
>
> Clint
> On Oct 29, 2015 9:01 AM, "Kai Wang"  wrote:
>
>> If you want the timestamp to be generated on the C* side, you need to
>> sync clocks among nodes to the nanosecond precision first. That alone might
>> be hard or impossible already. I think the safe bet is to generate the
>> timestamp on the client side. But depending on your data volume, if data
>> comes from multiple clients you still need to sync clocks among them.
>>
>>
>> On Thu, Oct 29, 2015 at 7:57 AM,  wrote:
>>
>>> Hi Doan,
>>>
>>>
>>>
>>> Is the timeBased() method available in Java driver similar to now() function
>>> in cqlsh. Does both provide identical results.
>>>
>>>
>>>
>>> Also, the preference is to generate values during record insertion from
>>> database side, rather than client side. Something similar to SYSTIMESTAMP
>>> in Oracle.
>>>
>>>
>>>
>>> Regards, Chandra Sekar KR
>>>
>>> *From:* DuyHai Doan [mailto:doanduy...@gmail.com]
>>> *Sent:* 29/10/2015 5:13 PM
>>> *To:* user@cassandra.apache.org
>>> *Subject:* Re: Oracle TIMESTAMP(9) equivalent in Cassandra
>>>
>>>
>>>
>>> You can use TimeUUID data type and provide the value yourself from
>>> client side.
>>>
>>>
>>>
>>> The Java driver offers an utility class
>>> com.datastax.driver.core.utils.UUIDs and the method timeBased() to generate
>>> the TimeUUID.
>>>
>>>
>>>
>>>  The precision is only guaranteed up to 100 nano seconds. So you can
>>> have possibly 10k distincts values for 1 millsec. For your requirement of
>>> 20k per sec, it should be enough.
>>>
>>>
>>>
>>> On Thu, Oct 29, 2015 at 12:10 PM,  wrote:
>>>
>>> Hi,
>>>
>>>
>>>
>>> Oracle Timestamp data type supports fractional seconds (upto 9 digits, 6
>>> is default). What is the Cassandra equivalent data type for Oracle
>>> TimeStamp nanosecond precision.
>>>
>>>
>>>
>>> This is required for determining the order of insertion of record where
>>> the number of records inserted per sec is close to 20K. Is TIMEUUID an
>>> alternate functionality which can determine the order of record insertion
>>> in Cassandra ?
>>>
>>>
>>>
>>> Regards, Chandra Sekar KR
>>>
>>> The information contained in this electronic message and any attachments
>>> to this message are intended for the exclusive use of the addressee(s) and
>>> may contain proprietary, confidential or privileged information. If you are
>>> not the intended recipient, you should not disseminate, distribute or copy
>>> this e-mail. Please notify the sender immediately and destroy all copies of
>>> this message and any attachments. WARNING: Computer viruses can be
>>> transmitted via email. The recipient should check this email and any
>>> attachments for the presence of viruses. The company accepts no liability
>>> for any damage caused by any virus transmitted by this email.
>>> www.wipro.com
>>>
>>>
>>> The information contained in this electronic message and any attachments
>>> to this message are intended for the exclusive use of the addressee(s) and
>>> may contain proprietary, confidential or privileged information. If you are
>>> not the intended recipient, you should not disseminate, distribute or copy
>>> this e-mail. Please notify the sender immediately and destroy all copies of
>>> this message and any attachments. WARNING: Computer viruses can be
>>> transmitted via email. The recipient should check this email and any
>>> attachments for the presence of viruses. The company accepts no liability
>>> for any damage caused by any virus transmitted by this email.
>>> www.wipro.com
>>>
>>
>>

Re: Re : Data restore to a new cluster

[Robert Coli]

On Mon, Oct 26, 2015 at 9:12 AM, sai krishnam raju potturi <
pskraj...@gmail.com> wrote:

> hi;
>we are working on a data backup and restore procedure to a new cluster.
> We are following the datastax documentation. It mentions a step
>
> "Restore the SSTable files snapshotted from the old cluster onto the new
> cluster using the same directories"
>
>
> http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_snapshot_restore_new_cluster.html
>
> Could not find a mention about  "SCHEMA" creation. Could somebody shed
> some light on this. At what point do we create the "SCHEMA", of required.
>

5.1) start the new node (as in step 6, with the correct initial_token list)
5.2) create schema
5.3) either run nodetool refresh OR restart the node again. I personally
would just restart the node.

Step 5 is also unclear :

"Restore the SSTable files snapshotted from the old cluster onto the new
cluster using the same directories."

If you restore the system keyspace alongside your data keyspaces, they will
contain a Cluster Name constraint. This Cluster Name should be different in
the target cluster and the source cluster, otherwise there is a nonzero
chance that they might talk to each other and that would be Very Bad. There
is no supported way to change Cluster Name. For reasons such as this one,
in general, IMO, one should not restore the system keyspace, but "nodetool
snapshot" with no keyspace argument does snapshot it...

The documents should also mention setting the conf files in the target
cluster to have target cluster seed nodes, not the seeds from the old
cluster.

I have cc:ed docs AT datastax for their information, and I personally thank
you for pointing out this omission.

=Rob

Re: Weird replace_address issue in 1.2

[Robert Coli]

On Thu, Oct 29, 2015 at 6:41 AM, Carlos Alonso  wrote:

> Tonight one of our 3 nodes cluster has died. This is Cassandra 1.2. and RF
> = 3
>
> After bringing up a new node and starting it with -Dreplace_address of the
> dead one different things are happening.
>
> On the new node nodetool status is showing the expected ring (him along
> with the other two working nodes), but his status is UN, when I would
> expect it to be UJ, because is joining, right?
>

I predict that this node has itself in its own seed list, and therefore
cannot bootstrap. I have no idea what is going on with the streams you
report.

What version of 1.2.x?

As an aside, you are likely to have a better latency on debugging such
issues if you join the #cassandra irc channel on freenode.

=Rob

Re: nodetool status Load not same with disk used

[Robert Coli]

On Thu, Oct 29, 2015 at 1:08 AM, qihuang.zheng  wrote:

> *We have some nodes Load too large, but some are normal.  *
>

tl;dr - Clear the snapshots on the nodes which are too large.

Longer :

Are you sure that the nodes which are too large differ in the actual *data*
size, or do they just contain snapshots?

Cassandra snapshots are hard links to SSTables, which means a number of odd
things :

1) Snapshots grow in actual disk usage over time, as they only consume
"extra" disk space when the SSTable they are a hard link to is removed from
the data directory.

2) Unless you use du --apparent-size, the order in which du sees files
determines which file is counted as using the disk, so you might see weird
results from du in the data directory if you are also involving the
snapshots.

   --apparent-size
  print  apparent sizes, rather than disk usage; although the
apparent size is usually smaller, it may be larger due to
  holes in (`sparse') files, internal fragmentation, indirect
blocks, and the like

=Rob

Re: Downtime-Limit for a node in Network-Topology-Replication-Cluster?

[Robert Coli]

On Wed, Oct 28, 2015 at 2:20 PM, Vasileios Vlachos <
vasileiosvlac...@gmail.com> wrote:

> Thanks for clarifying this Rob.
>
> -However no step other than 2) provides a *guarantee* of consistency. And
> it only provides that guarantee for data that exists when the repair starts.
>
> I read the above as "consistency is a strong term..." :) But that's
> understandable.
>

Yes, the Coli Conjecture is that if your app is suited for a distributed
database, consistency probably matters to you less than you think it does.

The foundation for this conjecture is that there are many circumstances and
modes of operation in which distributed database operators have
unexpectedly lost consistency, and almost none of them (or their customers)
noticed

=Rob

Re: Cassandra security using openssl or keytool

[Robert Coli]

On Thu, Oct 29, 2015 at 1:08 AM, Vishwajeet Singh 
wrote:

> But I want to do using OpenSSL because It's my requirement.
>
> Can somebody please guide me, How I will do Cassandra Client to node
> security using SSL and I want to use OpenSSL (Not keytool).
>

Google words like :

"
import openssl private key into keytool
"

Find results like :

http://stackoverflow.com/questions/906402/importing-an-existing-x509-certificate-and-private-key-in-java-keystore-to-use-i/8224863#8224863

?

It looks like, as Jason says, "a real pain" but should be doable if you
really have a requirement of using OpenSSL to generate the private key?

If you want to somehow not use keytool at all in the process, I think
you're out of luck.

=Rob

Cassandra stalls and dropped messages not due to GC

[Jeff Ferland]

Using DSE 4.8.1 / 2.1.11.872, Java version 1.8.0_66

We upgraded our cluster this weekend and have been having issues with dropped 
mutations since then. Intensely investigating a single node and toying with 
settings has revealed that GC stalls don’t make up enough time to explain the 
10 seconds of apparent stall that would cause a hangup.

tpstats output typically shows active threads in the low single digits and 
pending similar or 0. Before a failure, pending MutationStage will skyrocket 
into 5+ digits. System.log regularly shows the gossiper complaining, then slow 
log complaints, then logs dropped mutations.

For the entire minute of 00:55, the gc logging shows no single pause > .14 
seconds and most of them much smaller. Abbreviated GC log after switching to 
g1gc (problem also exhibited before G1GC):

2015-10-30T00:55:00.550+: 6752.857: [GC pause (G1 Evacuation Pause) (young)
2015-10-30T00:55:02.843+: 6755.150: [GC pause (GCLocker Initiated GC) 
(young)
2015-10-30T00:55:05.241+: 6757.548: [GC pause (G1 Evacuation Pause) (young)
2015-10-30T00:55:07.755+: 6760.062: [GC pause (G1 Evacuation Pause) (young)
2015-10-30T00:55:10.532+: 6762.839: [GC pause (G1 Evacuation Pause) (young)
2015-10-30T00:55:13.080+: 6765.387: [GC pause (G1 Evacuation Pause) (young)
2015-10-30T00:55:15.914+: 6768.221: [GC pause (G1 Evacuation Pause) (young)
2015-10-30T00:55:18.619+: 6770.926: [GC pause (GCLocker Initiated GC) 
(young)
2015-10-30T00:55:23.270+: 6775.578: [GC pause (GCLocker Initiated GC) 
(young)
2015-10-30T00:55:28.662+: 6780.969: [GC pause (GCLocker Initiated GC) 
(young)
2015-10-30T00:55:33.326+: 6785.633: [GC pause (G1 Evacuation Pause) (young)
2015-10-30T00:55:36.600+: 6788.907: [GC pause (G1 Evacuation Pause) (young)
2015-10-30T00:55:40.050+: 6792.357: [GC pause (G1 Evacuation Pause) (young)
2015-10-30T00:55:43.728+: 6796.035: [GC pause (G1 Evacuation Pause) (young)
2015-10-30T00:55:48.216+: 6800.523: [GC pause (G1 Evacuation Pause) (young)
2015-10-30T00:55:53.621+: 6805.928: [GC pause (G1 Evacuation Pause) (young)
2015-10-30T00:55:59.048+: 6811.355: [GC pause (GCLocker Initiated GC) 
(young)

System log snippet of the pattern I’m seeing:

WARN  [GossipTasks:1] 2015-10-30 00:55:25,129  Gossiper.java:747 - Gossip stage 
has 1 pending tasks; skipping status check (no nodes will be marked down)
INFO  [CompactionExecutor:210] 2015-10-30 00:55:26,006  CompactionTask.java:141 
- Compacting 
[SSTableReader(path='/mnt/cassandra/data/system/hints/system-hints-ka-8283-Data.db'),
 
SSTableReader(path='/mnt/cassandra/data/system/hints/system-hints-ka-8286-Data.db'),
 
SSTableReader(path='/mnt/cassandra/data/system/hints/system-hints-ka-8284-Data.db'),
 
SSTableReader(path='/mnt/cassandra/data/system/hints/system-hints-ka-8285-Data.db'),
 
SSTableReader(path='/mnt/cassandra/data/system/hints/system-hints-ka-8287-Data.db')]
WARN  [GossipTasks:1] 2015-10-30 00:55:26,230  Gossiper.java:747 - Gossip stage 
has 3 pending tasks; skipping status check (no nodes will be marked down)
WARN  [GossipTasks:1] 2015-10-30 00:55:27,330  Gossiper.java:747 - Gossip stage 
has 5 pending tasks; skipping status check (no nodes will be marked down)
WARN  [GossipTasks:1] 2015-10-30 00:55:28,431  Gossiper.java:747 - Gossip stage 
has 7 pending tasks; skipping status check (no nodes will be marked down)
WARN  [GossipTasks:1] 2015-10-30 00:55:29,531  Gossiper.java:747 - Gossip stage 
has 10 pending tasks; skipping status check (no nodes will be marked down)
INFO  [CqlSlowLog-Writer-thread-0] 2015-10-30 00:55:32,448  
CqlSlowLogWriter.java:151 - Recording statements with duration of 16042 in slow 
log
INFO  [CqlSlowLog-Writer-thread-0] 2015-10-30 00:55:32,451  
CqlSlowLogWriter.java:151 - Recording statements with duration of 16047 in slow 
log
INFO  [CqlSlowLog-Writer-thread-0] 2015-10-30 00:55:32,453  
CqlSlowLogWriter.java:151 - Recording statements with duration of 16018 in slow 
log
INFO  [CqlSlowLog-Writer-thread-0] 2015-10-30 00:55:32,454  
CqlSlowLogWriter.java:151 - Recording statements with duration of 16042 in slow 
log
INFO  [CqlSlowLog-Writer-thread-0] 2015-10-30 00:55:32,455  
CqlSlowLogWriter.java:151 - Recording statements with duration of 16024 in slow 
log
INFO  [CqlSlowLog-Writer-thread-0] 2015-10-30 00:55:32,457  
CqlSlowLogWriter.java:151 - Recording statements with duration of 16045 in slow 
log
INFO  [CqlSlowLog-Writer-thread-0] 2015-10-30 00:55:32,458  
CqlSlowLogWriter.java:151 - Recording statements with duration of 16041 in slow 
log
INFO  [CqlSlowLog-Writer-thread-0] 2015-10-30 00:55:32,459  
CqlSlowLogWriter.java:151 - Recording statements with duration of 16040 in slow 
log
INFO  [CqlSlowLog-Writer-thread-0] 2015-10-30 00:55:32,461  
CqlSlowLogWriter.java:151 - Recording statements with duration of 16037 in slow 
log
WARN  [GossipTasks:1] 2015-10-30 00:55:32,697  FailureDetector.java:257 - Not 
marking nodes down due to local pause of 8668629909 > 

Re: Oracle TIMESTAMP(9) equivalent in Cassandra

[Brice Dutheil]

My point is about the difficulty in having perfect clocks in a distributed
system. If nanosecond precision isn’t happening at Google scale, it’s
unlikely to be happening anywhere. The fact that dapper was written in the
context of tracing is irrelevant.

I agree with you : yes precise time at the nano scale is hard.
However while the context of *tracing* is indeed is irrelevant, the notion
of *measure* time ; this isn’t the same problem at all, the paper is about
measuring things that span across different software/hardware while the
problem here is the order of writes (as mentioned in the original question).

Anyway I wouldn’t even trust nanoTime to generate timestamp at the
*nanoscale*, let’s look at java.lang.System.nanotTime(), the javadoc

says this call gives the nano precision however the resolution is at least
as good the millisecond, indeed depending on OS or hardware there may be be
not the same *accuracy*, on Linux for example the code may be using using
an internal counter
,
and even if it doesn’t I’m not even sure that
Linux::clock_gettime(CLOCK_MONOTONIC,
); will even be *consistent* across different hardware thread (since
each core may run at different speed for several reasons like physical
differences, power management, etc…). This implies that java threads may
issue writes that may not be ordered at the nano second precision.

Multicore processors are already a distributed system, so yes working with
an accurate nanoseconds precision in a distributed system with network
latencies is incredibly hard if not impossible.

There’s till the possibility of using a single threaded writer (but there’s
still other issues).

— Brice

On Thu, Oct 29, 2015 at 8:02 PM, Jonathan Haddad  wrote:

My point is about the difficulty in having perfect clocks in a distributed
> system. If nanosecond precision isn't happening at Google scale, it's
> unlikely to be happening anywhere. The fact that dapper was written in the
> context of tracing is irrelevant.
> On Thu, Oct 29, 2015 at 7:27 PM Brice Dutheil 
> wrote:
>
>> Additionally if the time uuid is generated client side, make sure the
>> boxes that will perform the write hava correct ntp/ptp configuration.
>>
>> @John Haddad
>>
>> Keep in mind that in a distributed environment you probably have so much
>> variance that nanosecond precision is pointless. Even google notes that in
>> the paper, Dapper, a Large-Scale Distributed Systems Tracing Infrastructure
>> [http://research.google.com/pubs/pub36356.html]
>>
>> I agree with your statement about variance. Though I just like to mention
>> Dapper is about *tracing* query/code, more generally it’s about about
>> the execution overhead of tracing, which is a bit different that just
>> *timestamping*.
>> ​
>>
>> -- Brice
>>
>> On Thu, Oct 29, 2015 at 2:45 PM, Clint Martin <
>> clintlmar...@coolfiretechnologies.com> wrote:
>>
>>> Generating the time uuid on the server side via the now() function also
>>> makes the operation non idempotent. This may not be a huge problem for your
>>> application but it is something to keep in mind.
>>>
>>> Clint
>>> On Oct 29, 2015 9:01 AM, "Kai Wang"  wrote:
>>>
 If you want the timestamp to be generated on the C* side, you need to
 sync clocks among nodes to the nanosecond precision first. That alone might
 be hard or impossible already. I think the safe bet is to generate the
 timestamp on the client side. But depending on your data volume, if data
 comes from multiple clients you still need to sync clocks among them.


 On Thu, Oct 29, 2015 at 7:57 AM,  wrote:

> Hi Doan,
>
>
>
> Is the timeBased() method available in Java driver similar to now() 
> function
> in cqlsh. Does both provide identical results.
>
>
>
> Also, the preference is to generate values during record insertion
> from database side, rather than client side. Something similar to
> SYSTIMESTAMP in Oracle.
>
>
>
> Regards, Chandra Sekar KR
>
> *From:* DuyHai Doan [mailto:doanduy...@gmail.com]
> *Sent:* 29/10/2015 5:13 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Oracle TIMESTAMP(9) equivalent in Cassandra
>
>
>
> You can use TimeUUID data type and provide the value yourself from
> client side.
>
>
>
> The Java driver offers an utility class
> com.datastax.driver.core.utils.UUIDs and the method timeBased() to 
> generate
> the TimeUUID.
>
>
>
>  The precision is only guaranteed up to 100 nano seconds. So you can
> have possibly 10k distincts values for 1 millsec. For your requirement of
> 20k per sec, it should be enough.

Re: Cassandra security using openssl or keytool

[Robert Coli]

On Thu, Oct 29, 2015 at 4:18 PM, Jason J. W. Williams <
jasonjwwilli...@gmail.com> wrote:

> I wasted 4-5 hours of my life recently importing an OpenSSL key in a PEM
>> into a Cassandra keystore using exactly that article as a starting point
>> (the server's hostname already had a certificate and key in our ops CA, and
>> for various reasons we didn't want to revoke and reissue it.).
>>
>
I certainly don't vouch for the advisability of attempting a task you've
described as a "real pain" ... but if OP wants/needs to, it's their
funeral? :D

=Rob

Re: Cassandra security using openssl or keytool

[Jason J. W. Williams]

>
> Google words like :
>
> "
> import openssl private key into keytool
> "
>
> Find results like :
>
>
> http://stackoverflow.com/questions/906402/importing-an-existing-x509-certificate-and-private-key-in-java-keystore-to-use-i/8224863#8224863
>
>
I wasted 4-5 hours of my life recently importing an OpenSSL key in a PEM
into a Cassandra keystore using exactly that article as a starting point
(the server's hostname already had a certificate and key in our ops CA, and
for various reasons we didn't want to revoke and reissue it.).

Even when you get the key imported, keytool will then frequently refuse to
pair that key entry with the certificate when you import the
certificate...and it will instead store the certificate in a new keystore
entry. Which won't work because the alias names on the keystore entries for
the key and certificate will be different (you need one entry storing both
key and certificate).  I did _finally_ get it to work but I can't tell you
how I did it...it was a lot of manually editing PEM files, converting them
to DERs and then trying every possible combination of keytool import flags.

-J

Re: Cassandra security using openssl or keytool

[Jason J. W. Williams]

>
> I certainly don't vouch for the advisability of attempting a task you've
> described as a "real pain" ... but if OP wants/needs to, it's their
> funeral? :D
>

Agreed. I just wanted to elaborate what a "real pain" meant so OP would
know I wasn't just blowing him off.

-J

Re: Cassandra security using openssl or keytool

[Jason Williams]

Because when you use keytool it stores the generated private key in the 
keystore and tags it waiting for the certificate. Then when you import the 
issued certificate it is paired in the same record with the key. It's a real 
pain to get OpenSSL encoded private keys into a keytool keystore. Don't fight 
it, just use keytool. :)

Sent via iPhone

> On Oct 29, 2015, at 00:06, Vishwajeet Singh  wrote:
> 
> Hi,
> 
> I saw Cassandra documentation. 
> 
> http://docs.datastax.com/en/cassandra/2.1/cassandra/security/secureSSLCertificates_t.html
> 
> I found this line "SSL certificates must be generated using keytool".
> 
> Can somebody explain me why SSL certificates must be generated using keytool?
> 
> Can we use OpenSSL for generating certificates?
> I am trying using openssl but it's not working. Why?
> 
> Thanks,
> Vishwajeet

Re: Cassandra security using openssl or keytool

[Vishwajeet Singh]

But I want to do using OpenSSL because It's my requirement.

Can somebody please guide me, How I will do Cassandra Client to node
security using SSL and I want to use OpenSSL (Not keytool).

On Thu, Oct 29, 2015 at 12:40 PM, Jason Williams 
wrote:

> Because when you use keytool it stores the generated private key in the
> keystore and tags it waiting for the certificate. Then when you import the
> issued certificate it is paired in the same record with the key. It's a
> real pain to get OpenSSL encoded private keys into a keytool keystore.
> Don't fight it, just use keytool. :)
>
> Sent via iPhone
>
> On Oct 29, 2015, at 00:06, Vishwajeet Singh 
> wrote:
>
> Hi,
>
> I saw Cassandra documentation.
>
>
> http://docs.datastax.com/en/cassandra/2.1/cassandra/security/secureSSLCertificates_t.html
>
> I found this line "SSL certificates must be generated using keytool".
>
> Can somebody explain me why SSL certificates must be generated using
> keytool?
>
> Can we use OpenSSL for generating certificates?
> I am trying using openssl but it's not working. Why?
>
> Thanks,
> Vishwajeet
>
>

nodetool status Load not same with disk used

[qihuang.zheng]

We have some nodes Load too large, but some are normal. 
[qihuang.zheng@cass047221 forseti]$ /usr/install/cassandra/bin/nodetool status
-- AddressLoadTokens Owns  Host ID Rack
UN 192.168.47.221 2.66 TB  256   8.7%  87e100ed-85c4-44cb-9d9f-2d602d016038 RAC1
UN 192.168.47.204 614.58 GB 256   8.2%  91ad3d42-4207-46fe-8188-34c3f0b2dbd2 
RAC1


I check the node with df command, and find disk used only 715G.
[qihuang.zheng@cass047221 forseti]$ df -h
Filesystem   Size Used Avail Use% Mounted on
/dev/sda220G 8.6G  11G 47% /
tmpfs  16G   0  16G  0% /dev/shm
/dev/sda1190M  58M 123M 32% /boot
/dev/sda43.5T 715G 2.6T 22% /home


and this is a normal node’s disk used:
[qihuang.zheng@cass047204 ~]$ df -h
Filesystem  Size Used Avail Use% Mounted on
tmpfs 16G   0  16G  0% /dev/shm
/dev/sda1   485M  57M 403M 13% /boot
/dev/mapper/VolGroup-lv_home
   3.4T 659G 2.6T 21% /home


How does nodetool status Load come from? should't It based on sstable file size 
which also based on disk used?


Tks, qihuang.zheng

Cassandra security using openssl or keytool

[Vishwajeet Singh]

Hi,

I saw Cassandra documentation.

http://docs.datastax.com/en/cassandra/2.1/cassandra/security/secureSSLCertificates_t.html

I found this line "SSL certificates must be generated using keytool".

Can somebody explain me why SSL certificates must be generated using
keytool?

Can we use OpenSSL for generating certificates?
I am trying using openssl but it's not working. Why?

Thanks,
Vishwajeet

Oracle TIMESTAMP(9) equivalent in Cassandra

[chandrasekar.krc]

Hi,

Oracle Timestamp data type supports fractional seconds (upto 9 digits, 6 is 
default). What is the Cassandra equivalent data type for Oracle TimeStamp 
nanosecond precision.

This is required for determining the order of insertion of record where the 
number of records inserted per sec is close to 20K. Is TIMEUUID an alternate 
functionality which can determine the order of record insertion in Cassandra ?

Regards, Chandra Sekar KR
The information contained in this electronic message and any attachments to 
this message are intended for the exclusive use of the addressee(s) and may 
contain proprietary, confidential or privileged information. If you are not the 
intended recipient, you should not disseminate, distribute or copy this e-mail. 
Please notify the sender immediately and destroy all copies of this message and 
any attachments. WARNING: Computer viruses can be transmitted via email. The 
recipient should check this email and any attachments for the presence of 
viruses. The company accepts no liability for any damage caused by any virus 
transmitted by this email. www.wipro.com

Re: Cassandra stalls and dropped messages not due to GC

[Sebastian Estevez]

The thing about the CASSANDRA-9504 theory is that it was solved in 2.1.6
and Jeff's running 2.1.11.

@Jeff

How often does this happen? Can you watch ttop as soon as you notice
increased read/write latencies?

wget
> https://bintray.com/artifact/download/aragozin/generic/sjk-plus-0.3.6.jar
> java -jar sjk-plus-0.3.6.jar ttop -s localhost:7199 -n 30 -o CPU


This should at least tell you which Cassandra threads are causing high
memory allocations  and CPU consumption.

All the best,


[image: datastax_logo.png] 

Sebastián Estévez

Solutions Architect | 954 905 8615 | sebastian.este...@datastax.com

[image: linkedin.png]  [image:
facebook.png]  [image: twitter.png]
 [image: g+.png]







DataStax is the fastest, most scalable distributed database technology,
delivering Apache Cassandra to the world’s most innovative enterprises.
Datastax is built to be agile, always-on, and predictably scalable to any
size. With more than 500 customers in 45 countries, DataStax is the
database technology and transactional backbone of choice for the worlds
most innovative companies such as Netflix, Adobe, Intuit, and eBay.

On Thu, Oct 29, 2015 at 9:36 PM, Graham Sanderson  wrote:

> you didn’t say what you upgraded from, but if it is 2.0.x, then look at
> CASSANDRA-9504
>
> If so and you use
>
> commitlog_sync: batch
>
> Then you probably want to set
>
> commitlog_sync_batch_window_in_ms: 1 (or 2)
>
> Note I’m only slightly convinced this is the cause because of your
> READ_REPAIR issues (though if you are dropping a lot of MUTATIONS under
> load and your machines are overloaded, you’d be doing more READ_REPAIR than
> usual probably)
>
> On Oct 29, 2015, at 8:12 PM, Jeff Ferland  wrote:
>
> Using DSE 4.8.1 / 2.1.11.872, Java version 1.8.0_66
>
> We upgraded our cluster this weekend and have been having issues with
> dropped mutations since then. Intensely investigating a single node and
> toying with settings has revealed that GC stalls don’t make up enough time
> to explain the 10 seconds of apparent stall that would cause a hangup.
>
> tpstats output typically shows active threads in the low single digits and
> pending similar or 0. Before a failure, pending MutationStage will
> skyrocket into 5+ digits. System.log regularly shows the gossiper
> complaining, then slow log complaints, then logs dropped mutations.
>
> For the entire minute of 00:55, the gc logging shows no single pause > .14
> seconds and most of them much smaller. Abbreviated GC log after switching
> to g1gc (problem also exhibited before G1GC):
>
> 2015-10-30T00:55:00.550+: 6752.857: [GC pause (G1 Evacuation Pause)
> (young)
> 2015-10-30T00:55:02.843+: 6755.150: [GC pause (GCLocker Initiated GC)
> (young)
> 2015-10-30T00:55:05.241+: 6757.548: [GC pause (G1 Evacuation Pause)
> (young)
> 2015-10-30T00:55:07.755+: 6760.062: [GC pause (G1 Evacuation Pause)
> (young)
> 2015-10-30T00:55:10.532+: 6762.839: [GC pause (G1 Evacuation Pause)
> (young)
> 2015-10-30T00:55:13.080+: 6765.387: [GC pause (G1 Evacuation Pause)
> (young)
> 2015-10-30T00:55:15.914+: 6768.221: [GC pause (G1 Evacuation Pause)
> (young)
> 2015-10-30T00:55:18.619+: 6770.926: [GC pause (GCLocker Initiated GC)
> (young)
> 2015-10-30T00:55:23.270+: 6775.578: [GC pause (GCLocker Initiated GC)
> (young)
> 2015-10-30T00:55:28.662+: 6780.969: [GC pause (GCLocker Initiated GC)
> (young)
> 2015-10-30T00:55:33.326+: 6785.633: [GC pause (G1 Evacuation Pause)
> (young)
> 2015-10-30T00:55:36.600+: 6788.907: [GC pause (G1 Evacuation Pause)
> (young)
> 2015-10-30T00:55:40.050+: 6792.357: [GC pause (G1 Evacuation Pause)
> (young)
> 2015-10-30T00:55:43.728+: 6796.035: [GC pause (G1 Evacuation Pause)
> (young)
> 2015-10-30T00:55:48.216+: 6800.523: [GC pause (G1 Evacuation Pause)
> (young)
> 2015-10-30T00:55:53.621+: 6805.928: [GC pause (G1 Evacuation Pause)
> (young)
> 2015-10-30T00:55:59.048+: 6811.355: [GC pause (GCLocker Initiated GC)
> (young)
>
> System log snippet of the pattern I’m seeing:
>
> WARN  [GossipTasks:1] 2015-10-30 00:55:25,129  Gossiper.java:747 - Gossip
> stage has 1 pending tasks; skipping status check (no nodes will be marked
> down)
> INFO  [CompactionExecutor:210] 2015-10-30 00:55:26,006
>  CompactionTask.java:141 - Compacting
> [SSTableReader(path='/mnt/cassandra/data/system/hints/system-hints-ka-8283-Data.db'),
> SSTableReader(path='/mnt/cassandra/data/system/hints/system-hints-ka-8286-Data.db'),
> SSTableReader(path='/mnt/cassandra/data/system/hints/system-hints-ka-8284-Data.db'),
> SSTableReader(path='/mnt/cassandra/data/system/hints/system-hints-ka-8285-Data.db'),
> 

cassandra state listener sends duplicate signals one with public ip and another with private ip

[Dhyan Muralidharan]

Hi,
 Can some one respond to this ?

http://stackoverflow.com/questions/33428094/cassandra-state-listener-sends-duplicate-signals-one-with-public-ip-and-another
 


—Dhyan

Re: Cassandra -stress write - Storage location

[Sebastian Estevez]

By default this will go in Keyspace1 Standard1.

All the best,


[image: datastax_logo.png] 

Sebastián Estévez

Solutions Architect | 954 905 8615 | sebastian.este...@datastax.com

[image: linkedin.png]  [image:
facebook.png]  [image: twitter.png]
 [image: g+.png]







DataStax is the fastest, most scalable distributed database technology,
delivering Apache Cassandra to the world’s most innovative enterprises.
Datastax is built to be agile, always-on, and predictably scalable to any
size. With more than 500 customers in 45 countries, DataStax is the
database technology and transactional backbone of choice for the worlds
most innovative companies such as Netflix, Adobe, Intuit, and eBay.

On Fri, Oct 30, 2015 at 12:07 AM, Arun Sandu  wrote:

> Hi,
>
> I am currently working on load testing my cluster. When we write 10 to
> cassandra, where does the writes data gets stored in Cassandra and the same
> for read operation too.
>
> ./cassandra-stress write n=10 -rate threads=100 -node 10.34.100.13
>
> ./cassandra-stress read n=10 -node 10.34.100.13
>
>
> --
> Thanks
> Arun
>

Re: Cassandra stalls and dropped messages not due to GC

[Jeff Ferland]

Upgraded from 2.0.x. Using the other commit log sync method and 10 seconds. 
Enabling batch mode is like swallowing a grenade.

It’s starting to look to me like it’s possibly related to brief IO spikes that 
are smaller than my usual graphing granularity. It feels surprising to me that 
these would affect the Gossip threads, but it’s the best current lead I have 
with my debugging right now. More to come when I learn it.

-Jeff

> On Oct 29, 2015, at 6:36 PM, Graham Sanderson  wrote:
> 
> you didn’t say what you upgraded from, but if it is 2.0.x, then look at 
> CASSANDRA-9504
> 
> If so and you use
> commitlog_sync: batch
> Then you probably want to set
> commitlog_sync_batch_window_in_ms: 1 (or 2)
> Note I’m only slightly convinced this is the cause because of your 
> READ_REPAIR issues (though if you are dropping a lot of MUTATIONS under load 
> and your machines are overloaded, you’d be doing more READ_REPAIR than usual 
> probably)
> 
>> On Oct 29, 2015, at 8:12 PM, Jeff Ferland > > wrote:
>> 
>> Using DSE 4.8.1 / 2.1.11.872, Java version 1.8.0_66
>> 
>> We upgraded our cluster this weekend and have been having issues with 
>> dropped mutations since then. Intensely investigating a single node and 
>> toying with settings has revealed that GC stalls don’t make up enough time 
>> to explain the 10 seconds of apparent stall that would cause a hangup.
>> 
>> tpstats output typically shows active threads in the low single digits and 
>> pending similar or 0. Before a failure, pending MutationStage will skyrocket 
>> into 5+ digits. System.log regularly shows the gossiper complaining, then 
>> slow log complaints, then logs dropped mutations.
>> 
>> For the entire minute of 00:55, the gc logging shows no single pause > .14 
>> seconds and most of them much smaller. Abbreviated GC log after switching to 
>> g1gc (problem also exhibited before G1GC):
>> 
>> 2015-10-30T00:55:00.550+: 6752.857: [GC pause (G1 Evacuation Pause) 
>> (young)
>> 2015-10-30T00:55:02.843+: 6755.150: [GC pause (GCLocker Initiated GC) 
>> (young)
>> 2015-10-30T00:55:05.241+: 6757.548: [GC pause (G1 Evacuation Pause) 
>> (young)
>> 2015-10-30T00:55:07.755+: 6760.062: [GC pause (G1 Evacuation Pause) 
>> (young)
>> 2015-10-30T00:55:10.532+: 6762.839: [GC pause (G1 Evacuation Pause) 
>> (young)
>> 2015-10-30T00:55:13.080+: 6765.387: [GC pause (G1 Evacuation Pause) 
>> (young)
>> 2015-10-30T00:55:15.914+: 6768.221: [GC pause (G1 Evacuation Pause) 
>> (young)
>> 2015-10-30T00:55:18.619+: 6770.926: [GC pause (GCLocker Initiated GC) 
>> (young)
>> 2015-10-30T00:55:23.270+: 6775.578: [GC pause (GCLocker Initiated GC) 
>> (young)
>> 2015-10-30T00:55:28.662+: 6780.969: [GC pause (GCLocker Initiated GC) 
>> (young)
>> 2015-10-30T00:55:33.326+: 6785.633: [GC pause (G1 Evacuation Pause) 
>> (young)
>> 2015-10-30T00:55:36.600+: 6788.907: [GC pause (G1 Evacuation Pause) 
>> (young)
>> 2015-10-30T00:55:40.050+: 6792.357: [GC pause (G1 Evacuation Pause) 
>> (young)
>> 2015-10-30T00:55:43.728+: 6796.035: [GC pause (G1 Evacuation Pause) 
>> (young)
>> 2015-10-30T00:55:48.216+: 6800.523: [GC pause (G1 Evacuation Pause) 
>> (young)
>> 2015-10-30T00:55:53.621+: 6805.928: [GC pause (G1 Evacuation Pause) 
>> (young)
>> 2015-10-30T00:55:59.048+: 6811.355: [GC pause (GCLocker Initiated GC) 
>> (young)
>> 
>> System log snippet of the pattern I’m seeing:
>> 
>> WARN  [GossipTasks:1] 2015-10-30 00:55:25,129  Gossiper.java:747 - Gossip 
>> stage has 1 pending tasks; skipping status check (no nodes will be marked 
>> down)
>> INFO  [CompactionExecutor:210] 2015-10-30 00:55:26,006  
>> CompactionTask.java:141 - Compacting 
>> [SSTableReader(path='/mnt/cassandra/data/system/hints/system-hints-ka-8283-Data.db'),
>>  
>> SSTableReader(path='/mnt/cassandra/data/system/hints/system-hints-ka-8286-Data.db'),
>>  
>> SSTableReader(path='/mnt/cassandra/data/system/hints/system-hints-ka-8284-Data.db'),
>>  
>> SSTableReader(path='/mnt/cassandra/data/system/hints/system-hints-ka-8285-Data.db'),
>>  
>> SSTableReader(path='/mnt/cassandra/data/system/hints/system-hints-ka-8287-Data.db')]
>> WARN  [GossipTasks:1] 2015-10-30 00:55:26,230  Gossiper.java:747 - Gossip 
>> stage has 3 pending tasks; skipping status check (no nodes will be marked 
>> down)
>> WARN  [GossipTasks:1] 2015-10-30 00:55:27,330  Gossiper.java:747 - Gossip 
>> stage has 5 pending tasks; skipping status check (no nodes will be marked 
>> down)
>> WARN  [GossipTasks:1] 2015-10-30 00:55:28,431  Gossiper.java:747 - Gossip 
>> stage has 7 pending tasks; skipping status check (no nodes will be marked 
>> down)
>> WARN  [GossipTasks:1] 2015-10-30 00:55:29,531  Gossiper.java:747 - Gossip 
>> stage has 10 pending tasks; skipping status check (no nodes will be marked 
>> down)
>> INFO  [CqlSlowLog-Writer-thread-0] 2015-10-30 00:55:32,448  
>> CqlSlowLogWriter.java:151 - Recording 

Re: Cassandra stalls and dropped messages not due to GC

[Graham Sanderson]

Only if you actually change cassandra.yaml (that was the change in 2.1.6 which 
is why it matters what version he upgraded from)

> On Oct 29, 2015, at 10:06 PM, Sebastian Estevez 
>  wrote:
> 
> The thing about the CASSANDRA-9504 theory is that it was solved in 2.1.6 and 
> Jeff's running 2.1.11.
> 
> @Jeff
> 
> How often does this happen? Can you watch ttop as soon as you notice 
> increased read/write latencies?
> 
> wget https://bintray.com/artifact/download/aragozin/generic/sjk-plus-0.3.6.jar
>  
> java
>  -jar sjk-plus-0.3.6.jar ttop -s localhost:7199 -n 30 -o CPU
> 
> This should at least tell you which Cassandra threads are causing high memory 
> allocations  and CPU consumption.
> 
> All the best,
> 
>  
> Sebastián Estévez
> Solutions Architect | 954 905 8615 | sebastian.este...@datastax.com 
> 
>    
>     
>   
> 
>  
> 
>  
> 
> DataStax is the fastest, most scalable distributed database technology, 
> delivering Apache Cassandra to the world’s most innovative enterprises. 
> Datastax is built to be agile, always-on, and predictably scalable to any 
> size. With more than 500 customers in 45 countries, DataStax is the database 
> technology and transactional backbone of choice for the worlds most 
> innovative companies such as Netflix, Adobe, Intuit, and eBay. 
> 
> On Thu, Oct 29, 2015 at 9:36 PM, Graham Sanderson  > wrote:
> you didn’t say what you upgraded from, but if it is 2.0.x, then look at 
> CASSANDRA-9504
> 
> If so and you use
> commitlog_sync: batch
> Then you probably want to set
> commitlog_sync_batch_window_in_ms: 1 (or 2)
> Note I’m only slightly convinced this is the cause because of your 
> READ_REPAIR issues (though if you are dropping a lot of MUTATIONS under load 
> and your machines are overloaded, you’d be doing more READ_REPAIR than usual 
> probably)
> 
>> On Oct 29, 2015, at 8:12 PM, Jeff Ferland > > wrote:
>> 
>> Using DSE 4.8.1 / 2.1.11.872, Java version 1.8.0_66
>> 
>> We upgraded our cluster this weekend and have been having issues with 
>> dropped mutations since then. Intensely investigating a single node and 
>> toying with settings has revealed that GC stalls don’t make up enough time 
>> to explain the 10 seconds of apparent stall that would cause a hangup.
>> 
>> tpstats output typically shows active threads in the low single digits and 
>> pending similar or 0. Before a failure, pending MutationStage will skyrocket 
>> into 5+ digits. System.log regularly shows the gossiper complaining, then 
>> slow log complaints, then logs dropped mutations.
>> 
>> For the entire minute of 00:55, the gc logging shows no single pause > .14 
>> seconds and most of them much smaller. Abbreviated GC log after switching to 
>> g1gc (problem also exhibited before G1GC):
>> 
>> 2015-10-30T00:55:00.550+: 6752.857: [GC pause (G1 Evacuation Pause) 
>> (young)
>> 2015-10-30T00:55:02.843+: 6755.150: [GC pause (GCLocker Initiated GC) 
>> (young)
>> 2015-10-30T00:55:05.241+: 6757.548: [GC pause (G1 Evacuation Pause) 
>> (young)
>> 2015-10-30T00:55:07.755+: 6760.062: [GC pause (G1 Evacuation Pause) 
>> (young)
>> 2015-10-30T00:55:10.532+: 6762.839: [GC pause (G1 Evacuation Pause) 
>> (young)
>> 2015-10-30T00:55:13.080+: 6765.387: [GC pause (G1 Evacuation Pause) 
>> (young)
>> 2015-10-30T00:55:15.914+: 6768.221: [GC pause (G1 Evacuation Pause) 
>> (young)
>> 2015-10-30T00:55:18.619+: 6770.926: [GC pause (GCLocker Initiated GC) 
>> (young)
>> 2015-10-30T00:55:23.270+: 6775.578: [GC pause (GCLocker Initiated GC) 
>> (young)
>> 2015-10-30T00:55:28.662+: 6780.969: [GC pause (GCLocker Initiated GC) 
>> (young)
>> 2015-10-30T00:55:33.326+: 6785.633: [GC pause (G1 Evacuation Pause) 
>> (young)
>> 2015-10-30T00:55:36.600+: 6788.907: [GC pause (G1 Evacuation Pause) 
>> (young)
>> 2015-10-30T00:55:40.050+: 6792.357: [GC pause (G1 Evacuation Pause) 
>> (young)
>> 2015-10-30T00:55:43.728+: 6796.035: [GC pause (G1 Evacuation Pause) 
>> (young)
>> 2015-10-30T00:55:48.216+: 6800.523: [GC pause (G1 Evacuation Pause) 
>> (young)
>> 2015-10-30T00:55:53.621+: 6805.928: [GC pause (G1 Evacuation Pause) 
>> (young)
>> 2015-10-30T00:55:59.048+: 6811.355: [GC pause (GCLocker Initiated GC) 
>> (young)
>> 
>> System log snippet of the pattern I’m seeing:
>> 
>> WARN  [GossipTasks:1] 2015-10-30 00:55:25,129  Gossiper.java:747 - Gossip 
>> stage has 1 pending tasks; skipping status check (no nodes will be marked 
>> down)

Re: unsubscribe

[Gene]

You have been unsuccessfully unsubscribed from the mailing list.

We are sad to see you go.  While waiting for you to return we will be
watching this on repeat.

https://www.youtube.com/watch?v=eh7lp9umG2I

On Thu, Oct 29, 2015 at 7:55 PM, M.Tarkeshwar Rao 
wrote:

>

Re: Cassandra -stress write - Storage location

[Sebastian Estevez]

You can do a describe table to see the table layout and you can select to
see some sample rows. Stress is pretty powerful though.

I just dropped a blog post tonight on doing more targeted benchmarking /
sizing with stress and my data modeler. Take a look:

http://www.sestevez.com/data-modeler/
On Oct 30, 2015 1:01 AM, "Arun Sandu"  wrote:

> Thanks . Can I know, the format of the data that gets stored? Can you
> please suggest me some ways to perform load testing? I need a big picture
> of all the statistics.
>
> Thanks again
> Arun
>
> On Thu, Oct 29, 2015 at 9:41 PM, Sebastian Estevez <
> sebastian.este...@datastax.com> wrote:
>
>> By default this will go in Keyspace1 Standard1.
>>
>> All the best,
>>
>>
>> [image: datastax_logo.png] 
>>
>> Sebastián Estévez
>>
>> Solutions Architect | 954 905 8615 | sebastian.este...@datastax.com
>>
>> [image: linkedin.png]  [image:
>> facebook.png]  [image: twitter.png]
>>  [image: g+.png]
>> 
>> 
>> 
>>
>>
>> 
>>
>> DataStax is the fastest, most scalable distributed database technology,
>> delivering Apache Cassandra to the world’s most innovative enterprises.
>> Datastax is built to be agile, always-on, and predictably scalable to any
>> size. With more than 500 customers in 45 countries, DataStax is the
>> database technology and transactional backbone of choice for the worlds
>> most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>
>> On Fri, Oct 30, 2015 at 12:07 AM, Arun Sandu 
>> wrote:
>>
>>> Hi,
>>>
>>> I am currently working on load testing my cluster. When we write 10
>>> to cassandra, where does the writes data gets stored in Cassandra and the
>>> same for read operation too.
>>>
>>> ./cassandra-stress write n=10 -rate threads=100 -node 10.34.100.13
>>>
>>> ./cassandra-stress read n=10 -node 10.34.100.13
>>>
>>>
>>> --
>>> Thanks
>>> Arun
>>>
>>
>>
>
>
> --
> Thanks
> Arun Kumar S
> 816-699-3039
>
> *"This Moment Is Not Permanent...!!"*
>

unsubscribe

[M.Tarkeshwar Rao]

Cassandra -stress write - Storage location

[Arun Sandu]

Hi,

I am currently working on load testing my cluster. When we write 10 to
cassandra, where does the writes data gets stored in Cassandra and the same
for read operation too.

./cassandra-stress write n=10 -rate threads=100 -node 10.34.100.13

./cassandra-stress read n=10 -node 10.34.100.13


-- 
Thanks
Arun

Re: Cassandra -stress write - Storage location

[Arun Sandu]

Thanks . Can I know, the format of the data that gets stored? Can you
please suggest me some ways to perform load testing? I need a big picture
of all the statistics.

Thanks again
Arun

On Thu, Oct 29, 2015 at 9:41 PM, Sebastian Estevez <
sebastian.este...@datastax.com> wrote:

> By default this will go in Keyspace1 Standard1.
>
> All the best,
>
>
> [image: datastax_logo.png] 
>
> Sebastián Estévez
>
> Solutions Architect | 954 905 8615 | sebastian.este...@datastax.com
>
> [image: linkedin.png]  [image:
> facebook.png]  [image: twitter.png]
>  [image: g+.png]
> 
> 
> 
>
>
> 
>
> DataStax is the fastest, most scalable distributed database technology,
> delivering Apache Cassandra to the world’s most innovative enterprises.
> Datastax is built to be agile, always-on, and predictably scalable to any
> size. With more than 500 customers in 45 countries, DataStax is the
> database technology and transactional backbone of choice for the worlds
> most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>
> On Fri, Oct 30, 2015 at 12:07 AM, Arun Sandu 
> wrote:
>
>> Hi,
>>
>> I am currently working on load testing my cluster. When we write 10
>> to cassandra, where does the writes data gets stored in Cassandra and the
>> same for read operation too.
>>
>> ./cassandra-stress write n=10 -rate threads=100 -node 10.34.100.13
>>
>> ./cassandra-stress read n=10 -node 10.34.100.13
>>
>>
>> --
>> Thanks
>> Arun
>>
>
>


-- 
Thanks
Arun Kumar S
816-699-3039

*"This Moment Is Not Permanent...!!"*

Re: Cassandra stalls and dropped messages not due to GC

[Graham Sanderson]

you didn’t say what you upgraded from, but if it is 2.0.x, then look at 
CASSANDRA-9504

If so and you use
commitlog_sync: batch
Then you probably want to set
commitlog_sync_batch_window_in_ms: 1 (or 2)
Note I’m only slightly convinced this is the cause because of your READ_REPAIR 
issues (though if you are dropping a lot of MUTATIONS under load and your 
machines are overloaded, you’d be doing more READ_REPAIR than usual probably)

> On Oct 29, 2015, at 8:12 PM, Jeff Ferland  wrote:
> 
> Using DSE 4.8.1 / 2.1.11.872, Java version 1.8.0_66
> 
> We upgraded our cluster this weekend and have been having issues with dropped 
> mutations since then. Intensely investigating a single node and toying with 
> settings has revealed that GC stalls don’t make up enough time to explain the 
> 10 seconds of apparent stall that would cause a hangup.
> 
> tpstats output typically shows active threads in the low single digits and 
> pending similar or 0. Before a failure, pending MutationStage will skyrocket 
> into 5+ digits. System.log regularly shows the gossiper complaining, then 
> slow log complaints, then logs dropped mutations.
> 
> For the entire minute of 00:55, the gc logging shows no single pause > .14 
> seconds and most of them much smaller. Abbreviated GC log after switching to 
> g1gc (problem also exhibited before G1GC):
> 
> 2015-10-30T00:55:00.550+: 6752.857: [GC pause (G1 Evacuation Pause) 
> (young)
> 2015-10-30T00:55:02.843+: 6755.150: [GC pause (GCLocker Initiated GC) 
> (young)
> 2015-10-30T00:55:05.241+: 6757.548: [GC pause (G1 Evacuation Pause) 
> (young)
> 2015-10-30T00:55:07.755+: 6760.062: [GC pause (G1 Evacuation Pause) 
> (young)
> 2015-10-30T00:55:10.532+: 6762.839: [GC pause (G1 Evacuation Pause) 
> (young)
> 2015-10-30T00:55:13.080+: 6765.387: [GC pause (G1 Evacuation Pause) 
> (young)
> 2015-10-30T00:55:15.914+: 6768.221: [GC pause (G1 Evacuation Pause) 
> (young)
> 2015-10-30T00:55:18.619+: 6770.926: [GC pause (GCLocker Initiated GC) 
> (young)
> 2015-10-30T00:55:23.270+: 6775.578: [GC pause (GCLocker Initiated GC) 
> (young)
> 2015-10-30T00:55:28.662+: 6780.969: [GC pause (GCLocker Initiated GC) 
> (young)
> 2015-10-30T00:55:33.326+: 6785.633: [GC pause (G1 Evacuation Pause) 
> (young)
> 2015-10-30T00:55:36.600+: 6788.907: [GC pause (G1 Evacuation Pause) 
> (young)
> 2015-10-30T00:55:40.050+: 6792.357: [GC pause (G1 Evacuation Pause) 
> (young)
> 2015-10-30T00:55:43.728+: 6796.035: [GC pause (G1 Evacuation Pause) 
> (young)
> 2015-10-30T00:55:48.216+: 6800.523: [GC pause (G1 Evacuation Pause) 
> (young)
> 2015-10-30T00:55:53.621+: 6805.928: [GC pause (G1 Evacuation Pause) 
> (young)
> 2015-10-30T00:55:59.048+: 6811.355: [GC pause (GCLocker Initiated GC) 
> (young)
> 
> System log snippet of the pattern I’m seeing:
> 
> WARN  [GossipTasks:1] 2015-10-30 00:55:25,129  Gossiper.java:747 - Gossip 
> stage has 1 pending tasks; skipping status check (no nodes will be marked 
> down)
> INFO  [CompactionExecutor:210] 2015-10-30 00:55:26,006  
> CompactionTask.java:141 - Compacting 
> [SSTableReader(path='/mnt/cassandra/data/system/hints/system-hints-ka-8283-Data.db'),
>  
> SSTableReader(path='/mnt/cassandra/data/system/hints/system-hints-ka-8286-Data.db'),
>  
> SSTableReader(path='/mnt/cassandra/data/system/hints/system-hints-ka-8284-Data.db'),
>  
> SSTableReader(path='/mnt/cassandra/data/system/hints/system-hints-ka-8285-Data.db'),
>  
> SSTableReader(path='/mnt/cassandra/data/system/hints/system-hints-ka-8287-Data.db')]
> WARN  [GossipTasks:1] 2015-10-30 00:55:26,230  Gossiper.java:747 - Gossip 
> stage has 3 pending tasks; skipping status check (no nodes will be marked 
> down)
> WARN  [GossipTasks:1] 2015-10-30 00:55:27,330  Gossiper.java:747 - Gossip 
> stage has 5 pending tasks; skipping status check (no nodes will be marked 
> down)
> WARN  [GossipTasks:1] 2015-10-30 00:55:28,431  Gossiper.java:747 - Gossip 
> stage has 7 pending tasks; skipping status check (no nodes will be marked 
> down)
> WARN  [GossipTasks:1] 2015-10-30 00:55:29,531  Gossiper.java:747 - Gossip 
> stage has 10 pending tasks; skipping status check (no nodes will be marked 
> down)
> INFO  [CqlSlowLog-Writer-thread-0] 2015-10-30 00:55:32,448  
> CqlSlowLogWriter.java:151 - Recording statements with duration of 16042 in 
> slow log
> INFO  [CqlSlowLog-Writer-thread-0] 2015-10-30 00:55:32,451  
> CqlSlowLogWriter.java:151 - Recording statements with duration of 16047 in 
> slow log
> INFO  [CqlSlowLog-Writer-thread-0] 2015-10-30 00:55:32,453  
> CqlSlowLogWriter.java:151 - Recording statements with duration of 16018 in 
> slow log
> INFO  [CqlSlowLog-Writer-thread-0] 2015-10-30 00:55:32,454  
> CqlSlowLogWriter.java:151 - Recording statements with duration of 16042 in 
> slow log
> INFO  [CqlSlowLog-Writer-thread-0] 2015-10-30 00:55:32,455  
> CqlSlowLogWriter.java:151 - Recording statements with duration of 16024 in 
> slow