Driver consistency issue
Hi All,
i have a KeySpace in Cassandra (cassandra version 3.0.9- total 12 Servers
)With below definition:
{'DC1': '2', 'class':
'org.apache.cassandra.locator.NetworkTopologyStrategy'}
Some time i am getting below exception
com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra
timeout during write query at consistency QUORUM (3 replica were required
but only 2 acknowledged the write)
at
com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:73)
at
com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:26)
at
com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
at
com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)
at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:68)
at
com.toi.stream.data.AdImprLogDaoImpl.updateImpr(AdImprLogDaoImpl.java:158)
at
com.toi.stream.service.AdClickLogAndAdimprLogServiceImpl.updateGoalsOnImpr(AdClickLogAndAdimprLogServiceImpl.java:522)
at
com.toi.stream.service.ConversionBillingLastAttributionServiceV2Impl.attribute(ConversionBillingLastAttributionServiceV2Impl.java:456)
at
com.toi.stream.service.ConversionBillingLastAttributionServiceV2Impl.attributeAdTracker(ConversionBillingLastAttributionServiceV2Impl.java:228)
at
com.toi.stream.process.AdTrackerStreamProcessorV2.process(AdTrackerStreamProcessorV2.java:86)
at
com.toi.stream.kafka.KafkaGroupConsumer.run(KafkaGroupConsumer.java:175)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException:
Cassandra timeout during write query at consistency QUORUM (3 replica were
required but only 2 acknowledged the write)
at
com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:100)
at
com.datastax.driver.core.Responses$Error.asException(Responses.java:134)
at
com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:525)
at
com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1077)
why its waiting for acknowledged from 3rd server as replication factor is 2?
--
*Thanks & Regards,*
*Abhishek Kumar Maheshwari*
*+91- 805591 (Mobile)*
Times Internet Ltd. | A Times of India Group Company
FC - 6, Sector 16A, Film City, Noida, U.P. 201301 | INDIA
*P** Please do not print this email unless it is absolutely necessary.
Spread environmental awareness.*
Re: Driver consistency issue
On Tue, Feb 27, 2018 at 9:45 AM, Abhishek Kumar Maheshwari <
abhishek.maheshw...@timesinternet.in> wrote:
>
> i have a KeySpace in Cassandra (cassandra version 3.0.9- total 12 Servers
> )With below definition:
>
> {'DC1': '2', 'class': 'org.apache.cassandra.locator.
> NetworkTopologyStrategy'}
>
> Some time i am getting below exception
>
> [snip]
> Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException:
> Cassandra timeout during write query at consistency QUORUM (3 replica were
> required but only 2 acknowledged the write)
> at com.datastax.driver.core.exceptions.WriteTimeoutException.
> copy(WriteTimeoutException.java:100)
> at com.datastax.driver.core.Responses$Error.asException(Respons
> es.java:134)
> at com.datastax.driver.core.RequestHandler$SpeculativeExecution
> .onSet(RequestHandler.java:525)
> at com.datastax.driver.core.Connection$Dispatcher.channelRead0(
> Connection.java:1077)
>
> why its waiting for acknowledged from 3rd server as replication factor is
> 2?
>
I see two possibilities:
1) The data in this keyspace is replicated to another DC, so there is also
'DC2': '2', for example, but you didn't show it. In this case QUORUM
requires more than 2 nodes.
2) The write was targeting a table in a different keyspace than you think.
In any case QUORUM (or LOCAL_QUORUM) with RF=2 is equivalent of ALL. Not
sure why would you use it in the first place.
For consistency levels involving quorum you want to go with RF=3 in a
single DC. For multi DC you should think if you want QUORUM or EACH_QUORUM
for your writes and figure out the RFs from that.
Cheers,
--
Alex
Re: Driver consistency issue
Hi Alex,
i have only One DC (with name DC1) and have only one keyspace. So i dont
think so both of the scenario is possible. (yes in my case QUORUM is
equivalent
of ALL)
cqlsh> SELECT * FROM system_schema.keyspaces where keyspace_name='adlog' ;
keyspace_name | durable_writes | replication
---++---
adlog | True | {'DC1': '2', 'class':
'org.apache.cassandra.locator.NetworkTopologyStrategy'}
On Tue, Feb 27, 2018 at 2:27 PM, Oleksandr Shulgin <
oleksandr.shul...@zalando.de> wrote:
> On Tue, Feb 27, 2018 at 9:45 AM, Abhishek Kumar Maheshwari <
> abhishek.maheshw...@timesinternet.in> wrote:
>
>>
>> i have a KeySpace in Cassandra (cassandra version 3.0.9- total 12 Servers
>> )With below definition:
>>
>> {'DC1': '2', 'class': 'org.apache.cassandra.locator.
>> NetworkTopologyStrategy'}
>>
>> Some time i am getting below exception
>>
>> [snip]
>
>> Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException:
>> Cassandra timeout during write query at consistency QUORUM (3 replica were
>> required but only 2 acknowledged the write)
>> at com.datastax.driver.core.exceptions.WriteTimeoutException.co
>> py(WriteTimeoutException.java:100)
>> at com.datastax.driver.core.Responses$Error.asException(Respons
>> es.java:134)
>> at com.datastax.driver.core.RequestHandler$SpeculativeExecution
>> .onSet(RequestHandler.java:525)
>> at com.datastax.driver.core.Connection$Dispatcher.channelRead0(
>> Connection.java:1077)
>>
>> why its waiting for acknowledged from 3rd server as replication factor
>> is 2?
>>
>
> I see two possibilities:
>
> 1) The data in this keyspace is replicated to another DC, so there is also
> 'DC2': '2', for example, but you didn't show it. In this case QUORUM
> requires more than 2 nodes.
> 2) The write was targeting a table in a different keyspace than you think.
>
> In any case QUORUM (or LOCAL_QUORUM) with RF=2 is equivalent of ALL. Not
> sure why would you use it in the first place.
>
> For consistency levels involving quorum you want to go with RF=3 in a
> single DC. For multi DC you should think if you want QUORUM or EACH_QUORUM
> for your writes and figure out the RFs from that.
>
> Cheers,
> --
> Alex
>
>
--
*Thanks & Regards,*
*Abhishek Kumar Maheshwari*
*+91- 805591 (Mobile)*
Times Internet Ltd. | A Times of India Group Company
FC - 6, Sector 16A, Film City, Noida, U.P. 201301 | INDIA
*P** Please do not print this email unless it is absolutely necessary.
Spread environmental awareness.*
Re: Driver consistency issue
Hi,
Adding the java-driver ML for further question, because this does look like
a bug
Are you able to reproduce it a clean environnement using the same C*
version and driver version ?
On 27 February 2018 at 10:05, Abhishek Kumar Maheshwari <
abhishek.maheshw...@timesinternet.in> wrote:
> Hi Alex,
>
> i have only One DC (with name DC1) and have only one keyspace. So i dont
> think so both of the scenario is possible. (yes in my case QUORUM is
> equivalent
> of ALL)
>
> cqlsh> SELECT * FROM system_schema.keyspaces where keyspace_name='adlog' ;
>
> keyspace_name | durable_writes | replication
> ---++---
>
> adlog | True | {'DC1': '2', 'class':
> 'org.apache.cassandra.locator.NetworkTopologyStrategy'}
>
>
> On Tue, Feb 27, 2018 at 2:27 PM, Oleksandr Shulgin <
> oleksandr.shul...@zalando.de> wrote:
>
>> On Tue, Feb 27, 2018 at 9:45 AM, Abhishek Kumar Maheshwari <
>> abhishek.maheshw...@timesinternet.in> wrote:
>>
>>>
>>> i have a KeySpace in Cassandra (cassandra version 3.0.9- total 12
>>> Servers )With below definition:
>>>
>>> {'DC1': '2', 'class': 'org.apache.cassandra.locator.
>>> NetworkTopologyStrategy'}
>>>
>>> Some time i am getting below exception
>>>
>>> [snip]
>>
>>> Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException:
>>> Cassandra timeout during write query at consistency QUORUM (3 replica were
>>> required but only 2 acknowledged the write)
>>> at com.datastax.driver.core.exceptions.WriteTimeoutException.co
>>> py(WriteTimeoutException.java:100)
>>> at com.datastax.driver.core.Responses$Error.asException(Respons
>>> es.java:134)
>>> at com.datastax.driver.core.RequestHandler$SpeculativeExecution
>>> .onSet(RequestHandler.java:525)
>>> at com.datastax.driver.core.Connection$Dispatcher.channelRead0(
>>> Connection.java:1077)
>>>
>>> why its waiting for acknowledged from 3rd server as replication factor
>>> is 2?
>>>
>>
>> I see two possibilities:
>>
>> 1) The data in this keyspace is replicated to another DC, so there is
>> also 'DC2': '2', for example, but you didn't show it. In this case QUORUM
>> requires more than 2 nodes.
>> 2) The write was targeting a table in a different keyspace than you think.
>>
>> In any case QUORUM (or LOCAL_QUORUM) with RF=2 is equivalent of ALL. Not
>> sure why would you use it in the first place.
>>
>> For consistency levels involving quorum you want to go with RF=3 in a
>> single DC. For multi DC you should think if you want QUORUM or EACH_QUORUM
>> for your writes and figure out the RFs from that.
>>
>> Cheers,
>> --
>> Alex
>>
>>
>
>
> --
>
> *Thanks & Regards,*
> *Abhishek Kumar Maheshwari*
> *+91- 805591 <+91%208%2005591> (Mobile)*
>
> Times Internet Ltd. | A Times of India Group Company
>
> FC - 6, Sector 16A, Film City, Noida, U.P. 201301 | INDIA
>
> *P** Please do not print this email unless it is absolutely necessary.
> Spread environmental awareness.*
>
Re: Driver consistency issue
Hi,
Not always. Randomly i am getting this exception. (one observation, mostly
i got this exception when i add new node in cluster.)
On Tue, Feb 27, 2018 at 4:29 PM, Nicolas Guyomar
wrote:
> Hi,
>
> Adding the java-driver ML for further question, because this does look
> like a bug
>
> Are you able to reproduce it a clean environnement using the same C*
> version and driver version ?
>
>
> On 27 February 2018 at 10:05, Abhishek Kumar Maheshwari <
> abhishek.maheshw...@timesinternet.in> wrote:
>
>> Hi Alex,
>>
>> i have only One DC (with name DC1) and have only one keyspace. So i dont
>> think so both of the scenario is possible. (yes in my case QUORUM is
>> equivalent
>> of ALL)
>>
>> cqlsh> SELECT * FROM system_schema.keyspaces where keyspace_name='adlog'
>> ;
>>
>> keyspace_name | durable_writes | replication
>> ---++---
>>
>> adlog | True | {'DC1': '2', 'class':
>> 'org.apache.cassandra.locator.NetworkTopologyStrategy'}
>>
>>
>> On Tue, Feb 27, 2018 at 2:27 PM, Oleksandr Shulgin <
>> oleksandr.shul...@zalando.de> wrote:
>>
>>> On Tue, Feb 27, 2018 at 9:45 AM, Abhishek Kumar Maheshwari <
>>> abhishek.maheshw...@timesinternet.in> wrote:
>>>
i have a KeySpace in Cassandra (cassandra version 3.0.9- total 12
Servers )With below definition:
{'DC1': '2', 'class': 'org.apache.cassandra.locator.
NetworkTopologyStrategy'}
Some time i am getting below exception
[snip]
>>>
Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException:
Cassandra timeout during write query at consistency QUORUM (3 replica were
required but only 2 acknowledged the write)
at com.datastax.driver.core.exceptions.WriteTimeoutException.co
py(WriteTimeoutException.java:100)
at com.datastax.driver.core.Responses$Error.asException(Respons
es.java:134)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution
.onSet(RequestHandler.java:525)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(
Connection.java:1077)
why its waiting for acknowledged from 3rd server as replication factor
is 2?
>>>
>>> I see two possibilities:
>>>
>>> 1) The data in this keyspace is replicated to another DC, so there is
>>> also 'DC2': '2', for example, but you didn't show it. In this case QUORUM
>>> requires more than 2 nodes.
>>> 2) The write was targeting a table in a different keyspace than you
>>> think.
>>>
>>> In any case QUORUM (or LOCAL_QUORUM) with RF=2 is equivalent of ALL.
>>> Not sure why would you use it in the first place.
>>>
>>> For consistency levels involving quorum you want to go with RF=3 in a
>>> single DC. For multi DC you should think if you want QUORUM or EACH_QUORUM
>>> for your writes and figure out the RFs from that.
>>>
>>> Cheers,
>>> --
>>> Alex
>>>
>>>
>>
>>
>> --
>>
>> *Thanks & Regards,*
>> *Abhishek Kumar Maheshwari*
>> *+91- 805591 <+91%208%2005591> (Mobile)*
>>
>> Times Internet Ltd. | A Times of India Group Company
>>
>> FC - 6, Sector 16A, Film City, Noida, U.P. 201301 | INDIA
>>
>> *P** Please do not print this email unless it is absolutely necessary.
>> Spread environmental awareness.*
>>
>
>
--
*Thanks & Regards,*
*Abhishek Kumar Maheshwari*
*+91- 805591 (Mobile)*
Times Internet Ltd. | A Times of India Group Company
FC - 6, Sector 16A, Film City, Noida, U.P. 201301 | INDIA
*P** Please do not print this email unless it is absolutely necessary.
Spread environmental awareness.*
Re: Driver consistency issue
Hi Abhishek & everyone else,
might it be related to https://issues.apache.org/jira/browse/CASSANDRA-7868
?
regards,
Christian
On Tue, Feb 27, 2018 at 12:46 PM, Abhishek Kumar Maheshwari <
abhishek.maheshw...@timesinternet.in> wrote:
> Hi,
>
> Not always. Randomly i am getting this exception. (one observation, mostly
> i got this exception when i add new node in cluster.)
>
> On Tue, Feb 27, 2018 at 4:29 PM, Nicolas Guyomar <
> nicolas.guyo...@gmail.com> wrote:
>
>> Hi,
>>
>> Adding the java-driver ML for further question, because this does look
>> like a bug
>>
>> Are you able to reproduce it a clean environnement using the same C*
>> version and driver version ?
>>
>>
>> On 27 February 2018 at 10:05, Abhishek Kumar Maheshwari <
>> abhishek.maheshw...@timesinternet.in> wrote:
>>
>>> Hi Alex,
>>>
>>> i have only One DC (with name DC1) and have only one keyspace. So i dont
>>> think so both of the scenario is possible. (yes in my case QUORUM is
>>> equivalent
>>> of ALL)
>>>
>>> cqlsh> SELECT * FROM system_schema.keyspaces where
>>> keyspace_name='adlog' ;
>>>
>>> keyspace_name | durable_writes | replication
>>> ---++---
>>>
>>> adlog | True | {'DC1': '2', 'class':
>>> 'org.apache.cassandra.locator.NetworkTopologyStrategy'}
>>>
>>>
>>> On Tue, Feb 27, 2018 at 2:27 PM, Oleksandr Shulgin <
>>> oleksandr.shul...@zalando.de> wrote:
>>>
On Tue, Feb 27, 2018 at 9:45 AM, Abhishek Kumar Maheshwari <
abhishek.maheshw...@timesinternet.in> wrote:
>
> i have a KeySpace in Cassandra (cassandra version 3.0.9- total 12
> Servers )With below definition:
>
> {'DC1': '2', 'class': 'org.apache.cassandra.locator.
> NetworkTopologyStrategy'}
>
> Some time i am getting below exception
>
> [snip]
> Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException:
> Cassandra timeout during write query at consistency QUORUM (3 replica were
> required but only 2 acknowledged the write)
> at com.datastax.driver.core.excep
> tions.WriteTimeoutException.copy(WriteTimeoutException.java:100)
> at com.datastax.driver.core.Respo
> nses$Error.asException(Responses.java:134)
> at com.datastax.driver.core.Reque
> stHandler$SpeculativeExecution.onSet(RequestHandler.java:525)
> at com.datastax.driver.core.Conne
> ction$Dispatcher.channelRead0(Connection.java:1077)
>
> why its waiting for acknowledged from 3rd server as replication
> factor is 2?
>
I see two possibilities:
1) The data in this keyspace is replicated to another DC, so there is
also 'DC2': '2', for example, but you didn't show it. In this case QUORUM
requires more than 2 nodes.
2) The write was targeting a table in a different keyspace than you
think.
In any case QUORUM (or LOCAL_QUORUM) with RF=2 is equivalent of ALL.
Not sure why would you use it in the first place.
For consistency levels involving quorum you want to go with RF=3 in a
single DC. For multi DC you should think if you want QUORUM or EACH_QUORUM
for your writes and figure out the RFs from that.
Cheers,
--
Alex
>>>
>>>
>>> --
>>>
>>> *Thanks & Regards,*
>>> *Abhishek Kumar Maheshwari*
>>> *+91- 805591 <+91%208%2005591> (Mobile)*
>>>
>>> Times Internet Ltd. | A Times of India Group Company
>>>
>>> FC - 6, Sector 16A, Film City, Noida, U.P. 201301 | INDIA
>>>
>>> *P** Please do not print this email unless it is absolutely necessary.
>>> Spread environmental awareness.*
>>>
>>
>>
>
>
> --
>
> *Thanks & Regards,*
> *Abhishek Kumar Maheshwari*
> *+91- 805591 <+91%208%2005591> (Mobile)*
>
> Times Internet Ltd. | A Times of India Group Company
>
> FC - 6, Sector 16A, Film City, Noida, U.P. 201301 | INDIA
>
> *P** Please do not print this email unless it is absolutely necessary.
> Spread environmental awareness.*
>
Re: Driver consistency issue
Please don’t cross-post multiple lists.
Datastax list removed
Abishek, do you have auth enabled? Does the system auth keyspace have rf=3?
--
Jeff Jirsa
> On Feb 27, 2018, at 2:59 AM, Nicolas Guyomar
> wrote:
>
> Hi,
>
> Adding the java-driver ML for further question, because this does look like a
> bug
>
> Are you able to reproduce it a clean environnement using the same C* version
> and driver version ?
>
>
>> On 27 February 2018 at 10:05, Abhishek Kumar Maheshwari
>> wrote:
>> Hi Alex,
>>
>> i have only One DC (with name DC1) and have only one keyspace. So i dont
>> think so both of the scenario is possible. (yes in my case QUORUM is
>> equivalent of ALL)
>>
>> cqlsh> SELECT * FROM system_schema.keyspaces where keyspace_name='adlog' ;
>>
>> keyspace_name | durable_writes | replication
>> ---++---
>> adlog | True | {'DC1': '2', 'class':
>> 'org.apache.cassandra.locator.NetworkTopologyStrategy'}
>>
>>
>>> On Tue, Feb 27, 2018 at 2:27 PM, Oleksandr Shulgin
>>> wrote:
On Tue, Feb 27, 2018 at 9:45 AM, Abhishek Kumar Maheshwari
wrote:
i have a KeySpace in Cassandra (cassandra version 3.0.9- total 12 Servers
)With below definition:
{'DC1': '2', 'class':
'org.apache.cassandra.locator.NetworkTopologyStrategy'}
Some time i am getting below exception
>>>
>>> [snip]
Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException:
Cassandra timeout during write query at consistency QUORUM (3 replica were
required but only 2 acknowledged the write)
at
com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:100)
at
com.datastax.driver.core.Responses$Error.asException(Responses.java:134)
at
com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:525)
at
com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1077)
why its waiting for acknowledged from 3rd server as replication factor is
2?
>>>
>>> I see two possibilities:
>>>
>>> 1) The data in this keyspace is replicated to another DC, so there is also
>>> 'DC2': '2', for example, but you didn't show it. In this case QUORUM
>>> requires more than 2 nodes.
>>> 2) The write was targeting a table in a different keyspace than you think.
>>>
>>> In any case QUORUM (or LOCAL_QUORUM) with RF=2 is equivalent of ALL. Not
>>> sure why would you use it in the first place.
>>>
>>> For consistency levels involving quorum you want to go with RF=3 in a
>>> single DC. For multi DC you should think if you want QUORUM or EACH_QUORUM
>>> for your writes and figure out the RFs from that.
>>>
>>> Cheers,
>>> --
>>> Alex
>>>
>>
>>
>>
>> --
>> Thanks & Regards,
>> Abhishek Kumar Maheshwari
>> +91- 805591 (Mobile)
>>
>> Times Internet Ltd. | A Times of India Group Company
>>
>> FC - 6, Sector 16A, Film City, Noida, U.P. 201301 | INDIA
>>
>> P Please do not print this email unless it is absolutely necessary. Spread
>> environmental awareness.
>>
>
