Driver consistency issue
Hi All, i have a KeySpace in Cassandra (cassandra version 3.0.9- total 12 Servers )With below definition: {'DC1': '2', 'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy'} Some time i am getting below exception com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency QUORUM (3 replica were required but only 2 acknowledged the write) at com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:73) at com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:26) at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37) at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:68) at com.toi.stream.data.AdImprLogDaoImpl.updateImpr(AdImprLogDaoImpl.java:158) at com.toi.stream.service.AdClickLogAndAdimprLogServiceImpl.updateGoalsOnImpr(AdClickLogAndAdimprLogServiceImpl.java:522) at com.toi.stream.service.ConversionBillingLastAttributionServiceV2Impl.attribute(ConversionBillingLastAttributionServiceV2Impl.java:456) at com.toi.stream.service.ConversionBillingLastAttributionServiceV2Impl.attributeAdTracker(ConversionBillingLastAttributionServiceV2Impl.java:228) at com.toi.stream.process.AdTrackerStreamProcessorV2.process(AdTrackerStreamProcessorV2.java:86) at com.toi.stream.kafka.KafkaGroupConsumer.run(KafkaGroupConsumer.java:175) at java.lang.Thread.run(Thread.java:748) Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency QUORUM (3 replica were required but only 2 acknowledged the write) at com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:100) at com.datastax.driver.core.Responses$Error.asException(Responses.java:134) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:525) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1077) why its waiting for acknowledged from 3rd server as replication factor is 2? -- *Thanks & Regards,* *Abhishek Kumar Maheshwari* *+91- 805591 (Mobile)* Times Internet Ltd. | A Times of India Group Company FC - 6, Sector 16A, Film City, Noida, U.P. 201301 | INDIA *P** Please do not print this email unless it is absolutely necessary. Spread environmental awareness.*
Re: Driver consistency issue
On Tue, Feb 27, 2018 at 9:45 AM, Abhishek Kumar Maheshwari < abhishek.maheshw...@timesinternet.in> wrote: > > i have a KeySpace in Cassandra (cassandra version 3.0.9- total 12 Servers > )With below definition: > > {'DC1': '2', 'class': 'org.apache.cassandra.locator. > NetworkTopologyStrategy'} > > Some time i am getting below exception > > [snip] > Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: > Cassandra timeout during write query at consistency QUORUM (3 replica were > required but only 2 acknowledged the write) > at com.datastax.driver.core.exceptions.WriteTimeoutException. > copy(WriteTimeoutException.java:100) > at com.datastax.driver.core.Responses$Error.asException(Respons > es.java:134) > at com.datastax.driver.core.RequestHandler$SpeculativeExecution > .onSet(RequestHandler.java:525) > at com.datastax.driver.core.Connection$Dispatcher.channelRead0( > Connection.java:1077) > > why its waiting for acknowledged from 3rd server as replication factor is > 2? > I see two possibilities: 1) The data in this keyspace is replicated to another DC, so there is also 'DC2': '2', for example, but you didn't show it. In this case QUORUM requires more than 2 nodes. 2) The write was targeting a table in a different keyspace than you think. In any case QUORUM (or LOCAL_QUORUM) with RF=2 is equivalent of ALL. Not sure why would you use it in the first place. For consistency levels involving quorum you want to go with RF=3 in a single DC. For multi DC you should think if you want QUORUM or EACH_QUORUM for your writes and figure out the RFs from that. Cheers, -- Alex
Re: Driver consistency issue
Hi Alex, i have only One DC (with name DC1) and have only one keyspace. So i dont think so both of the scenario is possible. (yes in my case QUORUM is equivalent of ALL) cqlsh> SELECT * FROM system_schema.keyspaces where keyspace_name='adlog' ; keyspace_name | durable_writes | replication ---++--- adlog | True | {'DC1': '2', 'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy'} On Tue, Feb 27, 2018 at 2:27 PM, Oleksandr Shulgin < oleksandr.shul...@zalando.de> wrote: > On Tue, Feb 27, 2018 at 9:45 AM, Abhishek Kumar Maheshwari < > abhishek.maheshw...@timesinternet.in> wrote: > >> >> i have a KeySpace in Cassandra (cassandra version 3.0.9- total 12 Servers >> )With below definition: >> >> {'DC1': '2', 'class': 'org.apache.cassandra.locator. >> NetworkTopologyStrategy'} >> >> Some time i am getting below exception >> >> [snip] > >> Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: >> Cassandra timeout during write query at consistency QUORUM (3 replica were >> required but only 2 acknowledged the write) >> at com.datastax.driver.core.exceptions.WriteTimeoutException.co >> py(WriteTimeoutException.java:100) >> at com.datastax.driver.core.Responses$Error.asException(Respons >> es.java:134) >> at com.datastax.driver.core.RequestHandler$SpeculativeExecution >> .onSet(RequestHandler.java:525) >> at com.datastax.driver.core.Connection$Dispatcher.channelRead0( >> Connection.java:1077) >> >> why its waiting for acknowledged from 3rd server as replication factor >> is 2? >> > > I see two possibilities: > > 1) The data in this keyspace is replicated to another DC, so there is also > 'DC2': '2', for example, but you didn't show it. In this case QUORUM > requires more than 2 nodes. > 2) The write was targeting a table in a different keyspace than you think. > > In any case QUORUM (or LOCAL_QUORUM) with RF=2 is equivalent of ALL. Not > sure why would you use it in the first place. > > For consistency levels involving quorum you want to go with RF=3 in a > single DC. For multi DC you should think if you want QUORUM or EACH_QUORUM > for your writes and figure out the RFs from that. > > Cheers, > -- > Alex > > -- *Thanks & Regards,* *Abhishek Kumar Maheshwari* *+91- 805591 (Mobile)* Times Internet Ltd. | A Times of India Group Company FC - 6, Sector 16A, Film City, Noida, U.P. 201301 | INDIA *P** Please do not print this email unless it is absolutely necessary. Spread environmental awareness.*
Re: Driver consistency issue
Hi, Adding the java-driver ML for further question, because this does look like a bug Are you able to reproduce it a clean environnement using the same C* version and driver version ? On 27 February 2018 at 10:05, Abhishek Kumar Maheshwari < abhishek.maheshw...@timesinternet.in> wrote: > Hi Alex, > > i have only One DC (with name DC1) and have only one keyspace. So i dont > think so both of the scenario is possible. (yes in my case QUORUM is > equivalent > of ALL) > > cqlsh> SELECT * FROM system_schema.keyspaces where keyspace_name='adlog' ; > > keyspace_name | durable_writes | replication > ---++--- > > adlog | True | {'DC1': '2', 'class': > 'org.apache.cassandra.locator.NetworkTopologyStrategy'} > > > On Tue, Feb 27, 2018 at 2:27 PM, Oleksandr Shulgin < > oleksandr.shul...@zalando.de> wrote: > >> On Tue, Feb 27, 2018 at 9:45 AM, Abhishek Kumar Maheshwari < >> abhishek.maheshw...@timesinternet.in> wrote: >> >>> >>> i have a KeySpace in Cassandra (cassandra version 3.0.9- total 12 >>> Servers )With below definition: >>> >>> {'DC1': '2', 'class': 'org.apache.cassandra.locator. >>> NetworkTopologyStrategy'} >>> >>> Some time i am getting below exception >>> >>> [snip] >> >>> Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: >>> Cassandra timeout during write query at consistency QUORUM (3 replica were >>> required but only 2 acknowledged the write) >>> at com.datastax.driver.core.exceptions.WriteTimeoutException.co >>> py(WriteTimeoutException.java:100) >>> at com.datastax.driver.core.Responses$Error.asException(Respons >>> es.java:134) >>> at com.datastax.driver.core.RequestHandler$SpeculativeExecution >>> .onSet(RequestHandler.java:525) >>> at com.datastax.driver.core.Connection$Dispatcher.channelRead0( >>> Connection.java:1077) >>> >>> why its waiting for acknowledged from 3rd server as replication factor >>> is 2? >>> >> >> I see two possibilities: >> >> 1) The data in this keyspace is replicated to another DC, so there is >> also 'DC2': '2', for example, but you didn't show it. In this case QUORUM >> requires more than 2 nodes. >> 2) The write was targeting a table in a different keyspace than you think. >> >> In any case QUORUM (or LOCAL_QUORUM) with RF=2 is equivalent of ALL. Not >> sure why would you use it in the first place. >> >> For consistency levels involving quorum you want to go with RF=3 in a >> single DC. For multi DC you should think if you want QUORUM or EACH_QUORUM >> for your writes and figure out the RFs from that. >> >> Cheers, >> -- >> Alex >> >> > > > -- > > *Thanks & Regards,* > *Abhishek Kumar Maheshwari* > *+91- 805591 <+91%208%2005591> (Mobile)* > > Times Internet Ltd. | A Times of India Group Company > > FC - 6, Sector 16A, Film City, Noida, U.P. 201301 | INDIA > > *P** Please do not print this email unless it is absolutely necessary. > Spread environmental awareness.* >
Re: Driver consistency issue
Hi, Not always. Randomly i am getting this exception. (one observation, mostly i got this exception when i add new node in cluster.) On Tue, Feb 27, 2018 at 4:29 PM, Nicolas Guyomar wrote: > Hi, > > Adding the java-driver ML for further question, because this does look > like a bug > > Are you able to reproduce it a clean environnement using the same C* > version and driver version ? > > > On 27 February 2018 at 10:05, Abhishek Kumar Maheshwari < > abhishek.maheshw...@timesinternet.in> wrote: > >> Hi Alex, >> >> i have only One DC (with name DC1) and have only one keyspace. So i dont >> think so both of the scenario is possible. (yes in my case QUORUM is >> equivalent >> of ALL) >> >> cqlsh> SELECT * FROM system_schema.keyspaces where keyspace_name='adlog' >> ; >> >> keyspace_name | durable_writes | replication >> ---++--- >> >> adlog | True | {'DC1': '2', 'class': >> 'org.apache.cassandra.locator.NetworkTopologyStrategy'} >> >> >> On Tue, Feb 27, 2018 at 2:27 PM, Oleksandr Shulgin < >> oleksandr.shul...@zalando.de> wrote: >> >>> On Tue, Feb 27, 2018 at 9:45 AM, Abhishek Kumar Maheshwari < >>> abhishek.maheshw...@timesinternet.in> wrote: >>> i have a KeySpace in Cassandra (cassandra version 3.0.9- total 12 Servers )With below definition: {'DC1': '2', 'class': 'org.apache.cassandra.locator. NetworkTopologyStrategy'} Some time i am getting below exception [snip] >>> Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency QUORUM (3 replica were required but only 2 acknowledged the write) at com.datastax.driver.core.exceptions.WriteTimeoutException.co py(WriteTimeoutException.java:100) at com.datastax.driver.core.Responses$Error.asException(Respons es.java:134) at com.datastax.driver.core.RequestHandler$SpeculativeExecution .onSet(RequestHandler.java:525) at com.datastax.driver.core.Connection$Dispatcher.channelRead0( Connection.java:1077) why its waiting for acknowledged from 3rd server as replication factor is 2? >>> >>> I see two possibilities: >>> >>> 1) The data in this keyspace is replicated to another DC, so there is >>> also 'DC2': '2', for example, but you didn't show it. In this case QUORUM >>> requires more than 2 nodes. >>> 2) The write was targeting a table in a different keyspace than you >>> think. >>> >>> In any case QUORUM (or LOCAL_QUORUM) with RF=2 is equivalent of ALL. >>> Not sure why would you use it in the first place. >>> >>> For consistency levels involving quorum you want to go with RF=3 in a >>> single DC. For multi DC you should think if you want QUORUM or EACH_QUORUM >>> for your writes and figure out the RFs from that. >>> >>> Cheers, >>> -- >>> Alex >>> >>> >> >> >> -- >> >> *Thanks & Regards,* >> *Abhishek Kumar Maheshwari* >> *+91- 805591 <+91%208%2005591> (Mobile)* >> >> Times Internet Ltd. | A Times of India Group Company >> >> FC - 6, Sector 16A, Film City, Noida, U.P. 201301 | INDIA >> >> *P** Please do not print this email unless it is absolutely necessary. >> Spread environmental awareness.* >> > > -- *Thanks & Regards,* *Abhishek Kumar Maheshwari* *+91- 805591 (Mobile)* Times Internet Ltd. | A Times of India Group Company FC - 6, Sector 16A, Film City, Noida, U.P. 201301 | INDIA *P** Please do not print this email unless it is absolutely necessary. Spread environmental awareness.*
Re: Driver consistency issue
Hi Abhishek & everyone else, might it be related to https://issues.apache.org/jira/browse/CASSANDRA-7868 ? regards, Christian On Tue, Feb 27, 2018 at 12:46 PM, Abhishek Kumar Maheshwari < abhishek.maheshw...@timesinternet.in> wrote: > Hi, > > Not always. Randomly i am getting this exception. (one observation, mostly > i got this exception when i add new node in cluster.) > > On Tue, Feb 27, 2018 at 4:29 PM, Nicolas Guyomar < > nicolas.guyo...@gmail.com> wrote: > >> Hi, >> >> Adding the java-driver ML for further question, because this does look >> like a bug >> >> Are you able to reproduce it a clean environnement using the same C* >> version and driver version ? >> >> >> On 27 February 2018 at 10:05, Abhishek Kumar Maheshwari < >> abhishek.maheshw...@timesinternet.in> wrote: >> >>> Hi Alex, >>> >>> i have only One DC (with name DC1) and have only one keyspace. So i dont >>> think so both of the scenario is possible. (yes in my case QUORUM is >>> equivalent >>> of ALL) >>> >>> cqlsh> SELECT * FROM system_schema.keyspaces where >>> keyspace_name='adlog' ; >>> >>> keyspace_name | durable_writes | replication >>> ---++--- >>> >>> adlog | True | {'DC1': '2', 'class': >>> 'org.apache.cassandra.locator.NetworkTopologyStrategy'} >>> >>> >>> On Tue, Feb 27, 2018 at 2:27 PM, Oleksandr Shulgin < >>> oleksandr.shul...@zalando.de> wrote: >>> On Tue, Feb 27, 2018 at 9:45 AM, Abhishek Kumar Maheshwari < abhishek.maheshw...@timesinternet.in> wrote: > > i have a KeySpace in Cassandra (cassandra version 3.0.9- total 12 > Servers )With below definition: > > {'DC1': '2', 'class': 'org.apache.cassandra.locator. > NetworkTopologyStrategy'} > > Some time i am getting below exception > > [snip] > Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: > Cassandra timeout during write query at consistency QUORUM (3 replica were > required but only 2 acknowledged the write) > at com.datastax.driver.core.excep > tions.WriteTimeoutException.copy(WriteTimeoutException.java:100) > at com.datastax.driver.core.Respo > nses$Error.asException(Responses.java:134) > at com.datastax.driver.core.Reque > stHandler$SpeculativeExecution.onSet(RequestHandler.java:525) > at com.datastax.driver.core.Conne > ction$Dispatcher.channelRead0(Connection.java:1077) > > why its waiting for acknowledged from 3rd server as replication > factor is 2? > I see two possibilities: 1) The data in this keyspace is replicated to another DC, so there is also 'DC2': '2', for example, but you didn't show it. In this case QUORUM requires more than 2 nodes. 2) The write was targeting a table in a different keyspace than you think. In any case QUORUM (or LOCAL_QUORUM) with RF=2 is equivalent of ALL. Not sure why would you use it in the first place. For consistency levels involving quorum you want to go with RF=3 in a single DC. For multi DC you should think if you want QUORUM or EACH_QUORUM for your writes and figure out the RFs from that. Cheers, -- Alex >>> >>> >>> -- >>> >>> *Thanks & Regards,* >>> *Abhishek Kumar Maheshwari* >>> *+91- 805591 <+91%208%2005591> (Mobile)* >>> >>> Times Internet Ltd. | A Times of India Group Company >>> >>> FC - 6, Sector 16A, Film City, Noida, U.P. 201301 | INDIA >>> >>> *P** Please do not print this email unless it is absolutely necessary. >>> Spread environmental awareness.* >>> >> >> > > > -- > > *Thanks & Regards,* > *Abhishek Kumar Maheshwari* > *+91- 805591 <+91%208%2005591> (Mobile)* > > Times Internet Ltd. | A Times of India Group Company > > FC - 6, Sector 16A, Film City, Noida, U.P. 201301 | INDIA > > *P** Please do not print this email unless it is absolutely necessary. > Spread environmental awareness.* >
Re: Driver consistency issue
Please don’t cross-post multiple lists. Datastax list removed Abishek, do you have auth enabled? Does the system auth keyspace have rf=3? -- Jeff Jirsa > On Feb 27, 2018, at 2:59 AM, Nicolas Guyomar > wrote: > > Hi, > > Adding the java-driver ML for further question, because this does look like a > bug > > Are you able to reproduce it a clean environnement using the same C* version > and driver version ? > > >> On 27 February 2018 at 10:05, Abhishek Kumar Maheshwari >> wrote: >> Hi Alex, >> >> i have only One DC (with name DC1) and have only one keyspace. So i dont >> think so both of the scenario is possible. (yes in my case QUORUM is >> equivalent of ALL) >> >> cqlsh> SELECT * FROM system_schema.keyspaces where keyspace_name='adlog' ; >> >> keyspace_name | durable_writes | replication >> ---++--- >> adlog | True | {'DC1': '2', 'class': >> 'org.apache.cassandra.locator.NetworkTopologyStrategy'} >> >> >>> On Tue, Feb 27, 2018 at 2:27 PM, Oleksandr Shulgin >>> wrote: On Tue, Feb 27, 2018 at 9:45 AM, Abhishek Kumar Maheshwari wrote: i have a KeySpace in Cassandra (cassandra version 3.0.9- total 12 Servers )With below definition: {'DC1': '2', 'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy'} Some time i am getting below exception >>> >>> [snip] Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency QUORUM (3 replica were required but only 2 acknowledged the write) at com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:100) at com.datastax.driver.core.Responses$Error.asException(Responses.java:134) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:525) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1077) why its waiting for acknowledged from 3rd server as replication factor is 2? >>> >>> I see two possibilities: >>> >>> 1) The data in this keyspace is replicated to another DC, so there is also >>> 'DC2': '2', for example, but you didn't show it. In this case QUORUM >>> requires more than 2 nodes. >>> 2) The write was targeting a table in a different keyspace than you think. >>> >>> In any case QUORUM (or LOCAL_QUORUM) with RF=2 is equivalent of ALL. Not >>> sure why would you use it in the first place. >>> >>> For consistency levels involving quorum you want to go with RF=3 in a >>> single DC. For multi DC you should think if you want QUORUM or EACH_QUORUM >>> for your writes and figure out the RFs from that. >>> >>> Cheers, >>> -- >>> Alex >>> >> >> >> >> -- >> Thanks & Regards, >> Abhishek Kumar Maheshwari >> +91- 805591 (Mobile) >> >> Times Internet Ltd. | A Times of India Group Company >> >> FC - 6, Sector 16A, Film City, Noida, U.P. 201301 | INDIA >> >> P Please do not print this email unless it is absolutely necessary. Spread >> environmental awareness. >> >