Re: iteration does not yield all data with consistency ONE
Great. Thanks for helping us track that down! On Mon, Nov 15, 2010 at 6:45 AM, Eric van Orsouw wrote: > Jonathan, I have downloaded 0.6.8 and the iteration problem is fixed. > Thanks, > Eric > > -Original Message- > From: Jonathan Ellis [mailto:jbel...@gmail.com] > Sent: donderdag 11 november 2010 23:05 > To: user > Subject: Re: iteration does not yield all data with consistency ONE > > Can you try with the latest 0.6 svn branch? > https://svn.apache.org/repos/asf/cassandra/branches/cassandra-0.6 It > may be caused by https://issues.apache.org/jira/browse/CASSANDRA-1722 > which is fixed there. > > On Thu, Nov 11, 2010 at 4:56 AM, Eric van Orsouw > wrote: >> I have tried it with RF=1 and 2 nodes and the result was the same. >> In this case I am not able to query anything, regardless the consistency >> level. >> >> I have conducted some more tests with 4 nodes and RF=2 and before each test >> I also completely cleared all datafiles (data/commit/saved_cache). >> Each time I do this the nodes get different locations on the ring and the >> noticed behavior is different (but similar) each time. >> I have added the details for 2 concrete scenario; >> >> >> 1) clear all 3 data directories for all nodes >> 2) startup all nodes; the ring is; >> >> # bin/nodetool -h 172.16.21.12 ring >> Address Status Load Range >> Ring >> 81296677393575658129031223607666487650 >> 172.16.21.12 Up 489 bytes >> 61644956546475371096781179489550878842 |<--| >> 172.16.21.14 Up 489 bytes >> 63811055530446251457866768401307252614 | | >> 172.16.21.13 Up 489 bytes >> 71622328744148061175473750202165524046 | | >> 172.16.21.11 Up 489 bytes >> 81296677393575658129031223607666487650 |-->| >> >> 3) I use the following ColumnFamily > ColumnType="Super" CompareWith="AsciiType"/> >> And add the following rows (key and values utf8 strings) >> Customer['1'].simple.id='1' >> Customer['2'].simple.id='2' >> Customer['3'].simple.id='3' >> Customer['4'].simple.id='4' >> >> 4) Iterating over the columns now yield; >> Iterating Nodes .11 thru.13 ==> all rows, regardless of consistency >> level >> Iterating Node.14 ==> only row '3' and '4' with consistency >> level ONE, all rows with consistency level All >> >> >> Repating all process steps again yields the following ring and iteration >> results >> # bin/nodetool -h 172.16.21.11 ring >> Address Status Load Range >> Ring >> 157208721428268855331814529932509412075 >> 172.16.21.11 Up 489 bytes >> 18662665007498667259581716616851966032 |<--| >> 172.16.21.13 Up 489 bytes >> 125997322478065775200568923383951796531 | | >> 172.16.21.12 Up 489 bytes >> 148125235552363592916130178190645177045 | | >> 172.16.21.14 Up 489 bytes >> 157208721428268855331814529932509412075 |-->| >> >> Iterating Nodes .11 .12 .14 ==> no data with consistency level One, >> all rows with consistency level All >> Iterating Node .13 ==> only row '3' and '4' with consistency >> level ONE, all rows with consistency level All >> >> I appears to me there is an inconsistency somewhere, I hope this info helps. >> >> Eric >> >> >> -Original Message- >> From: Jonathan Ellis [mailto:jbel...@gmail.com] >> Sent: woensdag 10 november 2010 17:48 >> To: user >> Subject: Re: iteration does not yield all data with consistency ONE >> >> Interesting. Does it simplify further to RF=1 and 2 nodes? >> >> On Wed, Nov 10, 2010 at 8:58 AM, Eric van Orsouw >> wrote: >>> No, all nodes were up and running while the single key was inserted. >>> The insert however was with consistency One. I assume however that the >>> replicas are still written in this case. >>> It is btw also very reproducible. >>> >>> -Original Message- >>> From: Jonathan Ellis [mailto:jbel...@gmail.com] >>> Sent: woensdag 10 november 2010 15:44 >>> To: user >>> Subject: Re: iteration does not yield all data with consis
RE: iteration does not yield all data with consistency ONE
Jonathan, I have downloaded 0.6.8 and the iteration problem is fixed. Thanks, Eric -Original Message- From: Jonathan Ellis [mailto:jbel...@gmail.com] Sent: donderdag 11 november 2010 23:05 To: user Subject: Re: iteration does not yield all data with consistency ONE Can you try with the latest 0.6 svn branch? https://svn.apache.org/repos/asf/cassandra/branches/cassandra-0.6 It may be caused by https://issues.apache.org/jira/browse/CASSANDRA-1722 which is fixed there. On Thu, Nov 11, 2010 at 4:56 AM, Eric van Orsouw wrote: > I have tried it with RF=1 and 2 nodes and the result was the same. > In this case I am not able to query anything, regardless the consistency > level. > > I have conducted some more tests with 4 nodes and RF=2 and before each test I > also completely cleared all datafiles (data/commit/saved_cache). > Each time I do this the nodes get different locations on the ring and the > noticed behavior is different (but similar) each time. > I have added the details for 2 concrete scenario; > > > 1) clear all 3 data directories for all nodes > 2) startup all nodes; the ring is; > > # bin/nodetool -h 172.16.21.12 ring > Address Status Load Range > Ring > 81296677393575658129031223607666487650 > 172.16.21.12 Up 489 bytes 61644956546475371096781179489550878842 > |<--| > 172.16.21.14 Up 489 bytes 63811055530446251457866768401307252614 > | | > 172.16.21.13 Up 489 bytes 71622328744148061175473750202165524046 > | | > 172.16.21.11 Up 489 bytes 81296677393575658129031223607666487650 > |-->| > > 3) I use the following ColumnFamily ColumnType="Super" CompareWith="AsciiType"/> > And add the following rows (key and values utf8 strings) > Customer['1'].simple.id='1' > Customer['2'].simple.id='2' > Customer['3'].simple.id='3' > Customer['4'].simple.id='4' > > 4) Iterating over the columns now yield; > Iterating Nodes .11 thru.13 ==> all rows, regardless of consistency > level > Iterating Node.14 ==> only row '3' and '4' with consistency > level ONE, all rows with consistency level All > > > Repating all process steps again yields the following ring and iteration > results > # bin/nodetool -h 172.16.21.11 ring > Address Status Load Range > Ring > 157208721428268855331814529932509412075 > 172.16.21.11 Up 489 bytes 18662665007498667259581716616851966032 > |<--| > 172.16.21.13 Up 489 bytes > 125997322478065775200568923383951796531 | | > 172.16.21.12 Up 489 bytes > 148125235552363592916130178190645177045 | | > 172.16.21.14 Up 489 bytes > 157208721428268855331814529932509412075 |-->| > > Iterating Nodes .11 .12 .14 ==> no data with consistency level One, > all rows with consistency level All > Iterating Node .13 ==> only row '3' and '4' with consistency > level ONE, all rows with consistency level All > > I appears to me there is an inconsistency somewhere, I hope this info helps. > > Eric > > > -Original Message- > From: Jonathan Ellis [mailto:jbel...@gmail.com] > Sent: woensdag 10 november 2010 17:48 > To: user > Subject: Re: iteration does not yield all data with consistency ONE > > Interesting. Does it simplify further to RF=1 and 2 nodes? > > On Wed, Nov 10, 2010 at 8:58 AM, Eric van Orsouw > wrote: >> No, all nodes were up and running while the single key was inserted. >> The insert however was with consistency One. I assume however that the >> replicas are still written in this case. >> It is btw also very reproducible. >> >> -Original Message- >> From: Jonathan Ellis [mailto:jbel...@gmail.com] >> Sent: woensdag 10 november 2010 15:44 >> To: user >> Subject: Re: iteration does not yield all data with consistency ONE >> >> Was the node that should have the other replica of this row down when >> it was inserted? >> >> On Wed, Nov 10, 2010 at 6:08 AM, Eric van Orsouw >> wrote: >>> >>> Hello, >>> >>> >>> >>> We have a cluster of 4 nodes (0.6.6) and use the random partitioner and a >>> replication of 2. >>> >>> When I insert a number of rows I can always retrieve them by their explicit >>> id (get_range_slices
Re: iteration does not yield all data with consistency ONE
Can you try with the latest 0.6 svn branch?
https://svn.apache.org/repos/asf/cassandra/branches/cassandra-0.6 It
may be caused by https://issues.apache.org/jira/browse/CASSANDRA-1722
which is fixed there.
On Thu, Nov 11, 2010 at 4:56 AM, Eric van Orsouw
wrote:
> I have tried it with RF=1 and 2 nodes and the result was the same.
> In this case I am not able to query anything, regardless the consistency
> level.
>
> I have conducted some more tests with 4 nodes and RF=2 and before each test I
> also completely cleared all datafiles (data/commit/saved_cache).
> Each time I do this the nodes get different locations on the ring and the
> noticed behavior is different (but similar) each time.
> I have added the details for 2 concrete scenario;
>
>
> 1) clear all 3 data directories for all nodes
> 2) startup all nodes; the ring is;
>
> # bin/nodetool -h 172.16.21.12 ring
> Address Status Load Range
> Ring
> 81296677393575658129031223607666487650
> 172.16.21.12 Up 489 bytes 61644956546475371096781179489550878842
> |<--|
> 172.16.21.14 Up 489 bytes 63811055530446251457866768401307252614
> | |
> 172.16.21.13 Up 489 bytes 71622328744148061175473750202165524046
> | |
> 172.16.21.11 Up 489 bytes 81296677393575658129031223607666487650
> |-->|
>
> 3) I use the following ColumnFamily ColumnType="Super" CompareWith="AsciiType"/>
> And add the following rows (key and values utf8 strings)
> Customer['1'].simple.id='1'
> Customer['2'].simple.id='2'
> Customer['3'].simple.id='3'
> Customer['4'].simple.id='4'
>
> 4) Iterating over the columns now yield;
> Iterating Nodes .11 thru.13 ==> all rows, regardless of consistency
> level
> Iterating Node.14 ==> only row '3' and '4' with consistency
> level ONE, all rows with consistency level All
>
>
> Repating all process steps again yields the following ring and iteration
> results
> # bin/nodetool -h 172.16.21.11 ring
> Address Status Load Range
> Ring
> 157208721428268855331814529932509412075
> 172.16.21.11 Up 489 bytes 18662665007498667259581716616851966032
> |<--|
> 172.16.21.13 Up 489 bytes
> 125997322478065775200568923383951796531 | |
> 172.16.21.12 Up 489 bytes
> 148125235552363592916130178190645177045 | |
> 172.16.21.14 Up 489 bytes
> 157208721428268855331814529932509412075 |-->|
>
> Iterating Nodes .11 .12 .14 ==> no data with consistency level One,
> all rows with consistency level All
> Iterating Node .13 ==> only row '3' and '4' with consistency
> level ONE, all rows with consistency level All
>
> I appears to me there is an inconsistency somewhere, I hope this info helps.
>
> Eric
>
>
> -Original Message-
> From: Jonathan Ellis [mailto:jbel...@gmail.com]
> Sent: woensdag 10 november 2010 17:48
> To: user
> Subject: Re: iteration does not yield all data with consistency ONE
>
> Interesting. Does it simplify further to RF=1 and 2 nodes?
>
> On Wed, Nov 10, 2010 at 8:58 AM, Eric van Orsouw
> wrote:
>> No, all nodes were up and running while the single key was inserted.
>> The insert however was with consistency One. I assume however that the
>> replicas are still written in this case.
>> It is btw also very reproducible.
>>
>> -Original Message-
>> From: Jonathan Ellis [mailto:jbel...@gmail.com]
>> Sent: woensdag 10 november 2010 15:44
>> To: user
>> Subject: Re: iteration does not yield all data with consistency ONE
>>
>> Was the node that should have the other replica of this row down when
>> it was inserted?
>>
>> On Wed, Nov 10, 2010 at 6:08 AM, Eric van Orsouw
>> wrote:
>>>
>>> Hello,
>>>
>>>
>>>
>>> We have a cluster of 4 nodes (0.6.6) and use the random partitioner and a
>>> replication of 2.
>>>
>>> When I insert a number of rows I can always retrieve them by their explicit
>>> id (get_range_slices("","", 1).
>>>
>>> Playing with consistency levels and temporarily shutting down a Cassandra
>>> node all yields the expected result.
>>>
>>>
>>>
>>> However when I use get_range_slic
Re: iteration does not yield all data with consistency ONE
On Thu, Nov 11, 2010 at 4:56 AM, Eric van Orsouw wrote: > I appears to me there is an inconsistency somewhere, I hope this info > helps. get_range_slices does not perform read repair, so inconsistencies at cl.ONE won't be resolved. Invoke nodetool repair everywhere and try again. -Brandon
RE: iteration does not yield all data with consistency ONE
I have tried it with RF=1 and 2 nodes and the result was the same.
In this case I am not able to query anything, regardless the consistency level.
I have conducted some more tests with 4 nodes and RF=2 and before each test I
also completely cleared all datafiles (data/commit/saved_cache).
Each time I do this the nodes get different locations on the ring and the
noticed behavior is different (but similar) each time.
I have added the details for 2 concrete scenario;
1) clear all 3 data directories for all nodes
2) startup all nodes; the ring is;
# bin/nodetool -h 172.16.21.12 ring
Address Status Load Range
Ring
81296677393575658129031223607666487650
172.16.21.12 Up 489 bytes 61644956546475371096781179489550878842
|<--|
172.16.21.14 Up 489 bytes 63811055530446251457866768401307252614
| |
172.16.21.13 Up 489 bytes 71622328744148061175473750202165524046
| |
172.16.21.11 Up 489 bytes 81296677393575658129031223607666487650
|-->|
3) I use the following ColumnFamily
And add the following rows (key and values utf8 strings)
Customer['1'].simple.id='1'
Customer['2'].simple.id='2'
Customer['3'].simple.id='3'
Customer['4'].simple.id='4'
4) Iterating over the columns now yield;
Iterating Nodes .11 thru.13 ==> all rows, regardless of consistency
level
Iterating Node.14 ==> only row '3' and '4' with consistency
level ONE, all rows with consistency level All
Repating all process steps again yields the following ring and iteration results
# bin/nodetool -h 172.16.21.11 ring
Address Status Load Range
Ring
157208721428268855331814529932509412075
172.16.21.11 Up 489 bytes 18662665007498667259581716616851966032
|<--|
172.16.21.13 Up 489 bytes 125997322478065775200568923383951796531
| |
172.16.21.12 Up 489 bytes 148125235552363592916130178190645177045
| |
172.16.21.14 Up 489 bytes 157208721428268855331814529932509412075
|-->|
Iterating Nodes .11 .12 .14 ==> no data with consistency level One, all
rows with consistency level All
Iterating Node .13 ==> only row '3' and '4' with consistency
level ONE, all rows with consistency level All
I appears to me there is an inconsistency somewhere, I hope this info helps.
Eric
-----Original Message-
From: Jonathan Ellis [mailto:jbel...@gmail.com]
Sent: woensdag 10 november 2010 17:48
To: user
Subject: Re: iteration does not yield all data with consistency ONE
Interesting. Does it simplify further to RF=1 and 2 nodes?
On Wed, Nov 10, 2010 at 8:58 AM, Eric van Orsouw
wrote:
> No, all nodes were up and running while the single key was inserted.
> The insert however was with consistency One. I assume however that the
> replicas are still written in this case.
> It is btw also very reproducible.
>
> -Original Message-
> From: Jonathan Ellis [mailto:jbel...@gmail.com]
> Sent: woensdag 10 november 2010 15:44
> To: user
> Subject: Re: iteration does not yield all data with consistency ONE
>
> Was the node that should have the other replica of this row down when
> it was inserted?
>
> On Wed, Nov 10, 2010 at 6:08 AM, Eric van Orsouw
> wrote:
>>
>> Hello,
>>
>>
>>
>> We have a cluster of 4 nodes (0.6.6) and use the random partitioner and a
>> replication of 2.
>>
>> When I insert a number of rows I can always retrieve them by their explicit
>> id (get_range_slices("","", 1).
>>
>> Playing with consistency levels and temporarily shutting down a Cassandra
>> node all yields the expected result.
>>
>>
>>
>> However when I use get_range_slices("","", n) to iterate over all rows, I
>> sometimes don't get anything (depending on the node).
>>
>>
>>
>> I then reduced the problem to inserting just a single row.
>>
>> Specifically, the 'iteration' only seems to succeed when I issue the request
>> to the node that contains the first copy.
>>
>> I Discovered that when I iterate using a consistency level of Quorum/All the
>> iteration always succeeds and I properly get the one row.
>>
>>
>>
>> So a solution would be to always use consistency level One/All but that has
>> a performance penalty.
>>
>>
>>
>> Can anyone explain why iterating using get_range_slices("","",n) does not
>>
Re: iteration does not yield all data with consistency ONE
Interesting. Does it simplify further to RF=1 and 2 nodes?
On Wed, Nov 10, 2010 at 8:58 AM, Eric van Orsouw
wrote:
> No, all nodes were up and running while the single key was inserted.
> The insert however was with consistency One. I assume however that the
> replicas are still written in this case.
> It is btw also very reproducible.
>
> -Original Message-
> From: Jonathan Ellis [mailto:jbel...@gmail.com]
> Sent: woensdag 10 november 2010 15:44
> To: user
> Subject: Re: iteration does not yield all data with consistency ONE
>
> Was the node that should have the other replica of this row down when
> it was inserted?
>
> On Wed, Nov 10, 2010 at 6:08 AM, Eric van Orsouw
> wrote:
>>
>> Hello,
>>
>>
>>
>> We have a cluster of 4 nodes (0.6.6) and use the random partitioner and a
>> replication of 2.
>>
>> When I insert a number of rows I can always retrieve them by their explicit
>> id (get_range_slices("","", 1).
>>
>> Playing with consistency levels and temporarily shutting down a Cassandra
>> node all yields the expected result.
>>
>>
>>
>> However when I use get_range_slices("","", n) to iterate over all rows, I
>> sometimes don't get anything (depending on the node).
>>
>>
>>
>> I then reduced the problem to inserting just a single row.
>>
>> Specifically, the 'iteration' only seems to succeed when I issue the request
>> to the node that contains the first copy.
>>
>> I Discovered that when I iterate using a consistency level of Quorum/All the
>> iteration always succeeds and I properly get the one row.
>>
>>
>>
>> So a solution would be to always use consistency level One/All but that has
>> a performance penalty.
>>
>>
>>
>> Can anyone explain why iterating using get_range_slices("","",n) does not
>> always function with consistency level One on all nodes?
>>
>>
>>
>> Thanks,
>>
>> Eric
>>
>>
>>
>> P.S. To rule out any discussion on whether or not to use iteration in the
>> first place, we only plan to use it for backup and periodic cleanup cycles.
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
RE: iteration does not yield all data with consistency ONE
No, all nodes were up and running while the single key was inserted.
The insert however was with consistency One. I assume however that the replicas
are still written in this case.
It is btw also very reproducible.
-Original Message-
From: Jonathan Ellis [mailto:jbel...@gmail.com]
Sent: woensdag 10 november 2010 15:44
To: user
Subject: Re: iteration does not yield all data with consistency ONE
Was the node that should have the other replica of this row down when
it was inserted?
On Wed, Nov 10, 2010 at 6:08 AM, Eric van Orsouw
wrote:
>
> Hello,
>
>
>
> We have a cluster of 4 nodes (0.6.6) and use the random partitioner and a
> replication of 2.
>
> When I insert a number of rows I can always retrieve them by their explicit
> id (get_range_slices("","", 1).
>
> Playing with consistency levels and temporarily shutting down a Cassandra
> node all yields the expected result.
>
>
>
> However when I use get_range_slices("","", n) to iterate over all rows, I
> sometimes don't get anything (depending on the node).
>
>
>
> I then reduced the problem to inserting just a single row.
>
> Specifically, the 'iteration' only seems to succeed when I issue the request
> to the node that contains the first copy.
>
> I Discovered that when I iterate using a consistency level of Quorum/All the
> iteration always succeeds and I properly get the one row.
>
>
>
> So a solution would be to always use consistency level One/All but that has a
> performance penalty.
>
>
>
> Can anyone explain why iterating using get_range_slices("","",n) does not
> always function with consistency level One on all nodes?
>
>
>
> Thanks,
>
> Eric
>
>
>
> P.S. To rule out any discussion on whether or not to use iteration in the
> first place, we only plan to use it for backup and periodic cleanup cycles.
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: iteration does not yield all data with consistency ONE
Was the node that should have the other replica of this row down when it was inserted? On Wed, Nov 10, 2010 at 6:08 AM, Eric van Orsouw wrote: > > Hello, > > > > We have a cluster of 4 nodes (0.6.6) and use the random partitioner and a > replication of 2. > > When I insert a number of rows I can always retrieve them by their explicit > id (get_range_slices(“”,””, 1). > > Playing with consistency levels and temporarily shutting down a Cassandra > node all yields the expected result. > > > > However when I use get_range_slices(“”,””, n) to iterate over all rows, I > sometimes don’t get anything (depending on the node). > > > > I then reduced the problem to inserting just a single row. > > Specifically, the ‘iteration’ only seems to succeed when I issue the request > to the node that contains the first copy. > > I Discovered that when I iterate using a consistency level of Quorum/All the > iteration always succeeds and I properly get the one row. > > > > So a solution would be to always use consistency level One/All but that has a > performance penalty. > > > > Can anyone explain why iterating using get_range_slices(“”,””,n) does not > always function with consistency level One on all nodes? > > > > Thanks, > > Eric > > > > P.S. To rule out any discussion on whether or not to use iteration in the > first place, we only plan to use it for backup and periodic cleanup cycles. -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com
