Re: Basic question on a write operation immediately followed by a read

[Roshan Dawrani]

2011/1/25 Wangpei (Peter) 

>  for your 1-node cluster,  ANY is the only consistency level that client
> may returns BEFORE node write to memory table.
>
> And read op on the node read both the memory table and SSTable.
>
>
>
> It real puzzle me. :(
>

Please don't be puzzled just yet. :-)

As I said from the beginning, I wasn't confirming yet that reads were "in
fact" missing the writes. I have just observed that kind of behavior at my
app level and I wanted to understand what was the possibility of it
happening from the Cassdandra side.

If reads were sure to read what was written (with QUORAM level, let's say),
then I can look at other causes inside the app.

Re: Basic question on a write operation immediately followed by a read

[Wangpei (Peter)]

for your 1-node cluster,  ANY is the only consistency level that client may 
returns BEFORE node write to memory table.
And read op on the node read both the memory table and SSTable.

It real puzzle me. :(

发件人: Roshan Dawrani [mailto:roshandawr...@gmail.com]
发送时间: 2011年1月25日 15:47
收件人: user@cassandra.apache.org; hector-us...@googlegroups.com
主题: Re: Basic question on a write operation immediately followed by a read

2011/1/25 Wangpei (Peter) 
mailto:peter.wang...@huawei.com>>
What is the ConsistencyLevel value? Is it ConsistencyLevel.ANY?

I am using Hector 0.7.0-22 and getting keyspace as HFactory.createKeyspace(), 
which seems to be defaulting the consistency level to QUORAM for both reads and 
writes.

Nowhere else, it is explicitly specified in my configuration.

Re: Basic question on a write operation immediately followed by a read

[Roshan Dawrani]

2011/1/25 Wangpei (Peter) 

>  What is the ConsistencyLevel value? Is it ConsistencyLevel.ANY?
>
>
I am using Hector 0.7.0-22 and getting keyspace as *
HFactory.createKeyspace()*, which seems to be defaulting the consistency
level to QUORAM for both reads and writes.

Nowhere else, it is explicitly specified in my configuration.

Re: Basic question on a write operation immediately followed by a read

[Oleg Anastasyev]

> Is there a possibility that my read operation may miss the data that just got
inserted?

If write operation did not resulted in exception and there was no other clients
writing to the same row/column concurrently - you will read exactly what you
just written.
> 
> Since there are no DB transactions in Cassandra, are writes immediately seen
to readers - even partially as they get written? 

Writes are atomic on column level. So concurrently reading client could not read
partially written column, but, in case of multicolumn batch insert, concurrent
readers may see not all columns of batch written.

> 
> Or can there be a delay sometimes due to flusing-to-SSTables, etc? 

No.

> 
> Or, the writes are first in-memory and immediately visible to readers and
flusing, etc is independent of all this and happens in background?

Exactly. All writes are first go to memtable, which is in-memory structure.
Reads of just written data are served from memory. All flushing, compactions,
repair, cluster management tasks are served by background threads.

Re: Basic question on a write operation immediately followed by a read

[Wangpei (Peter)]

What is the ConsistencyLevel value? Is it ConsistencyLevel.ANY?

Javadoc:
* Write consistency levels make the following guarantees before reporting 
success to the client:
*   ANY  Ensure that the write has been written once somewhere, 
including possibly being hinted in a non-target node.
*   ONE  Ensure that the write has been written to at least 1 node's 
commit log and memory table
*   QUORUM   Ensure that the write has been written to  
/ 2 + 1 nodes
*   LOCAL_QUORUM Ensure that the write has been written to  
/ 2 + 1 nodes, within the local datacenter (requires NetworkTopologyStrategy)
*   EACH_QUORUM  Ensure that the write has been written to  
/ 2 + 1 nodes in each datacenter (requires NetworkTopologyStrategy)
*   ALL  Ensure that the write is written to 
 nodes before responding to the client.



发件人: Roshan Dawrani [mailto:roshandawr...@gmail.com]
发送时间: 2011年1月25日 10:57
收件人: user@cassandra.apache.org; hector-us...@googlegroups.com
主题: Basic question on a write operation immediately followed by a read

Hi,

I have a basic question - maybe silly too.

Say, I have a 1-node Cassandra setup (no replication, eventual consistency, 
etc) and I do an insert into a column family and then very close in time to the 
insert, I do a read on it for the same data.

Is there a possibility that my read operation may miss the data that just got 
inserted?

Since there are no DB transactions in Cassandra, are writes immediately seen to 
readers - even partially as they get written?

Or can there be a delay sometimes due to flusing-to-SSTables, etc?

Or, the writes are first in-memory and immediately visible to readers and 
flusing, etc is independent of all this and happens in background?
Thanks.

--
Roshan
Blog: http://roshandawrani.wordpress.com/
Twitter: @roshandawrani
Skype: roshandawrani

Re: Basic question on a write operation immediately followed by a read

[Roshan Dawrani]

2011/1/25 Patricio Echagüe 

> Roshan, when a client invoke a write, the write goes first to commit log
> and later to memtable. After that it returns to the client.
>
> After it reaches the memtable, that data is ready to be read.
>
> The reads consolidates de data from the memtables and sstables unless there
> is a hit in the row cache.
>
> does it help?
>

Yes, it helps very much. Because I was unsure when a client call returned.
If it returns after data is pushed to memtable, then it all seems safe at
that front.

Thanks a lot.

Re: Basic question on a write operation immediately followed by a read

[Patricio Echagüe]

Roshan, when a client invoke a write, the write goes first to commit log and
later to memtable. After that it returns to the client.

After it reaches the memtable, that data is ready to be read.

The reads consolidates de data from the memtables and sstables unless there
is a hit in the row cache.

does it help?

On Mon, Jan 24, 2011 at 8:57 PM, Roshan Dawrani wrote:

> Thanks for your inputs, Victor.
>
> In my app, it's a bit event driven.
>
> We do writes and fire events and listeners then read - so we can't predict
> how soon the reads will come. Sometimes they came too fast, which is better
> for our app, if we can have a Cassandra DB level understanding that they
> won't miss the writes, if they come too fast :-)
>
> Or, if there is anything we should do to make sure that reads happen in an
> assured manner.
>
>
> On Tue, Jan 25, 2011 at 10:22 AM, Victor Kabdebon <
> victor.kabde...@gmail.com> wrote:
>
>> Again don't take it as a 100% sure answer because it is not an area that I
>> have really explored.
>>
>> So yes I think that reads are made from Memtables.
>> Theoretically yes, however it is very unlikely : your get must be build
>> and send before the commitlog updates the Memtable (which is like inserting
>> in a Memtable, a matter of microseconds).
>> Possible configuration : Just wait a second or so to do your "get" query
>> (I imagine that it works after one second because you don't report this
>> problem)... Other than that no not really. I have not done any application
>> with those time constraints.
>>
>> Best regards,
>> Victor Kabdebon
>>
>> 2011/1/24 Roshan Dawrani 
>>
>>> On Tue, Jan 25, 2011 at 9:57 AM, Victor Kabdebon <
>>> victor.kabde...@gmail.com> wrote:
>>>
>>>  As far as I remember, please correct me if I am wrong, on a one node
 cluster :
 First Commitlog is updated then almost immediatly after order is send to
 the memtable to add this new insert. You might have a very short delay
 between the two. I don't know your configuration but especially if you
 insert from a distant server to a node you should look at their sync..
 Otherwise if it doesn't appear I can offer no explanation for this
 behavior...

>>>
>>> As of now, I am on an app server with an embedded cassandra server, so no
>>> possibility of clocks out-of-sync.
>>>
>>> So, I understand from you that client call returns after updating the
>>> commit log and updates to memtables are async after that - with
>>> how-much-ever short a delay tried by Cassandra?
>>>
>>> And the reads are always off memtables?
>>>
>>> So, theoretically, there is a possibility of a read missing a write
>>> because it has not come to memtables from the commit log yet?
>>>
>>> Is there anything that I can tell about my configuration that would help?
>>>
>>
>>
>

Re: Basic question on a write operation immediately followed by a read

[Roshan Dawrani]

Thanks for your inputs, Victor.

In my app, it's a bit event driven.

We do writes and fire events and listeners then read - so we can't predict
how soon the reads will come. Sometimes they came too fast, which is better
for our app, if we can have a Cassandra DB level understanding that they
won't miss the writes, if they come too fast :-)

Or, if there is anything we should do to make sure that reads happen in an
assured manner.

On Tue, Jan 25, 2011 at 10:22 AM, Victor Kabdebon  wrote:

> Again don't take it as a 100% sure answer because it is not an area that I
> have really explored.
>
> So yes I think that reads are made from Memtables.
> Theoretically yes, however it is very unlikely : your get must be build and
> send before the commitlog updates the Memtable (which is like inserting in a
> Memtable, a matter of microseconds).
> Possible configuration : Just wait a second or so to do your "get" query (I
> imagine that it works after one second because you don't report this
> problem)... Other than that no not really. I have not done any application
> with those time constraints.
>
> Best regards,
> Victor Kabdebon
>
> 2011/1/24 Roshan Dawrani 
>
>> On Tue, Jan 25, 2011 at 9:57 AM, Victor Kabdebon <
>> victor.kabde...@gmail.com> wrote:
>>
>>  As far as I remember, please correct me if I am wrong, on a one node
>>> cluster :
>>> First Commitlog is updated then almost immediatly after order is send to
>>> the memtable to add this new insert. You might have a very short delay
>>> between the two. I don't know your configuration but especially if you
>>> insert from a distant server to a node you should look at their sync..
>>> Otherwise if it doesn't appear I can offer no explanation for this
>>> behavior...
>>>
>>
>> As of now, I am on an app server with an embedded cassandra server, so no
>> possibility of clocks out-of-sync.
>>
>> So, I understand from you that client call returns after updating the
>> commit log and updates to memtables are async after that - with
>> how-much-ever short a delay tried by Cassandra?
>>
>> And the reads are always off memtables?
>>
>> So, theoretically, there is a possibility of a read missing a write
>> because it has not come to memtables from the commit log yet?
>>
>> Is there anything that I can tell about my configuration that would help?
>>
>
>

Re: Basic question on a write operation immediately followed by a read

[Roshan Dawrani]

On Tue, Jan 25, 2011 at 9:57 AM, Victor Kabdebon
wrote:

> As far as I remember, please correct me if I am wrong, on a one node
> cluster :
> First Commitlog is updated then almost immediatly after order is send to
> the memtable to add this new insert. You might have a very short delay
> between the two. I don't know your configuration but especially if you
> insert from a distant server to a node you should look at their sync..
> Otherwise if it doesn't appear I can offer no explanation for this
> behavior...
>

As of now, I am on an app server with an embedded cassandra server, so no
possibility of clocks out-of-sync.

So, I understand from you that client call returns after updating the commit
log and updates to memtables are async after that - with how-much-ever short
a delay tried by Cassandra?

And the reads are always off memtables?

So, theoretically, there is a possibility of a read missing a write because
it has not come to memtables from the commit log yet?

Is there anything that I can tell about my configuration that would help?

Re: Basic question on a write operation immediately followed by a read

[Roshan Dawrani]

2011/1/25 Patricio Echagüe 

> Roshan, for specifics have a look at
> http://wiki.apache.org/cassandra/ArchitectureOverview specially where it
> says "Write path".


I went through the "Write path" section and have a small follow-up question,
please - again assuming a 1-node setup for now.

How much of a "write" is done before a client mutation call is returned?

a) Does it only write to the commit log and come back? And even updates to
memtables happen in the background after that?

b) Does it come back after writing to the commit log and updates to
memtables both have happened?

I am asking because if it is a), then is it is possible that a read
operation happens after commit log has been written to, but memtables are
not updated yet? (I assume reads are off memtables and not take into account
what is in commit logs)

Basic question again is whether in "any" scenario, a read might miss a write
that has just happened.

Thanks.

Re: Basic question on a write operation immediately followed by a read

[Roshan Dawrani]

On Tue, Jan 25, 2011 at 9:32 AM, Victor Kabdebon
wrote:

> Roshan, just remember, what you do on a one node cluster might not be valid
> on a 5 node cluster. Depending on the way your insert and query (QUORUM,
> ALL, ... ) your data might not be available to get, yet it will be in your
> cluster. I advise you to read Patricio's link it will give you some hints.
>

Yes, I understand and expect the differences between a 1-node and a n-node
setup.

It's just that, at this point, I am investigating a few issues in a 1-node
setup and if reads can follow the writes with guarantee, then I can
comfortably look for a cause elsewhere.

Thanks.

Re: Basic question on a write operation immediately followed by a read

[Roshan Dawrani]

Thanks, Victor.

Even in our application, it will be an advantage if the data can be
immediately read and reads don't miss it for even the smallest amount of
time after write queries have been executed at the app level.

I am in the middle of migrating an app to Cassandra and observing some small
app level issues, and wanted to rule out this one as a possibility.

Cheers,
Roshan

On Tue, Jan 25, 2011 at 9:23 AM, Victor Kabdebon
wrote:

> Hello,
>
> Roshan basically what happens to your one node server is :
> Your insert operation is recorded in the commitlog, then to the MemTable of
> the ColumnFamily. Flushing to Memtable is triggered after and your flush
> those memtables.
> The Memtable holds the data in your node memory and so your insert will be
> in either one of those structures. It can be immediately read. And if I can
> add this is one of the advantage of Cassandra : the data is inserted and
> "ready to use" immediately after it has been received by the node.
>
> Best Regards,
> Victor Kabdebon
> http://www.voxnucleus.fr
>
> 2011/1/24 Roshan Dawrani 
>
> Hi,
>>
>> I have a basic question - maybe silly too.
>>
>> Say, I have a 1-node Cassandra setup (no replication, eventual
>> consistency, etc) and I do an insert into a column family and then very
>> close in time to the insert, I do a read on it for the same data.
>>
>> Is there a possibility that my read operation may miss the data that just
>> got inserted?
>>
>> Since there are no DB transactions in Cassandra, are writes immediately
>> seen to readers - even partially as they get written?
>>
>> Or can there be a delay sometimes due to flusing-to-SSTables, etc?
>>
>> Or, the writes are first in-memory and immediately visible to readers and
>> flusing, etc is independent of all this and happens in background?
>>
>> Thanks.
>>
>> --
>> Roshan
>> Blog: http://roshandawrani.wordpress.com/
>> Twitter: @roshandawrani 
>> Skype: roshandawrani
>>
>