Hi Everyone!
We have a 3-node Cassandra cluster (single DC), and a table that get accessed
(for read/writes) by applications running on separate nodes.
Under heavy load, when both instances of the application are attempting to
update the same user entry (E.g. add attribute) we observe that update from
app1 (running on node1 succeeded - i.e. ResultSet#wasApplied returns true).
However, when app2 (on node2 reads the data, it is getting stale data before
app1 updated it).
I'd like to know why this is happening because LTW with serial consistency
should prevent this type of inconsistencies. Any help is much appreciated! We
are using the datastax Java driver.
Thanks!
Thira
Example: (based on application logs)
Initially the user has attribute A with value 1
Both app1 and app2 are adding new attributes; app1 adding B:2 and app2 adding
C:3
app1 read the correct data into memory, added new attribute B and wrote to
Cassandra successfully. The logs show the final attribute list having A:1 and
B:2 tuples.
app2 reads the data, but only see A:1 (the difference in time between the logs
from app1 and app2 is only 2ms; therefore could have happened in any order).
Once app2's write is completed, the end state only has A:1 and C:2, which is
incorrect.
More Information
The (prepared statements that perform) reads and writes to the table have the
following characteristics: - Write is a LWT (on IF EXISTS userid) - Both read
and writes use LOCAL_SERIAL consistency level
The table looks like (simple table with userid as the partition key):
CREATE TABLE my_table(userid ascii PRIMARY KEY,attributes map);
The logic within the app when updating the attributes map is:
User user = dao.getUser(userid);
user.addNewAttribute("key", "value");
dao.update(user);
