Python eh? What's that? Kidding. (Java guy over here...)
I grepped the logs for mutations but only see messages like:
2020-09-14 16:15:19,963 CommitLog.java:149 - Log replay complete, 0
replayed mutations
and
2020-09-17 16:22:13,020 CommitLog.java:149 - Log replay complete, 291708
replayed mutations
Typically, we read very soon after the write, which I thought was a
problem also; however at this point it's been 24+ hours since the data
has been written that I'm now trying to read. Happens very easily.
By determining the partition key, how will that help?
-Joe
On 12/2/2020 12:16 PM, Steve Lacerda wrote:
The digest mismatch typically shows the partition key info, with
something like this:
DecoratedKey(-1671292413668442751, 48343732322d3838353032)
That refers to the partition key, which you can gather like so:
python
import binascii
binascii.unhexlify('48343732322d3838353032')
'H4722-88502'
My assumption is that since you are reading and writing with one, that
some nodes have the data and others don't. Are you seeing any dropped
mutations in the logs? How long after the write are you attempting to
read the same data?
On Wed, Dec 2, 2020 at 9:12 AM Joe Obernberger
<joseph.obernber...@gmail.com <mailto:joseph.obernber...@gmail.com>>
wrote:
Hi Carl - thank you for replying.
I am using Cassandra 3.11.9-1
Rows are not typically being deleted - I assume you're referring
to Tombstones. I don't think that should be the case here as I
don't think we've deleted anything here.
This is a test cluster and some of the machines are small (hence
the one node with 128 tokens and 14.6% - it has a lot less disk
space than the other nodes). This is one of the features that I
really like with Cassandra - being able to size nodes based on
disk/CPU/RAM.
All data is currently written with ONE. All data is read with
ONE. I can replicate this issue at will, so can try different
things easily. I tried changing the read process to use QUORUM
and the issue still takes place. Right now I'm running a 'nodetool
repair' to see if that helps. Our largest table 'doc' has the
following stats:
Table: doc
SSTable count: 28
Space used (live): 113609995010
Space used (total): 113609995010
Space used by snapshots (total): 0
Off heap memory used (total): 225006197
SSTable Compression Ratio: 0.37730474570644196
Number of partitions (estimate): 93641747
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 3712
Local read count: 891065091
Local read latency: NaN ms
Local write count: 7448281135
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 0.0
Bloom filter false positives: 988
Bloom filter false ratio: 0.00001
Bloom filter space used: 151149880
Bloom filter off heap memory used: 151149656
Index summary off heap memory used: 38654701
Compression metadata off heap memory used: 35201840
Compacted partition minimum bytes: 104
Compacted partition maximum bytes: 3379391
Compacted partition mean bytes: 3389
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 8174438
Thoughts/ideas? Thank you!
-Joe
On 12/2/2020 11:49 AM, Carl Mueller wrote:
Why is one of your nodes only at 14.6% ownership? That's weird,
unless you have a small rowcount.
Are you frequently deleting rows? Are you frequently writing rows
at ONE?
What version of cassandra?
On Wed, Dec 2, 2020 at 9:56 AM Joe Obernberger
<joseph.obernber...@gmail.com
<mailto:joseph.obernber...@gmail.com>> wrote:
Hi All - this is my first post here. I've been using
Cassandra for
several months now and am loving it. We are moving from
Apache HBase to
Cassandra for a big data analytics platform.
I'm using java to get rows from Cassandra and very frequently
get a
java.util.NoSuchElementException when iterating through a
ResultSet. If
I retry this query again (often several times), it works.
The debug log
on the Cassandra nodes show this message:
org.apache.cassandra.service.DigestMismatchException:
Mismatch for key
DecoratedKey
My cluster looks like this:
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective)
Host
ID Rack
UN 172.16.100.224 340.5 GiB 512 50.9%
8ba646ac-2b33-49de-a220-ae9842f18806 rack1
UN 172.16.100.208 269.19 GiB 384 40.3%
4e0ba42f-649b-425a-857a-34497eb3036e rack1
UN 172.16.100.225 282.83 GiB 512 50.4%
247f3d70-d13b-4d68-9a53-2ed58e01a63e rack1
UN 172.16.110.3 409.78 GiB 768 63.2%
0abea102-06d2-4309-af36-a3163e8f00d8 rack1
UN 172.16.110.4 330.15 GiB 512 50.6%
2a5ae735-6304-4e99-924b-44d9d5ec86b7 rack1
UN 172.16.100.253 98.88 GiB 128 14.6%
6b528b0b-d7f7-4378-bba8-1857802d4f18 rack1
UN 172.16.100.254 204.5 GiB 256 30.0%
87d0cb48-a57d-460e-bd82-93e6e52e93ea rack1
I suspect this has to do with how I'm using consistency levels?
Typically I'm using ONE. I just set the
dclocal_read_repair_chance to
0.0, but I'm still seeing the issue. Any help/tips?
Thank you!
-Joe Obernberger
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
<mailto:user-unsubscr...@cassandra.apache.org>
For additional commands, e-mail:
user-h...@cassandra.apache.org
<mailto:user-h...@cassandra.apache.org>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.avg.com_email-2Dsignature-3Futm-5Fmedium-3Demail-26utm-5Fsource-3Dlink-26utm-5Fcampaign-3Dsig-2Demail-26utm-5Fcontent-3Demailclient&d=DwMDaQ&c=adz96Xi0w1RHqtPMowiL2g&r=R58SsZ6FLB8iCRFGJzNOH0d2HRPVtaWKKj5fzuMiGlo&m=ILue9yYCY9fLwFNLsm3-mIbyPh6ehPGUPwbWBgqtxe4&s=FmyWQpErRafN779unRB23GMeoiN49uZNIZvPDcD1iVs&e=>
Virus-free. www.avg.com
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.avg.com_email-2Dsignature-3Futm-5Fmedium-3Demail-26utm-5Fsource-3Dlink-26utm-5Fcampaign-3Dsig-2Demail-26utm-5Fcontent-3Demailclient&d=DwMDaQ&c=adz96Xi0w1RHqtPMowiL2g&r=R58SsZ6FLB8iCRFGJzNOH0d2HRPVtaWKKj5fzuMiGlo&m=ILue9yYCY9fLwFNLsm3-mIbyPh6ehPGUPwbWBgqtxe4&s=FmyWQpErRafN779unRB23GMeoiN49uZNIZvPDcD1iVs&e=>
<#m_-330889386023926826_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
--
Steve Lacerda
e. steve.lace...@datastax.com <mailto:steve.lace...@datastax.com>
w. www.datastax.com <http://www.datastax.com>
