For now I'm giving up, but I'll have to "refresh" this thread in future ;-)
The last thing I found out is that entry that I marked in previous mail
as "LAST VALID KEY/VALUE PAIR" is the problem - it is fine itself, but
it "breaks" the stream somehow. Removing it fixes the problem, but I
still
Hmmm... In general it seems that for some reason Cassandra reads invalid
value when trying to get key length (it should be ~100-150, but it gets
2048), then basing on this value it reads too much data and when trying
to read next key's length again it reads some garbage translating it to
a garg
Sounds like a nasty heisenbug, can you replace or rebuild the machine?
Heisenbug :D
(never heard this name before :-) )
I thought so too, but I finally managed to reproduce it locally (it
requires 3 nodes, one of them needs to have a specific token assigned),
the rest just have to be present
> Any other ideas?
Sounds like a nasty heisenbug, can you replace or rebuild the machine?
Cheers
-
Aaron Morton
Freelance Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 21/05/2013, at 9:36 PM, Michal Michalski wrote:
> I've finally had some time
I've finally had some time to experiment a bit with this problem (it
occured twice again) and here's what I found:
1. So far (three occurences in total), *when* it happened, it happened
only for streaming to *one* specific C* node (but it works on this node
too for 99,9% of the time)
2. It ha
Strange things happen.
It wasn't a single row, but one single "part" file of the Hadoop's input
that failed - we didn't manage to find a specific row that causes the
problem. However, it keeps failing only on production, where we can't
experiment with it a lot. We tried to reproduce it in a fe
Thanks for reply, Aaron. Unluckily, I think it's not the case - we did
some quick tests last week and for now it _seems_ that:
1) There was no empty / zero-lenght key in data we loaded - that was the
first thing we checked
2) By "bisecting" the data, we found out that the row that makes the
pr
> but yesterday one of 600 mappers failed
>
:)
> From what I can understand by looking into the C* source, it seems to me that
> the problem is caused by a empty (or surprisingly finished?) input buffer (?)
> causing token to be set to -1 which is improper for RandomPartitioner:
Yes, there is
We're streaming data to Cassandra directly from MapReduce job using
BulkOutputFormat. It's been working for more than a year without any
problems, but yesterday one of 600 mappers faild and we got a
strange-looking exception on one of the C* nodes.
IMPORTANT: It happens on one node and on one
