+1
On 26.05.2017 18:36, Damian Guy wrote:
In that case, though, every access to that key is doomed to failure as the
database is corrupted. So i think it should probably die in a steaming heap
at that point!
On Fri, 26 May 2017 at 17:33 Eno Thereska <eno.there...@gmail.com> wrote:
Hi Damian,
I was thinking of cases when there is bit-rot on the storage itself and we
get a malformed record that cannot be de-serialized. There is an
interesting intersection here with CRCs in both Kafka (already there, they
throw on deserialization) and potentially local storage (we don't have CRCs
here on the data files, though RocksDB has them on its write-ahead log
records).
Basically in a nutshell, I'm saying that every deserialization exception
should go through this new path. The user can decide to fail or continue.
We could start with just poison pills from Kafka though and punt the
storage one to later.
Eno
On 26 May 2017, at 16:59, Damian Guy <damian....@gmail.com> wrote:
Eno,
Under what circumstances would you get a deserialization exception from
the
state store? I can only think of the case where someone has provided a
bad
deserializer to a method that creates a state store. In which case it
would
be a user error and probably should just abort?
Thanks,
Damian
On Fri, 26 May 2017 at 16:32 Eno Thereska <eno.there...@gmail.com>
wrote:
See latest reply to Jan's note. I think I unnecessarily broadened the
scope of this KIP to the point where it sounded like it handles all
sorts
of exceptions. The scope should be strictly limited to "poison pill"
records for now. Will update KIP,
Thanks
Eno
On 26 May 2017, at 16:16, Matthias J. Sax <matth...@confluent.io>
wrote:
"bad" for this case would mean, that we got an
`DeserializationException`. I am not sure if any other processing error
should be covered?
@Eno: this raises one one question. Might it be better to allow for two
handlers instead of one? One for deserialization exception and one for
all other exceptions from user code?
Just a thought.
-Matthias
On 5/26/17 7:49 AM, Jim Jagielski wrote:
On May 26, 2017, at 5:13 AM, Eno Thereska <eno.there...@gmail.com>
wrote:
With regard to `DeserializationException`, do you thing it might
make
sense to have a "dead letter queue" as a feature to provide
out-of-the-box?
We could provide a special topic where bad messages go to, and then
we'd have to add a config option for the user to provide a topic. Is
that
what you're thinking?
For various definitions of "bad"??
