[jira] [Updated] (KAFKA-16858) Flatten SMT throws NPE
[
https://issues.apache.org/jira/browse/KAFKA-16858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Strickland updated KAFKA-16858:
Attachment: proto.proto
> Flatten SMT throws NPE
> --
>
> Key: KAFKA-16858
> URL: https://issues.apache.org/jira/browse/KAFKA-16858
> Project: Kafka
> Issue Type: Bug
> Components: connect
>Affects Versions: 3.6.0
> Environment: Kafka 3.6 by way of CP 7.6.0
>Reporter: Adam Strickland
>Priority: Major
> Attachments: FlattenTest.java, proto.proto
>
>
> {{ConnectSchema.expectedClassesFor}} sometimes will throw an NPE as part of a
> call to an SMT chain. Stack trace snippet:
> {{at
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.apply(MomentFlatten.java:84)}}
> {{at
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.applyWithSchema(MomentFlatten.java:173)}}
> {{at
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:280)}}
> {{at
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:280)}}
> {{at
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:280)}}
> {{at
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:274)}}
> {{at org.apache.kafka.connect.data.Struct.put(Struct.java:203)}}
> {{at org.apache.kafka.connect.data.Struct.put(Struct.java:216)}}
> {{at
> org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:255)}}
> {{at
> org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:213)}}
> {{at
> org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:224)}}
> {{at
> org.apache.kafka.connect.data.ConnectSchema.expectedClassesFor(ConnectSchema.java:268)}}
> (the above transform is a sub-class of
> {{{}o.a.k.connect.transforms.Flatten{}}}; have confirmed that the error
> occurs regardless).
> The field being transformed is an array of structs. If the call to
> {{Schema#valueSchema()}} (o.a.k.connect.data.ConnectSchema.java:255) returns
> {{{}null{}}}, the subsequent call to {{Schema#name()}} at
> o.a.k.connect.data.ConnectSchema:268 throws an NPE.
> The strange thing that we have observed is that this doesn't always happen;
> *sometimes* the struct's schema is found and sometimes it is not. We have
> been unable to determine the root cause, but have constructed a test that
> replicates the problem as observed (see attachment).
> In our case we have worked around the issue with the aforementioned sub-class
> of {{{}Flatten{}}}, catching and logging the NPE on that specific use-case.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16858) Flatten SMT throws NPE
[
https://issues.apache.org/jira/browse/KAFKA-16858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850866#comment-17850866
]
Adam Strickland commented on KAFKA-16858:
-
*
{quote}Are you able to provide an anonymized form of your schema directly,
rather than just a high-level "Array of Structs"? I'm wondering if your schema
is capable of triggering the use of the mutable SchemaWrapper
[https://github.com/confluentinc/schema-registry/blob/7b886f309c83041d4f2a5b41b5910f3b8002413a/protobuf-converter/src/main/java/io/confluent/connect/protobuf/ProtobufData.java#L1779]
inside the ProtobufConverter.{quote}
Yes; see [^proto.proto]
*
{quote}w.r.t. the variable validateValue depth: Are you saying that in _error
cases_ the recursion depth is unpredictable, or in general? The validateValue
should be called at every or almost every location in the tree of values, so I
would expect to see lots of different recursion depths. Maybe you can share
some more stacktraces as examples.{quote}
We were only looking at error cases. I specifically recall seeing the same
message break on the 3rd recursion and on the 7th; the array for the message in
question contained 2 Structs.
*
{quote}So far in this investigation, I'm trying to find the source of the null
in hopes that we can prevent it, and get well-formed data to the Flatten SMT.
Regardless of the result of that investigation, I think we can consider this
input malformed, and throw an intentional DataException instead of
NullPointerException. Would that be an acceptable solution for you, or does
this data need to make it all the way through the pipeline?{quote}
Throwing a DataException would be preferable to the NPE. I'm not sure I
understand what you mean by "need to make it all the way through the
pipeline"... As it stands the particular problematic attribute is not something
we care about right now, which is why we can simply squash the Exception.
> Flatten SMT throws NPE
> --
>
> Key: KAFKA-16858
> URL: https://issues.apache.org/jira/browse/KAFKA-16858
> Project: Kafka
> Issue Type: Bug
> Components: connect
>Affects Versions: 3.6.0
> Environment: Kafka 3.6 by way of CP 7.6.0
>Reporter: Adam Strickland
>Priority: Major
> Attachments: FlattenTest.java, proto.proto
>
>
> {{ConnectSchema.expectedClassesFor}} sometimes will throw an NPE as part of a
> call to an SMT chain. Stack trace snippet:
> {{at
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.apply(MomentFlatten.java:84)}}
> {{at
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.applyWithSchema(MomentFlatten.java:173)}}
> {{at
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:280)}}
> {{at
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:280)}}
> {{at
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:280)}}
> {{at
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:274)}}
> {{at org.apache.kafka.connect.data.Struct.put(Struct.java:203)}}
> {{at org.apache.kafka.connect.data.Struct.put(Struct.java:216)}}
> {{at
> org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:255)}}
> {{at
> org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:213)}}
> {{at
> org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:224)}}
> {{at
> org.apache.kafka.connect.data.ConnectSchema.expectedClassesFor(ConnectSchema.java:268)}}
> (the above transform is a sub-class of
> {{{}o.a.k.connect.transforms.Flatten{}}}; have confirmed that the error
> occurs regardless).
> The field being transformed is an array of structs. If the call to
> {{Schema#valueSchema()}} (o.a.k.connect.data.ConnectSchema.java:255) returns
> {{{}null{}}}, the subsequent call to {{Schema#name()}} at
> o.a.k.connect.data.ConnectSchema:268 throws an NPE.
> The strange thing that we have observed is that this doesn't always happen;
> *sometimes* the struct's schema is found and sometimes it is not. We have
> been unable to determine the root cause, but have constructed a test that
> replicates the problem as observed (see attachment).
> In our case we have worked around the issue with the aforementioned sub-class
> of {{{}Flatten{}}}, catching and logging the NPE on that specific use-case.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16858) Flatten SMT throws NPE
[
https://issues.apache.org/jira/browse/KAFKA-16858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850375#comment-17850375
]
Adam Strickland commented on KAFKA-16858:
-
{quote}Do you see this fault with another SMT in the chain ahead of Flatten?
{quote}
No. It consistently shows up only from the Flatten SMT and always on the same
attribute.
{quote}What Converter (sink connector) or source connector implementation are
you using?
{quote}
This is a sink connector, writing to Postgres using
`io.confluent.connect.jdbc.JdbcSinkConnector`. The value converter is
`io.confluent.connect.protobuf.ProtobufConverter`.
Some other data points:
* We use this same SMT in multiple connectors; we see the NPE in only one of
them.
* While it always throws from the same attribute, it doesn't *always* throw...
For instance, the NPE was logged 52 times within the last 24 hours, out of 234
messages processed via that connector.
* The connector we see it on has multiple SMTs; the Flatten is the 4th SMT
applied.
* For reference, the SMTs being applied prior to Flatten are, in order:
## `io.confluent.connect.transforms.Filter$Value`,
## `o.a.k.connect.transforms.ExtravtField$Value`, and
## another `o.a.k.connect.transforms.ExtravtField$Value`
* The attribute it breaks on is an array of `Struct`s; the array can be empty
and the attribute is itself optional. We have seen the break occur in 3 states
(attribute missing, attribute present with empty list, and attribute present
with a list). We have not seen it occur when the attribute is present but the
value is `null`. However, I think that's simply a use case we do not have.
* We have not used the trace logger you linked to, but at one point inserted
our own logging statements at various levels in our custom transform; this was
how we found the faulty attribute. One other thing we have noticed is the stack
recurses all the way down into `ConnectSchema.validateValue` an unexpected
number of times, resolving to `Schema\{STRUCT}` some number of times prior to
returning `null`. The number of times is not consistent and does not correlate
(as far as we can tell) with the number of items in the array or any other
measure we have checked.
> Flatten SMT throws NPE
> --
>
> Key: KAFKA-16858
> URL: https://issues.apache.org/jira/browse/KAFKA-16858
> Project: Kafka
> Issue Type: Bug
> Components: connect
>Affects Versions: 3.6.0
> Environment: Kafka 3.6 by way of CP 7.6.0
>Reporter: Adam Strickland
>Priority: Major
> Attachments: FlattenTest.java
>
>
> {{ConnectSchema.expectedClassesFor}} sometimes will throw an NPE as part of a
> call to an SMT chain. Stack trace snippet:
> {{at
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.apply(MomentFlatten.java:84)}}
> {{at
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.applyWithSchema(MomentFlatten.java:173)}}
> {{at
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:280)}}
> {{at
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:280)}}
> {{at
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:280)}}
> {{at
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:274)}}
> {{at org.apache.kafka.connect.data.Struct.put(Struct.java:203)}}
> {{at org.apache.kafka.connect.data.Struct.put(Struct.java:216)}}
> {{at
> org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:255)}}
> {{at
> org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:213)}}
> {{at
> org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:224)}}
> {{at
> org.apache.kafka.connect.data.ConnectSchema.expectedClassesFor(ConnectSchema.java:268)}}
> (the above transform is a sub-class of
> {{{}o.a.k.connect.transforms.Flatten{}}}; have confirmed that the error
> occurs regardless).
> The field being transformed is an array of structs. If the call to
> {{Schema#valueSchema()}} (o.a.k.connect.data.ConnectSchema.java:255) returns
> {{{}null{}}}, the subsequent call to {{Schema#name()}} at
> o.a.k.connect.data.ConnectSchema:268 throws an NPE.
> The strange thing that we have observed is that this doesn't always happen;
> *sometimes* the struct's schema is found and sometimes it is not. We have
> been unable to determine the root cause, but have constructed a test that
> replicates the problem as observed (see attachment).
> In our case we have worked around the issue with the aforementioned sub-class
> of {{{}Flatten{}}}, catching and logging the NPE on that specific use-case.
--
This message was sent by Atlassian Jira
[jira] [Commented] (KAFKA-16858) Flatten SMT throws NPE
[
https://issues.apache.org/jira/browse/KAFKA-16858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850361#comment-17850361
]
Adam Strickland commented on KAFKA-16858:
-
{quote}It really confused me that the stacktrace is in reverse order, is there
a tool that does that by default?
{quote}
Oh yeah, sorry about that, I guess I've gotten used to it. It's from streaming
the logs to DataDog and not accounting for the linebreaks in Java stacktraces.
One of the many items on the list...
> Flatten SMT throws NPE
> --
>
> Key: KAFKA-16858
> URL: https://issues.apache.org/jira/browse/KAFKA-16858
> Project: Kafka
> Issue Type: Bug
> Components: connect
>Affects Versions: 3.6.0
> Environment: Kafka 3.6 by way of CP 7.6.0
>Reporter: Adam Strickland
>Priority: Major
> Attachments: FlattenTest.java
>
>
> {{ConnectSchema.expectedClassesFor}} sometimes will throw an NPE as part of a
> call to an SMT chain. Stack trace snippet:
> {{at
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.apply(MomentFlatten.java:84)}}
> {{at
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.applyWithSchema(MomentFlatten.java:173)}}
> {{at
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:280)}}
> {{at
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:280)}}
> {{at
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:280)}}
> {{at
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:274)}}
> {{at org.apache.kafka.connect.data.Struct.put(Struct.java:203)}}
> {{at org.apache.kafka.connect.data.Struct.put(Struct.java:216)}}
> {{at
> org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:255)}}
> {{at
> org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:213)}}
> {{at
> org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:224)}}
> {{at
> org.apache.kafka.connect.data.ConnectSchema.expectedClassesFor(ConnectSchema.java:268)}}
> (the above transform is a sub-class of
> {{{}o.a.k.connect.transforms.Flatten{}}}; have confirmed that the error
> occurs regardless).
> The field being transformed is an array of structs. If the call to
> {{Schema#valueSchema()}} (o.a.k.connect.data.ConnectSchema.java:255) returns
> {{{}null{}}}, the subsequent call to {{Schema#name()}} at
> o.a.k.connect.data.ConnectSchema:268 throws an NPE.
> The strange thing that we have observed is that this doesn't always happen;
> *sometimes* the struct's schema is found and sometimes it is not. We have
> been unable to determine the root cause, but have constructed a test that
> replicates the problem as observed (see attachment).
> In our case we have worked around the issue with the aforementioned sub-class
> of {{{}Flatten{}}}, catching and logging the NPE on that specific use-case.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-16858) Flatten SMT throws NPE
Adam Strickland created KAFKA-16858:
---
Summary: Flatten SMT throws NPE
Key: KAFKA-16858
URL: https://issues.apache.org/jira/browse/KAFKA-16858
Project: Kafka
Issue Type: Bug
Components: connect
Affects Versions: 3.6.0
Environment: Kafka 3.6 by way of CP 7.6.0
Reporter: Adam Strickland
Attachments: FlattenTest.java
{{ConnectSchema.expectedClassesFor}} sometimes will throw an NPE as part of a
call to an SMT chain. Stack trace snippet:
{{at
com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.apply(MomentFlatten.java:84)}}
{{at
com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.applyWithSchema(MomentFlatten.java:173)}}
{{at
com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:280)}}
{{at
com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:280)}}
{{at
com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:280)}}
{{at
com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:274)}}
{{at org.apache.kafka.connect.data.Struct.put(Struct.java:203)}}
{{at org.apache.kafka.connect.data.Struct.put(Struct.java:216)}}
{{at
org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:255)}}
{{at
org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:213)}}
{{at
org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:224)}}
{{at
org.apache.kafka.connect.data.ConnectSchema.expectedClassesFor(ConnectSchema.java:268)}}
(the above transform is a sub-class of
{{{}o.a.k.connect.transforms.Flatten{}}}; have confirmed that the error occurs
regardless).
The field being transformed is an array of structs. If the call to
{{Schema#valueSchema()}} (o.a.k.connect.data.ConnectSchema.java:255) returns
{{{}null{}}}, the subsequent call to {{Schema#name()}} at
o.a.k.connect.data.ConnectSchema:268 throws an NPE.
The strange thing that we have observed is that this doesn't always happen;
*sometimes* the struct's schema is found and sometimes it is not. We have been
unable to determine the root cause, but have constructed a test that replicates
the problem as observed (see attachment).
In our case we have worked around the issue with the aforementioned sub-class
of {{{}Flatten{}}}, catching and logging the NPE on that specific use-case.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
