[
https://issues.apache.org/jira/browse/SPARK-26806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cheng Lian updated SPARK-26806:
-------------------------------
Description:
Right now, EventTimeStats.merge doesn't handle "zero.merge(zero)". This will
make "avg" become "NaN". And whatever gets merged with the result of
"zero.merge(zero)", "avg" will still be "NaN". Then finally, "NaN".toLong will
return "0" and the user will see the following incorrect report:
{code:java}
"eventTime" : {
"avg" : "1970-01-01T00:00:00.000Z",
"max" : "2019-01-31T12:57:00.000Z",
"min" : "2019-01-30T18:44:04.000Z",
"watermark" : "1970-01-01T00:00:00.000Z"
}
{code}
was:
Right now, EventTimeStats.merge doesn't handle "zero.merge(zero)". This will
make "avg" become "NaN". And whatever gets merged with the result of
"zero.merge(zero)", "avg" will still be "NaN". Then finally, "NaN".toLong will
return "0" and the user will see the following incorrect report:
{code}
"eventTime" : {
"avg" : "1970-01-01T00:00:00.000Z",
"max" : "2019-01-31T12:57:00.000Z",
"min" : "2019-01-30T18:44:04.000Z",
"watermark" : "1970-01-01T00:00:00.000Z"
}
{code}
This issue was reported by [~liancheng]
> EventTimeStats.merge doesn't handle "zero.merge(zero)" correctly
> ----------------------------------------------------------------
>
> Key: SPARK-26806
> URL: https://issues.apache.org/jira/browse/SPARK-26806
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 2.2.1, 2.2.2, 2.2.3, 2.3.0, 2.3.1, 2.3.2, 2.3.3, 2.4.0
> Reporter: Cheng Lian
> Assignee: Shixiong Zhu
> Priority: Major
> Fix For: 2.2.4, 2.3.3, 2.4.1, 3.0.0
>
>
> Right now, EventTimeStats.merge doesn't handle "zero.merge(zero)". This will
> make "avg" become "NaN". And whatever gets merged with the result of
> "zero.merge(zero)", "avg" will still be "NaN". Then finally, "NaN".toLong
> will return "0" and the user will see the following incorrect report:
> {code:java}
> "eventTime" : {
> "avg" : "1970-01-01T00:00:00.000Z",
> "max" : "2019-01-31T12:57:00.000Z",
> "min" : "2019-01-30T18:44:04.000Z",
> "watermark" : "1970-01-01T00:00:00.000Z"
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
