[
https://issues.apache.org/jira/browse/SPARK-26783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16757842#comment-16757842
]
Jungtaek Lim edited comment on SPARK-26783 at 2/1/19 1:01 AM:
--------------------------------------------------------------
I'm not sure about what you're referring to here from SPARK-23685:
{quote}
Originally this pr was created as "failOnDataLoss" doesn't have any impact when
set in structured streaming. But found out that ,the variable that needs to be
used is "failondataloss" (all in lower case).
{quote}
I just played with test "failOnDataLoss=false should not return duplicated
records: v1" in KafkaDontFailOnDataLossSuite, and looks like it works as
intended (as case-insensitive manner).
"failOnDataLoss" -> "false" // passed
"failondataloss" -> "false" // passed
"FAILONDATALOSS" -> "false" // passed
<default value> // failed
"failOnDataLoss" -> "true" // failed
was (Author: kabhwan):
I'm not sure about what [~sindiri] left a comment on the PR:
{quote}
Originally this pr was created as "failOnDataLoss" doesn't have any impact when
set in structured streaming. But found out that ,the variable that needs to be
used is "failondataloss" (all in lower case).
{quote}
I just played with test "failOnDataLoss=false should not return duplicated
records: v1" in KafkaDontFailOnDataLossSuite, and looks like it works as
intended (as case-insensitive manner).
"failOnDataLoss" -> "false" // passed
"failondataloss" -> "false" // passed
"FAILONDATALOSS" -> "false" // passed
<default value> // failed
"failOnDataLoss" -> "true" // failed
> Kafka parameter documentation doesn't match with the reality (upper/lowercase)
> ------------------------------------------------------------------------------
>
> Key: SPARK-26783
> URL: https://issues.apache.org/jira/browse/SPARK-26783
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 3.0.0
> Reporter: Gabor Somogyi
> Priority: Minor
>
> A good example for this is "failOnDataLoss" which is reported in SPARK-23685.
> I've just checked and there are several other parameters which suffer from
> the same issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
