[jira] [Comment Edited] (SPARK-26783) Kafka parameter documentation doesn't match with the reality (upper/lowercase)
[
https://issues.apache.org/jira/browse/SPARK-26783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16779408#comment-16779408
]
Gabor Somogyi edited comment on SPARK-26783 at 2/27/19 2:56 PM:
I've gone through the whole chain on v2.4.0 and it looks good (the batch part
only because the original issue was there).
The only thing what I can assume [~sindiri] has seen is [this
line|https://github.com/apache/spark/blob/b0450d07bd5a77a519a662351ca5b5d562e61a58/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala#L460]
when mentioned the following:
{quote}the variable that needs to be used is "failondataloss" (all in lower
case)
{quote}
I'm closing this issue. Please come back if you have further info or correct me
if my assumption is wrong.
was (Author: gsomogyi):
I've gone through the whole chain on v2.4.0 and it looks good (the batch part
only because the original issue was there).
The only thing what I can assume [~sindiri] has seen is [this
line|https://github.com/apache/spark/blob/b0450d07bd5a77a519a662351ca5b5d562e61a58/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala#L460]
when mentioned the following:
{quote}the variable that needs to be used is "failondataloss" (all in lower
case)
{quote}
I'm closing this issue. Please come back and correct me if you have further
info or my assumption is wrong.
> Kafka parameter documentation doesn't match with the reality (upper/lowercase)
> --
>
> Key: SPARK-26783
> URL: https://issues.apache.org/jira/browse/SPARK-26783
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
>Affects Versions: 3.0.0
>Reporter: Gabor Somogyi
>Priority: Minor
>
> A good example for this is "failOnDataLoss" which is reported in SPARK-23685.
> I've just checked and there are several other parameters which suffer from
> the same issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-26783) Kafka parameter documentation doesn't match with the reality (upper/lowercase)
[
https://issues.apache.org/jira/browse/SPARK-26783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16757842#comment-16757842
]
Jungtaek Lim edited comment on SPARK-26783 at 2/1/19 1:01 AM:
--
I'm not sure about what you're referring to here from SPARK-23685:
{quote}
Originally this pr was created as "failOnDataLoss" doesn't have any impact when
set in structured streaming. But found out that ,the variable that needs to be
used is "failondataloss" (all in lower case).
{quote}
I just played with test "failOnDataLoss=false should not return duplicated
records: v1" in KafkaDontFailOnDataLossSuite, and looks like it works as
intended (as case-insensitive manner).
"failOnDataLoss" -> "false" // passed
"failondataloss" -> "false" // passed
"FAILONDATALOSS" -> "false" // passed
// failed
"failOnDataLoss" -> "true" // failed
was (Author: kabhwan):
I'm not sure about what [~sindiri] left a comment on the PR:
{quote}
Originally this pr was created as "failOnDataLoss" doesn't have any impact when
set in structured streaming. But found out that ,the variable that needs to be
used is "failondataloss" (all in lower case).
{quote}
I just played with test "failOnDataLoss=false should not return duplicated
records: v1" in KafkaDontFailOnDataLossSuite, and looks like it works as
intended (as case-insensitive manner).
"failOnDataLoss" -> "false" // passed
"failondataloss" -> "false" // passed
"FAILONDATALOSS" -> "false" // passed
// failed
"failOnDataLoss" -> "true" // failed
> Kafka parameter documentation doesn't match with the reality (upper/lowercase)
> --
>
> Key: SPARK-26783
> URL: https://issues.apache.org/jira/browse/SPARK-26783
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
>Affects Versions: 3.0.0
>Reporter: Gabor Somogyi
>Priority: Minor
>
> A good example for this is "failOnDataLoss" which is reported in SPARK-23685.
> I've just checked and there are several other parameters which suffer from
> the same issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-26783) Kafka parameter documentation doesn't match with the reality (upper/lowercase)
[ https://issues.apache.org/jira/browse/SPARK-26783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756994#comment-16756994 ] Gabor Somogyi edited comment on SPARK-26783 at 1/31/19 8:14 AM: cc [~joseph.torres] [~kabhwan] [~LI,Xiao] AFAIK you're leading DSv2, can you make judgement in this area? Namely now sources receiving lowercase parameters but: * APIs sometimes doesn't state that or state case-insensitive only * Documentation writes parameters camel case So the question is: Are the parameters accidentally or officially lowercase? It's required heavily by documentation. was (Author: gsomogyi): cc [~joseph.torres] [~kabhwan] [~LI,Xiao] AFAIK you're leading DSv2, can you make judgement in this area? Namely now sources receiving lowercase parameters but: * APIs sometimes doesn't state that or state case-insensitive only * Documentation writes parameters camel case So the question is: Are the parameters accidentally or officially lowercase? It's required heavily by documentation. > Kafka parameter documentation doesn't match with the reality (upper/lowercase) > -- > > Key: SPARK-26783 > URL: https://issues.apache.org/jira/browse/SPARK-26783 > Project: Spark > Issue Type: Bug > Components: Structured Streaming >Affects Versions: 3.0.0 >Reporter: Gabor Somogyi >Priority: Minor > > A good example for this is "failOnDataLoss" which is reported in SPARK-23685. > I've just checked and there are several other parameters which suffer from > the same issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
