Github user srowen commented on the issue:
https://github.com/apache/spark/pull/22887
@gjhkael can you clarify further what the undesirable behavior is, and what
behavior you are looking for?
---
-
To unsubscribe,
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/22887
> So it's reasonble for users to expect that, if they set hadoop config via
the SQL SET command, it should override the one in spark-defaults.conf.
I agree with that. But the previous
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22887
Spark SQL SET command can't update any static config or Spark core configs,
but I think hadoop configs are different. It's not static as users can update
it via
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/22887
ok, that makes sense as in I understand what you're saying, but not sure
it's what you actually want?
Why shouldn't "set spark.hadoop.*" override spark-defaults.conf?
But, in any
Github user gjhkael commented on the issue:
https://github.com/apache/spark/pull/22887
@vanzin @cloud-fan
The simplest description: user set 'spark.hadoop.xxx' through setCommand
will not cover the same configutation that set in spark-defaults.conf file.
I don't know if that
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/22887
> I think this is what this PR tries to fix?
To be fair I'm not sure I fully understand the PR description. But I know
that the previous patch (which I commented on) broke the functionality
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22887
> Basically, if my "core-size.xml" says
"mapreduce.input.fileinputformat.split.maxsize" is 2, and my Spark conf says
"spark.hadoop.mapreduce.input.fileinputformat.split.maxsize" is 3, then the
Github user gjhkael commented on the issue:
https://github.com/apache/spark/pull/22887
@vanzin Thanks for you review, I add a new commit to let the user's "set"
command take effect. Let me know if you have an easier way. Thanks.
---
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/22887
Sorry, this is a breaking change. It changes the behavior from "I can
currently override any Hadoop configs, even final ones, using spark.hadoop.*"
to "I can never do that".
If there's an
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22887
looks reasonable, cc @gatorsmile
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user cxzl25 commented on the issue:
https://github.com/apache/spark/pull/22887
user set hadoop conf can't overwrite spark-defaults.conf
**SparkHadoopUtil.get.appendS3AndSparkHadoopConfigurations** overwrite the
user-set spark.hadoop with the default configuration
Github user gjhkael commented on the issue:
https://github.com/apache/spark/pull/22887
> can you explain more about why you make the change?
Some hadoop configuration set it in spark-default.conf, we want it to be
global, but in some cases, user need to override the
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/22887
Hi @gjhkael ,
can you explain more about why you make the change?
Did you try `spark.SessionState.newHadoopConf()`?
---
Github user gjhkael commented on the issue:
https://github.com/apache/spark/pull/22887
test this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22887
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22887
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
16 matches
Mail list logo
