Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/14022
Merging to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14022
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62637/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14022
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14022
**[Test build #62637 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62637/consoleFull)**
for PR 14022 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14022
**[Test build #62637 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62637/consoleFull)**
for PR 14022 at commit
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/14022
I'll push this later after another test run. retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14022
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62494/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14022
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14022
**[Test build #62494 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62494/consoleFull)**
for PR 14022 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14022
**[Test build #62489 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62489/consoleFull)**
for PR 14022 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14022
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14022
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62489/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14022
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14022
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62488/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14022
**[Test build #62488 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62488/consoleFull)**
for PR 14022 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14022
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62485/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14022
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14022
**[Test build #62485 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62485/consoleFull)**
for PR 14022 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14022
**[Test build #62494 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62494/consoleFull)**
for PR 14022 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14022
**[Test build #62489 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62489/consoleFull)**
for PR 14022 at commit
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/14022
> I really think you should keep the "spark." requirement.
I reverted my last change since I forgot to change the test code anyway.
I'll spend some time to see if it's easy to
Github user ericl commented on the issue:
https://github.com/apache/spark/pull/14022
It looks good to me, except I really think you should keep the "spark."
requirement. It seems plausible that some reason a string like '${x}' can show
up in spark confs, perhaps things like
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14022
**[Test build #62488 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62488/consoleFull)**
for PR 14022 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14022
**[Test build #62485 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62485/consoleFull)**
for PR 14022 at commit
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/14022
> if someone adds a spark conf that doesn't start with "spark.", which I
think is allowed by spark conf.
Well, it is and it isn't. You can't set non-`spark.` configs from
spark-submit or
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14022
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62339/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14022
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14022
**[Test build #62339 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62339/consoleFull)**
for PR 14022 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14022
**[Test build #62339 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62339/consoleFull)**
for PR 14022 at commit
Github user ericl commented on the issue:
https://github.com/apache/spark/pull/14022
IMO `${sparkconf:spark.master}` is more clear to the unfamiliar reader, but
it also seems ok to go with `${spark.master}`. Thought there might also be an
issue if someone adds a spark conf that
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/14022
That's mostly how it works; I would like to avoid an explicit `sparkconf:`
prefix to avoid things like `sparkconf:spark.master`, but I can enforce that
only variables starting with `spark.` are
Github user ericl commented on the issue:
https://github.com/apache/spark/pull/14022
To reduce the risk, how about changing the semantics to
```
* - spark/sparkconf/hiveconf: looks for the value in the spark config
* - system: looks for the value in the system
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/14022
> Wouldn't enabling it by default break backwards compatibility?
Yes, maybe. But having a flag to disable everything would also potentially
break features that rely on it... although you
Github user ericl commented on the issue:
https://github.com/apache/spark/pull/14022
Wouldn't enabling it by default break backwards compatibility? I agree that
would be better, but it seems likely that '${...}' may be used in existing
configs.
---
If your project is set up for it,
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/14022
> how about a global flag for enabling config expansion
I think that would be more confusing. Why would someone disable expansion?
My only concern with enabling it for all options is
Github user ericl commented on the issue:
https://github.com/apache/spark/pull/14022
Instead of selectively enabling this for certain confs / config builders,
how about a global flag for enabling config expansion? I think that would be
less likely to be confusing. Also, perhaps a
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/14022
I'll leave this around a little more but if I don't see more feedback I
plan to push it before the weekend.
---
If your project is set up for it, you can reply to this email and have your
reply
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14022
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62275/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14022
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14022
**[Test build #62275 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62275/consoleFull)**
for PR 14022 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14022
**[Test build #62275 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62275/consoleFull)**
for PR 14022 at commit
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/14022
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14022
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62259/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14022
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14022
**[Test build #62259 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62259/consoleFull)**
for PR 14022 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14022
**[Test build #62259 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62259/consoleFull)**
for PR 14022 at commit
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/14022
Any more comments? I'd like to get this is so I can work on cleaning up the
code duplication between SQL and core.
---
If your project is set up for it, you can reply to this email and have your
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14022
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61924/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14022
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14022
**[Test build #61924 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61924/consoleFull)**
for PR 14022 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14022
**[Test build #61924 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61924/consoleFull)**
for PR 14022 at commit
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/14022
> Spark SQL allows env:xxx and system:xxx. We should follow the same here.
Sounds good. I looked briefly at the code and they could potentially be
merged later, but to avoid issues like "how
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/14022
Spark SQL allows `env:xxx` and `system:xxx`. We should follow the same here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14022
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14022
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61795/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14022
**[Test build #61795 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61795/consoleFull)**
for PR 14022 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14022
**[Test build #61795 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61795/consoleFull)**
for PR 14022 at commit
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/14022
> Do we have weird corner cases here like circular references?
That's a good question; the current code doesn't handle that (if you
reference a config that itself has references, those are
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/14022
My only concern is the added complexity vs how useful it would be. Do we
have weird corner cases here like circular references? or resolving references
among many sources in the right order leading
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/14022
Randomly pinging a couple of people... @JoshRosen @srowen
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14022
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61643/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14022
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14022
**[Test build #61643 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61643/consoleFull)**
for PR 14022 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14022
**[Test build #61643 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61643/consoleFull)**
for PR 14022 at commit
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/14022
Note this contains the code for #14020 also since I needed that to test the
changes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well.
65 matches
Mail list logo
