Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16330
yea log file should be fine to put in temp dir.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/16330
Right, I was thinking of adding note into the API doc and we could add a
log too - to say when you call `sparkR.session()` it will create a metastore
and a warehouse location.
I think
Github user shivaram commented on the issue:
https://github.com/apache/spark/pull/16330
Its a bit tricky to ask users permission during installation (actually I'm
not sure how we can create such an option ?) -- I think a viable option could
be to do add `logWarning` that shows where
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16330
yea that's a good point, if we use temp dir by default, then Spark may lose
data without notice. So I'm not sure if we really want to do this in Spark R,
maybe we can ask users to allow Spark R
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/16330
hmm, this is actually an excellent point - this is more than metadata, it
could be data files for warehouse and so on. @gatorsmile @cloud-fan @shivaram
I think in this case all of these are
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16330
> when running as a R package when the guideline is not to have anything
written to user's space (unless under tempdir)
@cloud-fan I think what @felixcheung wants is to create both
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/16330
That would be awesome. One thing to check though, I'm wondering if the
reason for using a predictable location (current directory) is so that when
starting another job it would re-use the
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16330
Seems it's more reasonable to always use temp location for derby and
warehouse, not only Spark R. By doing this, no extra configs are needed and we
don't to touch Spark R. What do you think?
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/16330
thank you thank you @gatorsmile I'll update this.
As for the meatstore client, since by default it goes to tempdir
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16330
In Hive metastore execution client, we face the same issue. See the
[code](https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala#L379-L383)
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16330
The extra flag is needed only when using Hive metastore. How about renaming
the flag to `spark.sql.hive.metastore.default.derby.dir`? The value is unable
to be changed at runtime. Thus, the best
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/16330
Hi @gatorsmile @cloud-fan @yhuai could you give us some feedback on this?
This is blocking release of Spark to R/CRAN - we don't need a very detailed
review - just your thought of whether
Github user shivaram commented on the issue:
https://github.com/apache/spark/pull/16330
@yhuai Could you take another look at this ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16330
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16330
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70435/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16330
**[Test build #70435 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70435/testReport)**
for PR 16330 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16330
**[Test build #70435 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70435/testReport)**
for PR 16330 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16330
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70337/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16330
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16330
**[Test build #70337 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70337/testReport)**
for PR 16330 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16330
**[Test build #70337 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70337/testReport)**
for PR 16330 at commit
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/16330
that's weird, I'm seeing a lot of seemingly unrelated flaky test failure
lately?
```
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/16330
jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16330
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16330
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70332/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16330
**[Test build #70332 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70332/testReport)**
for PR 16330 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16330
**[Test build #70332 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70332/testReport)**
for PR 16330 at commit
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/16330
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16330
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16330
**[Test build #70324 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70324/testReport)**
for PR 16330 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16330
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70324/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16330
**[Test build #70324 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70324/testReport)**
for PR 16330 at commit
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/16330
@shivaram
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if
33 matches
Mail list logo
