[jira] [Commented] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[
https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931848#comment-15931848
]
Felix Cheung commented on SPARK-18817:
--
we have more discussions on the PR thread and agreed that spark-warehouse,
metastore_db are really user data that we wouldn't want it to go away by
default under tempdir. (these are managed table (ie. with data) and metadata)
we are making change to move derby.log only.
> Ensure nothing is written outside R's tempdir() by default
> --
>
> Key: SPARK-18817
> URL: https://issues.apache.org/jira/browse/SPARK-18817
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
>Reporter: Brendan Dwyer
>Assignee: Felix Cheung
>Priority: Critical
> Fix For: 2.1.1, 2.2.0
>
>
> Per CRAN policies
> https://cran.r-project.org/web/packages/policies.html
> {quote}
> - Packages should not write in the users’ home filespace, nor anywhere else
> on the file system apart from the R session’s temporary directory (or during
> installation in the location pointed to by TMPDIR: and such usage should be
> cleaned up). Installing into the system’s R installation (e.g., scripts to
> its bin directory) is not allowed.
> Limited exceptions may be allowed in interactive sessions if the package
> obtains confirmation from the user.
> - Packages should not modify the global environment (user’s workspace).
> {quote}
> Currently "spark-warehouse" gets created in the working directory when
> sparkR.session() is called.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[
https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15759635#comment-15759635
]
Apache Spark commented on SPARK-18817:
--
User 'felixcheung' has created a pull request for this issue:
https://github.com/apache/spark/pull/16330
> Ensure nothing is written outside R's tempdir() by default
> --
>
> Key: SPARK-18817
> URL: https://issues.apache.org/jira/browse/SPARK-18817
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
>Reporter: Brendan Dwyer
>Priority: Critical
>
> Per CRAN policies
> https://cran.r-project.org/web/packages/policies.html
> {quote}
> - Packages should not write in the users’ home filespace, nor anywhere else
> on the file system apart from the R session’s temporary directory (or during
> installation in the location pointed to by TMPDIR: and such usage should be
> cleaned up). Installing into the system’s R installation (e.g., scripts to
> its bin directory) is not allowed.
> Limited exceptions may be allowed in interactive sessions if the package
> obtains confirmation from the user.
> - Packages should not modify the global environment (user’s workspace).
> {quote}
> Currently "spark-warehouse" gets created in the working directory when
> sparkR.session() is called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[
https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15759368#comment-15759368
]
Felix Cheung commented on SPARK-18817:
--
testing fix, will open a PR shortly.
> Ensure nothing is written outside R's tempdir() by default
> --
>
> Key: SPARK-18817
> URL: https://issues.apache.org/jira/browse/SPARK-18817
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
>Reporter: Brendan Dwyer
>Priority: Critical
>
> Per CRAN policies
> https://cran.r-project.org/web/packages/policies.html
> {quote}
> - Packages should not write in the users’ home filespace, nor anywhere else
> on the file system apart from the R session’s temporary directory (or during
> installation in the location pointed to by TMPDIR: and such usage should be
> cleaned up). Installing into the system’s R installation (e.g., scripts to
> its bin directory) is not allowed.
> Limited exceptions may be allowed in interactive sessions if the package
> obtains confirmation from the user.
> - Packages should not modify the global environment (user’s workspace).
> {quote}
> Currently "spark-warehouse" gets created in the working directory when
> sparkR.session() is called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[
https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15758005#comment-15758005
]
Felix Cheung commented on SPARK-18817:
--
And as a side note, I feel like spark-warehouse, metastore_db, and derby.log
should all be in temporary directory that is cleaned out when the app is done,
much like how Spark Thrift Server is doing currently (at least to metastore_db,
derby.log). Perhaps that is the more correct fix longer term.
> Ensure nothing is written outside R's tempdir() by default
> --
>
> Key: SPARK-18817
> URL: https://issues.apache.org/jira/browse/SPARK-18817
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
>Reporter: Brendan Dwyer
>Priority: Critical
>
> Per CRAN policies
> https://cran.r-project.org/web/packages/policies.html
> {quote}
> - Packages should not write in the users’ home filespace, nor anywhere else
> on the file system apart from the R session’s temporary directory (or during
> installation in the location pointed to by TMPDIR: and such usage should be
> cleaned up). Installing into the system’s R installation (e.g., scripts to
> its bin directory) is not allowed.
> Limited exceptions may be allowed in interactive sessions if the package
> obtains confirmation from the user.
> - Packages should not modify the global environment (user’s workspace).
> {quote}
> Currently "spark-warehouse" gets created in the working directory when
> sparkR.session() is called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[
https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15757998#comment-15757998
]
Felix Cheung commented on SPARK-18817:
--
Aside from changing the existing shipped behavior, there are a few mentions of
this behavior in various documentation that would become wrong and would need
to be updated.
IMO more importantly we still have a feature that can be turned on (as
documented or suggested in documentations) that would cause files to be written
without the user explicitly agreeing to it (or understanding it). This to me
doesn't seem like we would be addressing the root of the issue fully, merely
side-stepping it?
I've managed to track down the fix to move metastore_db and derby.log though.
There are two separate switches to set that it doable from pure R; but I'd
recommend doing in
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala#L116
in order to respect any existing value from hive-site.xml if given one.
How about we introduce something like spark.sql.default.derby.dir and fix this
that way?
> Ensure nothing is written outside R's tempdir() by default
> --
>
> Key: SPARK-18817
> URL: https://issues.apache.org/jira/browse/SPARK-18817
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
>Reporter: Brendan Dwyer
>Priority: Critical
>
> Per CRAN policies
> https://cran.r-project.org/web/packages/policies.html
> {quote}
> - Packages should not write in the users’ home filespace, nor anywhere else
> on the file system apart from the R session’s temporary directory (or during
> installation in the location pointed to by TMPDIR: and such usage should be
> cleaned up). Installing into the system’s R installation (e.g., scripts to
> its bin directory) is not allowed.
> Limited exceptions may be allowed in interactive sessions if the package
> obtains confirmation from the user.
> - Packages should not modify the global environment (user’s workspace).
> {quote}
> Currently "spark-warehouse" gets created in the working directory when
> sparkR.session() is called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[
https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15756403#comment-15756403
]
Shivaram Venkataraman commented on SPARK-18817:
---
[~felixcheung] What do you think are the down sides of disabling Hive by
default ? https://github.com/apache/spark/pull/16290 should fix the warehouse
dir issue and if we disable hive by default then derby.log and metastore_db
should not be created.
> Ensure nothing is written outside R's tempdir() by default
> --
>
> Key: SPARK-18817
> URL: https://issues.apache.org/jira/browse/SPARK-18817
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
>Reporter: Brendan Dwyer
>Priority: Critical
>
> Per CRAN policies
> https://cran.r-project.org/web/packages/policies.html
> {quote}
> - Packages should not write in the users’ home filespace, nor anywhere else
> on the file system apart from the R session’s temporary directory (or during
> installation in the location pointed to by TMPDIR: and such usage should be
> cleaned up). Installing into the system’s R installation (e.g., scripts to
> its bin directory) is not allowed.
> Limited exceptions may be allowed in interactive sessions if the package
> obtains confirmation from the user.
> - Packages should not modify the global environment (user’s workspace).
> {quote}
> Currently "spark-warehouse" gets created in the working directory when
> sparkR.session() is called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[
https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753588#comment-15753588
]
Shivaram Venkataraman commented on SPARK-18817:
---
Just to check - Is your Spark installation built with Hive support (i.e. with
-Phive) ?
> Ensure nothing is written outside R's tempdir() by default
> --
>
> Key: SPARK-18817
> URL: https://issues.apache.org/jira/browse/SPARK-18817
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
>Reporter: Brendan Dwyer
>Priority: Critical
>
> Per CRAN policies
> https://cran.r-project.org/web/packages/policies.html
> {quote}
> - Packages should not write in the users’ home filespace, nor anywhere else
> on the file system apart from the R session’s temporary directory (or during
> installation in the location pointed to by TMPDIR: and such usage should be
> cleaned up). Installing into the system’s R installation (e.g., scripts to
> its bin directory) is not allowed.
> Limited exceptions may be allowed in interactive sessions if the package
> obtains confirmation from the user.
> - Packages should not modify the global environment (user’s workspace).
> {quote}
> Currently "spark-warehouse" gets created in the working directory when
> sparkR.session() is called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[
https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753462#comment-15753462
]
Felix Cheung commented on SPARK-18817:
--
I ran more of this but wasn't seeinng derby.log or metastore_db
> Ensure nothing is written outside R's tempdir() by default
> --
>
> Key: SPARK-18817
> URL: https://issues.apache.org/jira/browse/SPARK-18817
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
>Reporter: Brendan Dwyer
>Priority: Critical
>
> Per CRAN policies
> https://cran.r-project.org/web/packages/policies.html
> {quote}
> - Packages should not write in the users’ home filespace, nor anywhere else
> on the file system apart from the R session’s temporary directory (or during
> installation in the location pointed to by TMPDIR: and such usage should be
> cleaned up). Installing into the system’s R installation (e.g., scripts to
> its bin directory) is not allowed.
> Limited exceptions may be allowed in interactive sessions if the package
> obtains confirmation from the user.
> - Packages should not modify the global environment (user’s workspace).
> {quote}
> Currently "spark-warehouse" gets created in the working directory when
> sparkR.session() is called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[
https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753414#comment-15753414
]
Felix Cheung commented on SPARK-18817:
--
It looks like javax.jdo.option.ConnectionURL can also be set in Hive-site.xml?
In that sense we should only change javax.jdo.option.ConnectionURL and
spark.sql.default.warehouse.dir when they are not set in conf or hive-site, and
we need to handle both for a complete fix.
> Ensure nothing is written outside R's tempdir() by default
> --
>
> Key: SPARK-18817
> URL: https://issues.apache.org/jira/browse/SPARK-18817
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
>Reporter: Brendan Dwyer
>Priority: Critical
>
> Per CRAN policies
> https://cran.r-project.org/web/packages/policies.html
> {quote}
> - Packages should not write in the users’ home filespace, nor anywhere else
> on the file system apart from the R session’s temporary directory (or during
> installation in the location pointed to by TMPDIR: and such usage should be
> cleaned up). Installing into the system’s R installation (e.g., scripts to
> its bin directory) is not allowed.
> Limited exceptions may be allowed in interactive sessions if the package
> obtains confirmation from the user.
> - Packages should not modify the global environment (user’s workspace).
> {quote}
> Currently "spark-warehouse" gets created in the working directory when
> sparkR.session() is called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[
https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753227#comment-15753227
]
Wenchen Fan commented on SPARK-18817:
-
the warehouse path will be created no matter hive support is enabled or not,
but derbe.log and metastore_db will only be created with hive support. The
simplest solution will be: disable hive support by default. We can also change
the location of metastore_db by "javax.jdo.option.ConnectionURL". I'm not sure
how to do it in R side, may be tricky.
> Ensure nothing is written outside R's tempdir() by default
> --
>
> Key: SPARK-18817
> URL: https://issues.apache.org/jira/browse/SPARK-18817
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
>Reporter: Brendan Dwyer
>Priority: Critical
>
> Per CRAN policies
> https://cran.r-project.org/web/packages/policies.html
> {quote}
> - Packages should not write in the users’ home filespace, nor anywhere else
> on the file system apart from the R session’s temporary directory (or during
> installation in the location pointed to by TMPDIR: and such usage should be
> cleaned up). Installing into the system’s R installation (e.g., scripts to
> its bin directory) is not allowed.
> Limited exceptions may be allowed in interactive sessions if the package
> obtains confirmation from the user.
> - Packages should not modify the global environment (user’s workspace).
> {quote}
> Currently "spark-warehouse" gets created in the working directory when
> sparkR.session() is called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[
https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753095#comment-15753095
]
Brendan Dwyer commented on SPARK-18817:
---
{code}
library("SparkR")
sparkR.session()
df <- as.DataFrame(iris)
{code}
> Ensure nothing is written outside R's tempdir() by default
> --
>
> Key: SPARK-18817
> URL: https://issues.apache.org/jira/browse/SPARK-18817
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
>Reporter: Brendan Dwyer
>Priority: Critical
>
> Per CRAN policies
> https://cran.r-project.org/web/packages/policies.html
> {quote}
> - Packages should not write in the users’ home filespace, nor anywhere else
> on the file system apart from the R session’s temporary directory (or during
> installation in the location pointed to by TMPDIR: and such usage should be
> cleaned up). Installing into the system’s R installation (e.g., scripts to
> its bin directory) is not allowed.
> Limited exceptions may be allowed in interactive sessions if the package
> obtains confirmation from the user.
> - Packages should not modify the global environment (user’s workspace).
> {quote}
> Currently "spark-warehouse" gets created in the working directory when
> sparkR.session() is called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[
https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753093#comment-15753093
]
Brendan Dwyer commented on SPARK-18817:
---
I'm not sure the CRAN people would be okay with that.
> Ensure nothing is written outside R's tempdir() by default
> --
>
> Key: SPARK-18817
> URL: https://issues.apache.org/jira/browse/SPARK-18817
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
>Reporter: Brendan Dwyer
>Priority: Critical
>
> Per CRAN policies
> https://cran.r-project.org/web/packages/policies.html
> {quote}
> - Packages should not write in the users’ home filespace, nor anywhere else
> on the file system apart from the R session’s temporary directory (or during
> installation in the location pointed to by TMPDIR: and such usage should be
> cleaned up). Installing into the system’s R installation (e.g., scripts to
> its bin directory) is not allowed.
> Limited exceptions may be allowed in interactive sessions if the package
> obtains confirmation from the user.
> - Packages should not modify the global environment (user’s workspace).
> {quote}
> Currently "spark-warehouse" gets created in the working directory when
> sparkR.session() is called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[
https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753064#comment-15753064
]
Felix Cheung commented on SPARK-18817:
--
Actually, I'm not seeing derby.log or metastore_db in the quick tests I have,
{code}
> createOrReplaceTempView(a, "foo")
> sql("SELECT * from foo")
{code}
[~bdwyer]do you have the steps that create these files?
> Ensure nothing is written outside R's tempdir() by default
> --
>
> Key: SPARK-18817
> URL: https://issues.apache.org/jira/browse/SPARK-18817
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
>Reporter: Brendan Dwyer
>Priority: Critical
>
> Per CRAN policies
> https://cran.r-project.org/web/packages/policies.html
> {quote}
> - Packages should not write in the users’ home filespace, nor anywhere else
> on the file system apart from the R session’s temporary directory (or during
> installation in the location pointed to by TMPDIR: and such usage should be
> cleaned up). Installing into the system’s R installation (e.g., scripts to
> its bin directory) is not allowed.
> Limited exceptions may be allowed in interactive sessions if the package
> obtains confirmation from the user.
> - Packages should not modify the global environment (user’s workspace).
> {quote}
> Currently "spark-warehouse" gets created in the working directory when
> sparkR.session() is called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[
https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753048#comment-15753048
]
Felix Cheung commented on SPARK-18817:
--
Tested this just now, I still see spark-warehouse when enableHiveSupport = FALSE
> Ensure nothing is written outside R's tempdir() by default
> --
>
> Key: SPARK-18817
> URL: https://issues.apache.org/jira/browse/SPARK-18817
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
>Reporter: Brendan Dwyer
>Priority: Critical
>
> Per CRAN policies
> https://cran.r-project.org/web/packages/policies.html
> {quote}
> - Packages should not write in the users’ home filespace, nor anywhere else
> on the file system apart from the R session’s temporary directory (or during
> installation in the location pointed to by TMPDIR: and such usage should be
> cleaned up). Installing into the system’s R installation (e.g., scripts to
> its bin directory) is not allowed.
> Limited exceptions may be allowed in interactive sessions if the package
> obtains confirmation from the user.
> - Packages should not modify the global environment (user’s workspace).
> {quote}
> Currently "spark-warehouse" gets created in the working directory when
> sparkR.session() is called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[
https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753040#comment-15753040
]
Felix Cheung commented on SPARK-18817:
--
we could, but we did ship 2.0 with it enabled by default though.
perhaps
{code}
enableHiveSupport = !interactive()
{code}
as default?
shouldn't derby.log and metastore_db go to the warehouse.dir?
> Ensure nothing is written outside R's tempdir() by default
> --
>
> Key: SPARK-18817
> URL: https://issues.apache.org/jira/browse/SPARK-18817
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
>Reporter: Brendan Dwyer
>Priority: Critical
>
> Per CRAN policies
> https://cran.r-project.org/web/packages/policies.html
> {quote}
> - Packages should not write in the users’ home filespace, nor anywhere else
> on the file system apart from the R session’s temporary directory (or during
> installation in the location pointed to by TMPDIR: and such usage should be
> cleaned up). Installing into the system’s R installation (e.g., scripts to
> its bin directory) is not allowed.
> Limited exceptions may be allowed in interactive sessions if the package
> obtains confirmation from the user.
> - Packages should not modify the global environment (user’s workspace).
> {quote}
> Currently "spark-warehouse" gets created in the working directory when
> sparkR.session() is called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[
https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753041#comment-15753041
]
Felix Cheung commented on SPARK-18817:
--
we could, but we did ship 2.0 with it enabled by default though.
perhaps
{code}
enableHiveSupport = !interactive()
{code}
as default?
shouldn't derby.log and metastore_db go to the warehouse.dir?
> Ensure nothing is written outside R's tempdir() by default
> --
>
> Key: SPARK-18817
> URL: https://issues.apache.org/jira/browse/SPARK-18817
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
>Reporter: Brendan Dwyer
>Priority: Critical
>
> Per CRAN policies
> https://cran.r-project.org/web/packages/policies.html
> {quote}
> - Packages should not write in the users’ home filespace, nor anywhere else
> on the file system apart from the R session’s temporary directory (or during
> installation in the location pointed to by TMPDIR: and such usage should be
> cleaned up). Installing into the system’s R installation (e.g., scripts to
> its bin directory) is not allowed.
> Limited exceptions may be allowed in interactive sessions if the package
> obtains confirmation from the user.
> - Packages should not modify the global environment (user’s workspace).
> {quote}
> Currently "spark-warehouse" gets created in the working directory when
> sparkR.session() is called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[
https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752973#comment-15752973
]
Shivaram Venkataraman commented on SPARK-18817:
---
In that case an easier fix might be to just disable Hive support by default ?
cc [~felixcheung]
> Ensure nothing is written outside R's tempdir() by default
> --
>
> Key: SPARK-18817
> URL: https://issues.apache.org/jira/browse/SPARK-18817
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
>Reporter: Brendan Dwyer
>Priority: Critical
>
> Per CRAN policies
> https://cran.r-project.org/web/packages/policies.html
> {quote}
> - Packages should not write in the users’ home filespace, nor anywhere else
> on the file system apart from the R session’s temporary directory (or during
> installation in the location pointed to by TMPDIR: and such usage should be
> cleaned up). Installing into the system’s R installation (e.g., scripts to
> its bin directory) is not allowed.
> Limited exceptions may be allowed in interactive sessions if the package
> obtains confirmation from the user.
> - Packages should not modify the global environment (user’s workspace).
> {quote}
> Currently "spark-warehouse" gets created in the working directory when
> sparkR.session() is called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[
https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752951#comment-15752951
]
Brendan Dwyer commented on SPARK-18817:
---
[~shivaram] it does not happen if I disable Hive.
> Ensure nothing is written outside R's tempdir() by default
> --
>
> Key: SPARK-18817
> URL: https://issues.apache.org/jira/browse/SPARK-18817
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
>Reporter: Brendan Dwyer
>Priority: Critical
>
> Per CRAN policies
> https://cran.r-project.org/web/packages/policies.html
> {quote}
> - Packages should not write in the users’ home filespace, nor anywhere else
> on the file system apart from the R session’s temporary directory (or during
> installation in the location pointed to by TMPDIR: and such usage should be
> cleaned up). Installing into the system’s R installation (e.g., scripts to
> its bin directory) is not allowed.
> Limited exceptions may be allowed in interactive sessions if the package
> obtains confirmation from the user.
> - Packages should not modify the global environment (user’s workspace).
> {quote}
> Currently "spark-warehouse" gets created in the working directory when
> sparkR.session() is called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[
https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752902#comment-15752902
]
Shivaram Venkataraman commented on SPARK-18817:
---
[~bdwyer] Does this still happen if you disable Hive ? One way to test that is
to stop the sparkSession and create one with `enableHiveSupport=F`
> Ensure nothing is written outside R's tempdir() by default
> --
>
> Key: SPARK-18817
> URL: https://issues.apache.org/jira/browse/SPARK-18817
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
>Reporter: Brendan Dwyer
>Priority: Critical
>
> Per CRAN policies
> https://cran.r-project.org/web/packages/policies.html
> {quote}
> - Packages should not write in the users’ home filespace, nor anywhere else
> on the file system apart from the R session’s temporary directory (or during
> installation in the location pointed to by TMPDIR: and such usage should be
> cleaned up). Installing into the system’s R installation (e.g., scripts to
> its bin directory) is not allowed.
> Limited exceptions may be allowed in interactive sessions if the package
> obtains confirmation from the user.
> - Packages should not modify the global environment (user’s workspace).
> {quote}
> Currently "spark-warehouse" gets created in the working directory when
> sparkR.session() is called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[
https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752898#comment-15752898
]
Shivaram Venkataraman commented on SPARK-18817:
---
Yeah I dont know how to avoid creating those two -- Doesn't look like its
configurable
cc [~cloud_fan] [~rxin]
> Ensure nothing is written outside R's tempdir() by default
> --
>
> Key: SPARK-18817
> URL: https://issues.apache.org/jira/browse/SPARK-18817
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
>Reporter: Brendan Dwyer
>Priority: Critical
>
> Per CRAN policies
> https://cran.r-project.org/web/packages/policies.html
> {quote}
> - Packages should not write in the users’ home filespace, nor anywhere else
> on the file system apart from the R session’s temporary directory (or during
> installation in the location pointed to by TMPDIR: and such usage should be
> cleaned up). Installing into the system’s R installation (e.g., scripts to
> its bin directory) is not allowed.
> Limited exceptions may be allowed in interactive sessions if the package
> obtains confirmation from the user.
> - Packages should not modify the global environment (user’s workspace).
> {quote}
> Currently "spark-warehouse" gets created in the working directory when
> sparkR.session() is called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[
https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752844#comment-15752844
]
Brendan Dwyer commented on SPARK-18817:
---
I'm also seeing _derby.log_ and a folder named _metastore_db_ being created in
my working directory when I run the following:
{code}
library("SparkR")
sparkR.session()
df <- as.DataFrame(iris)
{code}
> Ensure nothing is written outside R's tempdir() by default
> --
>
> Key: SPARK-18817
> URL: https://issues.apache.org/jira/browse/SPARK-18817
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
>Reporter: Brendan Dwyer
>Priority: Critical
>
> Per CRAN policies
> https://cran.r-project.org/web/packages/policies.html
> {quote}
> - Packages should not write in the users’ home filespace, nor anywhere else
> on the file system apart from the R session’s temporary directory (or during
> installation in the location pointed to by TMPDIR: and such usage should be
> cleaned up). Installing into the system’s R installation (e.g., scripts to
> its bin directory) is not allowed.
> Limited exceptions may be allowed in interactive sessions if the package
> obtains confirmation from the user.
> - Packages should not modify the global environment (user’s workspace).
> {quote}
> Currently "spark-warehouse" gets created in the working directory when
> sparkR.session() is called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[
https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15750076#comment-15750076
]
Apache Spark commented on SPARK-18817:
--
User 'shivaram' has created a pull request for this issue:
https://github.com/apache/spark/pull/16290
> Ensure nothing is written outside R's tempdir() by default
> --
>
> Key: SPARK-18817
> URL: https://issues.apache.org/jira/browse/SPARK-18817
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
>Reporter: Brendan Dwyer
>Priority: Critical
>
> Per CRAN policies
> https://cran.r-project.org/web/packages/policies.html
> {quote}
> - Packages should not write in the users’ home filespace, nor anywhere else
> on the file system apart from the R session’s temporary directory (or during
> installation in the location pointed to by TMPDIR: and such usage should be
> cleaned up). Installing into the system’s R installation (e.g., scripts to
> its bin directory) is not allowed.
> Limited exceptions may be allowed in interactive sessions if the package
> obtains confirmation from the user.
> - Packages should not modify the global environment (user’s workspace).
> {quote}
> Currently "spark-warehouse" gets created in the working directory when
> sparkR.session() is called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[
https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15749617#comment-15749617
]
Felix Cheung commented on SPARK-18817:
--
this could block R package release
> Ensure nothing is written outside R's tempdir() by default
> --
>
> Key: SPARK-18817
> URL: https://issues.apache.org/jira/browse/SPARK-18817
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
>Reporter: Brendan Dwyer
>Priority: Critical
>
> Per CRAN policies
> https://cran.r-project.org/web/packages/policies.html
> {quote}
> - Packages should not write in the users’ home filespace, nor anywhere else
> on the file system apart from the R session’s temporary directory (or during
> installation in the location pointed to by TMPDIR: and such usage should be
> cleaned up). Installing into the system’s R installation (e.g., scripts to
> its bin directory) is not allowed.
> Limited exceptions may be allowed in interactive sessions if the package
> obtains confirmation from the user.
> - Packages should not modify the global environment (user’s workspace).
> {quote}
> Currently "spark-warehouse" gets created in the working directory when
> sparkR.session() is called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[ https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15738558#comment-15738558 ] Apache Spark commented on SPARK-18817: -- User 'bdwyer2' has created a pull request for this issue: https://github.com/apache/spark/pull/16247 > Ensure nothing is written outside R's tempdir() by default > -- > > Key: SPARK-18817 > URL: https://issues.apache.org/jira/browse/SPARK-18817 > Project: Spark > Issue Type: Sub-task > Components: SparkR >Reporter: Brendan Dwyer > > Per CRAN policies > https://cran.r-project.org/web/packages/policies.html > "Packages should not write in the users’ home filespace, nor anywhere else on > the file system apart from the R session’s temporary directory (or during > installation in the location pointed to by TMPDIR: and such usage should be > cleaned up). Installing into the system’s R installation (e.g., scripts to > its bin directory) is not allowed. > Limited exceptions may be allowed in interactive sessions if the package > obtains confirmation from the user. > - Packages should not modify the global environment (user’s workspace)." > Currently "spark-warehouse" gets created in the working directory when > sparkR.session() is called. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
