subject:"spark git commit\: \[SPARK\-12318\]\[SPARKR\] Save mode in SparkR should be error by default"

spark git commit: [SPARK-12318][SPARKR] Save mode in SparkR should be error by default

2015-12-16 Thread shivaram

Repository: spark
Updated Branches:
  refs/heads/master 54c512ba9 -> 2eb5af5f0


[SPARK-12318][SPARKR] Save mode in SparkR should be error by default

shivaram  Please help review.

Author: Jeff Zhang 

Closes #10290 from zjffdu/SPARK-12318.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/2eb5af5f
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/2eb5af5f
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/2eb5af5f

Branch: refs/heads/master
Commit: 2eb5af5f0d3c424dc617bb1a18dd0210ea9ba0bc
Parents: 54c512b
Author: Jeff Zhang 
Authored: Wed Dec 16 10:32:32 2015 -0800
Committer: Shivaram Venkataraman 
Committed: Wed Dec 16 10:32:32 2015 -0800

--
 R/pkg/R/DataFrame.R | 10 +-
 docs/sparkr.md  |  9 -
 2 files changed, 13 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/2eb5af5f/R/pkg/R/DataFrame.R
--
diff --git a/R/pkg/R/DataFrame.R b/R/pkg/R/DataFrame.R
index 764597d..380a13f 100644
--- a/R/pkg/R/DataFrame.R
+++ b/R/pkg/R/DataFrame.R
@@ -1886,7 +1886,7 @@ setMethod("except",
 #' @param df A SparkSQL DataFrame
 #' @param path A name for the table
 #' @param source A name for external data source
-#' @param mode One of 'append', 'overwrite', 'error', 'ignore' save mode
+#' @param mode One of 'append', 'overwrite', 'error', 'ignore' save mode (it 
is 'error' by default)
 #'
 #' @family DataFrame functions
 #' @rdname write.df
@@ -1903,7 +1903,7 @@ setMethod("except",
 #' }
 setMethod("write.df",
   signature(df = "DataFrame", path = "character"),
-  function(df, path, source = NULL, mode = "append", ...){
+  function(df, path, source = NULL, mode = "error", ...){
 if (is.null(source)) {
   sqlContext <- get(".sparkRSQLsc", envir = .sparkREnv)
   source <- callJMethod(sqlContext, "getConf", 
"spark.sql.sources.default",
@@ -1928,7 +1928,7 @@ setMethod("write.df",
 #' @export
 setMethod("saveDF",
   signature(df = "DataFrame", path = "character"),
-  function(df, path, source = NULL, mode = "append", ...){
+  function(df, path, source = NULL, mode = "error", ...){
 write.df(df, path, source, mode, ...)
   })
 
@@ -1951,7 +1951,7 @@ setMethod("saveDF",
 #' @param df A SparkSQL DataFrame
 #' @param tableName A name for the table
 #' @param source A name for external data source
-#' @param mode One of 'append', 'overwrite', 'error', 'ignore' save mode
+#' @param mode One of 'append', 'overwrite', 'error', 'ignore' save mode (it 
is 'error' by default)
 #'
 #' @family DataFrame functions
 #' @rdname saveAsTable
@@ -1968,7 +1968,7 @@ setMethod("saveDF",
 setMethod("saveAsTable",
   signature(df = "DataFrame", tableName = "character", source = 
"character",
 mode = "character"),
-  function(df, tableName, source = NULL, mode="append", ...){
+  function(df, tableName, source = NULL, mode="error", ...){
 if (is.null(source)) {
   sqlContext <- get(".sparkRSQLsc", envir = .sparkREnv)
   source <- callJMethod(sqlContext, "getConf", 
"spark.sql.sources.default",

http://git-wip-us.apache.org/repos/asf/spark/blob/2eb5af5f/docs/sparkr.md
--
diff --git a/docs/sparkr.md b/docs/sparkr.md
index 0114878..9ddd2ed 100644
--- a/docs/sparkr.md
+++ b/docs/sparkr.md
@@ -148,7 +148,7 @@ printSchema(people)
 
 
 The data sources API can also be used to save out DataFrames into multiple 
file formats. For example we can save the DataFrame from the previous example
-to a Parquet file using `write.df`
+to a Parquet file using `write.df` (Until Spark 1.6, the default mode for 
writes was `append`. It was changed in Spark 1.7 to `error` to match the Scala 
API)
 
 
 {% highlight r %}
@@ -387,3 +387,10 @@ The following functions are masked by the SparkR package:
 Since part of SparkR is modeled on the `dplyr` package, certain functions in 
SparkR share the same names with those in `dplyr`. Depending on the load order 
of the two packages, some functions from the package loaded first are masked by 
those in the package loaded after. In such case, prefix such calls with the 
package name, for instance, `SparkR::cume_dist(x)` or `dplyr::cume_dist(x)`.
   
 You can inspect the search path in R with 
[`search()`](https://stat.ethz.ch/R-manual/R-devel/library/base/html/search.html)
+
+
+# Migration Guide
+
+## Upgrading From SparkR 1.6 to 1.7
+
+ - Until Spark 1.6, the default mode for writes was `append`. It was changed 
in Spark 1.7 to `error` to match the Scala API.

spark git commit: [SPARK-12318][SPARKR] Save mode in SparkR should be error by default

2015-12-16 Thread shivaram

Repository: spark
Updated Branches:
  refs/heads/branch-1.6 16edd933d -> f81512729


[SPARK-12318][SPARKR] Save mode in SparkR should be error by default

shivaram  Please help review.

Author: Jeff Zhang 

Closes #10290 from zjffdu/SPARK-12318.

(cherry picked from commit 2eb5af5f0d3c424dc617bb1a18dd0210ea9ba0bc)
Signed-off-by: Shivaram Venkataraman 


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/f8151272
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/f8151272
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/f8151272

Branch: refs/heads/branch-1.6
Commit: f815127294c06320204d9affa4f35da7ec3a710d
Parents: 16edd93
Author: Jeff Zhang 
Authored: Wed Dec 16 10:32:32 2015 -0800
Committer: Shivaram Venkataraman 
Committed: Wed Dec 16 10:48:54 2015 -0800

--
 R/pkg/R/DataFrame.R | 10 +-
 docs/sparkr.md  |  9 -
 2 files changed, 13 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/f8151272/R/pkg/R/DataFrame.R
--
diff --git a/R/pkg/R/DataFrame.R b/R/pkg/R/DataFrame.R
index 7292433..0cfa12b9 100644
--- a/R/pkg/R/DataFrame.R
+++ b/R/pkg/R/DataFrame.R
@@ -1925,7 +1925,7 @@ setMethod("except",
 #' @param df A SparkSQL DataFrame
 #' @param path A name for the table
 #' @param source A name for external data source
-#' @param mode One of 'append', 'overwrite', 'error', 'ignore' save mode
+#' @param mode One of 'append', 'overwrite', 'error', 'ignore' save mode (it 
is 'error' by default)
 #'
 #' @family DataFrame functions
 #' @rdname write.df
@@ -1942,7 +1942,7 @@ setMethod("except",
 #' }
 setMethod("write.df",
   signature(df = "DataFrame", path = "character"),
-  function(df, path, source = NULL, mode = "append", ...){
+  function(df, path, source = NULL, mode = "error", ...){
 if (is.null(source)) {
   sqlContext <- get(".sparkRSQLsc", envir = .sparkREnv)
   source <- callJMethod(sqlContext, "getConf", 
"spark.sql.sources.default",
@@ -1967,7 +1967,7 @@ setMethod("write.df",
 #' @export
 setMethod("saveDF",
   signature(df = "DataFrame", path = "character"),
-  function(df, path, source = NULL, mode = "append", ...){
+  function(df, path, source = NULL, mode = "error", ...){
 write.df(df, path, source, mode, ...)
   })
 
@@ -1990,7 +1990,7 @@ setMethod("saveDF",
 #' @param df A SparkSQL DataFrame
 #' @param tableName A name for the table
 #' @param source A name for external data source
-#' @param mode One of 'append', 'overwrite', 'error', 'ignore' save mode
+#' @param mode One of 'append', 'overwrite', 'error', 'ignore' save mode (it 
is 'error' by default)
 #'
 #' @family DataFrame functions
 #' @rdname saveAsTable
@@ -2007,7 +2007,7 @@ setMethod("saveDF",
 setMethod("saveAsTable",
   signature(df = "DataFrame", tableName = "character", source = 
"character",
 mode = "character"),
-  function(df, tableName, source = NULL, mode="append", ...){
+  function(df, tableName, source = NULL, mode="error", ...){
 if (is.null(source)) {
   sqlContext <- get(".sparkRSQLsc", envir = .sparkREnv)
   source <- callJMethod(sqlContext, "getConf", 
"spark.sql.sources.default",

http://git-wip-us.apache.org/repos/asf/spark/blob/f8151272/docs/sparkr.md
--
diff --git a/docs/sparkr.md b/docs/sparkr.md
index 0114878..9ddd2ed 100644
--- a/docs/sparkr.md
+++ b/docs/sparkr.md
@@ -148,7 +148,7 @@ printSchema(people)
 
 
 The data sources API can also be used to save out DataFrames into multiple 
file formats. For example we can save the DataFrame from the previous example
-to a Parquet file using `write.df`
+to a Parquet file using `write.df` (Until Spark 1.6, the default mode for 
writes was `append`. It was changed in Spark 1.7 to `error` to match the Scala 
API)
 
 
 {% highlight r %}
@@ -387,3 +387,10 @@ The following functions are masked by the SparkR package:
 Since part of SparkR is modeled on the `dplyr` package, certain functions in 
SparkR share the same names with those in `dplyr`. Depending on the load order 
of the two packages, some functions from the package loaded first are masked by 
those in the package loaded after. In such case, prefix such calls with the 
package name, for instance, `SparkR::cume_dist(x)` or `dplyr::cume_dist(x)`.
   
 You can inspect the search path in R with 
[`search()`](https://stat.ethz.ch/R-manual/R-devel/library/base/html/search.html)
+
+
+# Migration Guide
+
+## Upgrading From SparkR 1.6 to 1.7
+
+ - Until Spark

spark git commit: [SPARK-12318][SPARKR] Save mode in SparkR should be error by default

spark git commit: [SPARK-12318][SPARKR] Save mode in SparkR should be error by default

2 matches

Site Navigation

Mail list logo

Footer information