[GitHub] spark pull request #22801: [SPARK-25656][SQL][DOC][EXAMPLE] Add a doc and ex...
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22801#discussion_r227434947 --- Diff: examples/src/main/scala/org/apache/spark/examples/sql/SQLDataSourceExample.scala --- @@ -56,6 +56,13 @@ object SQLDataSourceExample { .option("header", "true") .load("examples/src/main/resources/people.csv") // $example off:manual_load_options_csv$ +// $example on:manual_save_options_orc$ +usersDF.write.format("orc") + .option("orc.bloom.filter.columns", "favorite_color") + .option("orc.dictionary.key.threshold", "1.0") + .option("orc.column.encoding.direct", "name") + .save("users_with_options.orc") --- End diff -- Also, cc @dbtsai . This doc is only for Spark 3.0.0 since `orc.column.encoding.direct` is added to `master` branch. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22801: [SPARK-25656][SQL][DOC][EXAMPLE] Add a doc and ex...
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22801#discussion_r227457282 --- Diff: examples/src/main/r/RSparkSQLExample.R --- @@ -118,6 +118,10 @@ df <- read.df("examples/src/main/resources/people.csv", "csv", sep=";", inferSch namesAndAges <- select(df, "name", "age") # $example off:manual_load_options_csv$ +# $example on:manual_save_options_orc$ +df <- read.df("examples/src/main/resources/users.orc", "orc") +write.orc(df, "users_with_options.orc", orc.bloom.filter.columns="favorite_color", orc.dictionary.key.threshold=1.0, orc.column.encoding.direct="name") +# $example off:manual_save_options_orc$ --- End diff -- @felixcheung . Could you review this PR? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22801: [SPARK-25656][SQL][DOC][EXAMPLE] Add a doc and ex...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/22801#discussion_r227466415 --- Diff: examples/src/main/r/RSparkSQLExample.R --- @@ -118,6 +118,10 @@ df <- read.df("examples/src/main/resources/people.csv", "csv", sep=";", inferSch namesAndAges <- select(df, "name", "age") # $example off:manual_load_options_csv$ +# $example on:manual_save_options_orc$ +df <- read.df("examples/src/main/resources/users.orc", "orc") +write.orc(df, "users_with_options.orc", orc.bloom.filter.columns="favorite_color", orc.dictionary.key.threshold=1.0, orc.column.encoding.direct="name") +# $example off:manual_save_options_orc$ --- End diff -- we should put space after param (gosh same for csv example above) `orc.bloom.filter.columns = "favorite_color", orc.dictionary.key.threshold = 1.0, orc.column.encoding.direct = "name")` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22801: [SPARK-25656][SQL][DOC][EXAMPLE] Add a doc and ex...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/22801#discussion_r227466613 --- Diff: examples/src/main/python/sql/datasource.py --- @@ -57,6 +57,15 @@ def basic_datasource_example(spark): format="csv", sep=":", inferSchema="true", header="true") # $example off:manual_load_options_csv$ +# $example on:manual_save_options_orc$ +df = spark.read.orc("examples/src/main/resources/users.orc") +(df.write.format("orc") +.option("orc.bloom.filter.columns", "favorite_color") +.option("orc.dictionary.key.threshold", "1.0") +.option("orc.column.encoding.direct", 'name') --- End diff -- use same quote? `"` or `'` for name? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22801: [SPARK-25656][SQL][DOC][EXAMPLE] Add a doc and ex...
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22801#discussion_r227478845 --- Diff: examples/src/main/r/RSparkSQLExample.R --- @@ -118,6 +118,10 @@ df <- read.df("examples/src/main/resources/people.csv", "csv", sep=";", inferSch namesAndAges <- select(df, "name", "age") # $example off:manual_load_options_csv$ +# $example on:manual_save_options_orc$ +df <- read.df("examples/src/main/resources/users.orc", "orc") +write.orc(df, "users_with_options.orc", orc.bloom.filter.columns="favorite_color", orc.dictionary.key.threshold=1.0, orc.column.encoding.direct="name") +# $example off:manual_save_options_orc$ --- End diff -- Thank you! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22801: [SPARK-25656][SQL][DOC][EXAMPLE] Add a doc and ex...
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22801#discussion_r227478986 --- Diff: examples/src/main/python/sql/datasource.py --- @@ -57,6 +57,15 @@ def basic_datasource_example(spark): format="csv", sep=":", inferSchema="true", header="true") # $example off:manual_load_options_csv$ +# $example on:manual_save_options_orc$ +df = spark.read.orc("examples/src/main/resources/users.orc") +(df.write.format("orc") +.option("orc.bloom.filter.columns", "favorite_color") +.option("orc.dictionary.key.threshold", "1.0") +.option("orc.column.encoding.direct", 'name') --- End diff -- Yep! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22801: [SPARK-25656][SQL][DOC][EXAMPLE] Add a doc and ex...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22801 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org