[GitHub] spark pull request: [SPARK-14883][DOCS] Fix wrong R examples and m...
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12649#issuecomment-214135532 Thank you for merging, @shivaram . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14883][DOCS] Fix wrong R examples and m...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/12649 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14883][DOCS] Fix wrong R examples and m...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/12649#issuecomment-214133454 LGTM. Merging this --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14883][DOCS] Fix wrong R examples and m...
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12649#issuecomment-214132007 Thank you, @felixcheung ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14883][DOCS] Fix wrong R examples and m...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12649#issuecomment-214130308 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14883][DOCS] Fix wrong R examples and m...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12649#issuecomment-214130310 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56876/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14883][DOCS] Fix wrong R examples and m...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12649#issuecomment-214130277 **[Test build #56876 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56876/consoleFull)** for PR 12649 at commit [`47aa57e`](https://github.com/apache/spark/commit/47aa57ef1d6533ef5c5161fd2caf7c1ed3ff5818). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14883][DOCS] Fix wrong R examples and m...
Github user felixcheung commented on the pull request: https://github.com/apache/spark/pull/12649#issuecomment-214130171 looks good! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14883][DOCS] Fix wrong R examples and m...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12649#issuecomment-214124731 **[Test build #56876 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56876/consoleFull)** for PR 12649 at commit [`47aa57e`](https://github.com/apache/spark/commit/47aa57ef1d6533ef5c5161fd2caf7c1ed3ff5818). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14883][DOCS] Fix wrong R examples and m...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/12649#discussion_r60861763
--- Diff: docs/sql-programming-guide.md ---
@@ -1138,16 +1138,16 @@ for teenName in teenNames.collect():
schemaPeople # The DataFrame from the previous example.
# DataFrames can be saved as Parquet files, maintaining the schema
information.
-saveAsParquetFile(schemaPeople, "people.parquet")
+write.parquet(schemaPeople, "people.parquet")
# Read in the Parquet file created above. Parquet files are
self-describing so the schema is preserved.
# The result of loading a parquet file is also a DataFrame.
-parquetFile <- parquetFile(sqlContext, "people.parquet")
+parquetFile <- read.parquet(sqlContext, "people.parquet")
# Parquet files can also be registered as tables and then used in SQL
statements.
-registerTempTable(parquetFile, "parquetFile");
+registerTempTable(parquetFile, "parquetFile")
teenagers <- sql(sqlContext, "SELECT name FROM parquetFile WHERE age >= 13
AND age <= 19")
-teenNames <- map(teenagers, function(p) { paste("Name:", p$name)})
+teenNames <- SparkR:::lapply(teenagers, function(p) { paste("Name:",
p$name) })
--- End diff --
Thank you, @sun-rui .
Then, I will update this PR by simply removing those problematic lines.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14883][DOCS] Fix wrong R examples and m...
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/12649#discussion_r60861420
--- Diff: docs/sql-programming-guide.md ---
@@ -1138,16 +1138,16 @@ for teenName in teenNames.collect():
schemaPeople # The DataFrame from the previous example.
# DataFrames can be saved as Parquet files, maintaining the schema
information.
-saveAsParquetFile(schemaPeople, "people.parquet")
+write.parquet(schemaPeople, "people.parquet")
# Read in the Parquet file created above. Parquet files are
self-describing so the schema is preserved.
# The result of loading a parquet file is also a DataFrame.
-parquetFile <- parquetFile(sqlContext, "people.parquet")
+parquetFile <- read.parquet(sqlContext, "people.parquet")
# Parquet files can also be registered as tables and then used in SQL
statements.
-registerTempTable(parquetFile, "parquetFile");
+registerTempTable(parquetFile, "parquetFile")
teenagers <- sql(sqlContext, "SELECT name FROM parquetFile WHERE age >= 13
AND age <= 19")
-teenNames <- map(teenagers, function(p) { paste("Name:", p$name)})
+teenNames <- SparkR:::lapply(teenagers, function(p) { paste("Name:",
p$name) })
--- End diff --
You can delete it. I can add a line for dapply in my PR. @shivaram
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14883][DOCS] Fix wrong R examples and m...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/12649#discussion_r60850733
--- Diff: docs/sql-programming-guide.md ---
@@ -1138,16 +1138,16 @@ for teenName in teenNames.collect():
schemaPeople # The DataFrame from the previous example.
# DataFrames can be saved as Parquet files, maintaining the schema
information.
-saveAsParquetFile(schemaPeople, "people.parquet")
+write.parquet(schemaPeople, "people.parquet")
# Read in the Parquet file created above. Parquet files are
self-describing so the schema is preserved.
# The result of loading a parquet file is also a DataFrame.
-parquetFile <- parquetFile(sqlContext, "people.parquet")
+parquetFile <- read.parquet(sqlContext, "people.parquet")
# Parquet files can also be registered as tables and then used in SQL
statements.
-registerTempTable(parquetFile, "parquetFile");
+registerTempTable(parquetFile, "parquetFile")
teenagers <- sql(sqlContext, "SELECT name FROM parquetFile WHERE age >= 13
AND age <= 19")
-teenNames <- map(teenagers, function(p) { paste("Name:", p$name)})
+teenNames <- SparkR:::lapply(teenagers, function(p) { paste("Name:",
p$name) })
--- End diff --
Thank you for review, @shivaram . What about deleting the problematic line
and the rest and leaving a comment with JIRA issue number for `dapply`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14883][DOCS] Fix wrong R examples and m...
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/12649#discussion_r60849890
--- Diff: docs/sql-programming-guide.md ---
@@ -1138,16 +1138,16 @@ for teenName in teenNames.collect():
schemaPeople # The DataFrame from the previous example.
# DataFrames can be saved as Parquet files, maintaining the schema
information.
-saveAsParquetFile(schemaPeople, "people.parquet")
+write.parquet(schemaPeople, "people.parquet")
# Read in the Parquet file created above. Parquet files are
self-describing so the schema is preserved.
# The result of loading a parquet file is also a DataFrame.
-parquetFile <- parquetFile(sqlContext, "people.parquet")
+parquetFile <- read.parquet(sqlContext, "people.parquet")
# Parquet files can also be registered as tables and then used in SQL
statements.
-registerTempTable(parquetFile, "parquetFile");
+registerTempTable(parquetFile, "parquetFile")
teenagers <- sql(sqlContext, "SELECT name FROM parquetFile WHERE age >= 13
AND age <= 19")
-teenNames <- map(teenagers, function(p) { paste("Name:", p$name)})
+teenNames <- SparkR:::lapply(teenagers, function(p) { paste("Name:",
p$name) })
--- End diff --
We should really be having an example that uses `:::` -- Can we remove this
section for now ? We can add it back once `dapply` is checked in (cc @sun-rui )
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14883][DOCS] Fix wrong R examples and m...
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12649#issuecomment-213907830 Hi, @davies , @shivaram, @felixcheung . Could you review this PR when you have some time? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14883][DOCS] Fix wrong R examples and m...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12649#issuecomment-213906575 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14883][DOCS] Fix wrong R examples and m...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12649#issuecomment-213906576 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56835/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14883][DOCS] Fix wrong R examples and m...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12649#issuecomment-213906561 **[Test build #56835 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56835/consoleFull)** for PR 12649 at commit [`5d6d45e`](https://github.com/apache/spark/commit/5d6d45e07c15d17c5d1972733962013a6fcd228c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14883][DOCS] Fix wrong R examples and m...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12649#issuecomment-213905769 **[Test build #56835 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56835/consoleFull)** for PR 12649 at commit [`5d6d45e`](https://github.com/apache/spark/commit/5d6d45e07c15d17c5d1972733962013a6fcd228c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14883][DOCS] Fix wrong R examples and m...
GitHub user dongjoon-hyun opened a pull request:
https://github.com/apache/spark/pull/12649
[SPARK-14883][DOCS] Fix wrong R examples and make them up-to-date
## What changes were proposed in this pull request?
This issue aims to fix some errors in R examples and make them up-to-date
in docs and example modules.
- Fix the wrong usage of map. We need to use `lapply` if needed. However,
the usage of `lapply` also needs to be reviewed since it's private.
```
-teenNames <- map(teenagers, function(p) { paste("Name:", p$name)})
+teenNames <- SparkR:::lapply(teenagers, function(p) { paste("Name:",
p$name) })
```
- Fix the wrong example in Section `Generic Load/Save Functions` of
`docs/sql-programming-guide.md` for consistency
- Fix datatypes in `sparkr.md`.
- Update a data result in `sparkr.md`.
- Replace deprecated functions to remove warnings: jsonFile -> read.json,
parquetFile -> read.parquet
- Use up-to-date R-like functions: loadDF -> read.df, saveDF -> write.df,
saveAsParquetFile -> write.parquet
- Replace `SparkR DataFrame` with `SparkDataFrame` in `dataframe.R` and
`data-manipulation.R`.
- Other minor syntax fixes and a typo.
## How was this patch tested?
Manual.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dongjoon-hyun/spark SPARK-14883
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12649.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12649
commit 5d6d45e07c15d17c5d1972733962013a6fcd228c
Author: Dongjoon Hyun
Date: 2016-04-24T06:43:45Z
[SPARK-14883][DOCS] Fix wrong R examples and make them up-to-date
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
