Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/12817
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216092645
I also created a follow-up ticket for moving the options to function
arguments: https://issues.apache.org/jira/browse/SPARK-15050
---
If your project is set up for
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216092582
Thanks - merging in master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216092409
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216092410
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216092342
**[Test build #57491 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57491/consoleFull)**
for PR 12817 at commit
Github user HyukjinKwon commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216090703
@rxin I see. Thank you.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216090565
Don't worry about it. We can just build it on top of this (we should still
document them, just better as function arguments).
---
If your project is set up for it, you
Github user HyukjinKwon commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216090207
Sure. Thank you. Do you want me to remove Python documentation here?
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216089939
The scala changes lgtm.
One thing I just realized -- for Python, I think we should turn those into
named arguments, rather than just options. Can you do that in a
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216084669
**[Test build #57491 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57491/consoleFull)**
for PR 12817 at commit
Github user HyukjinKwon commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216084433
@rxin I am so sorry, I think I totally misunderstood your initial comments
before. I just addressed your comments later. Thank you.
---
If your project is set up
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216064461
Thanks - this is really close. Let's fix the minor issues and then we can
merge.
---
If your project is set up for it, you can reply to this email and have your
reply
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12817#discussion_r61691203
--- Diff: python/pyspark/sql/readwriter.py ---
@@ -282,6 +282,45 @@ def csv(self, paths):
:param paths: string, or list of strings, for input
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12817#discussion_r61691200
--- Diff: python/pyspark/sql/readwriter.py ---
@@ -282,6 +282,45 @@ def csv(self, paths):
:param paths: string, or list of strings, for input
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12817#discussion_r61691211
--- Diff: python/pyspark/sql/readwriter.py ---
@@ -282,6 +282,45 @@ def csv(self, paths):
:param paths: string, or list of strings, for input
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12817#discussion_r61691199
--- Diff: python/pyspark/sql/readwriter.py ---
@@ -663,6 +702,19 @@ def csv(self, path, mode=None, compression=None):
known
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12817#discussion_r61691201
--- Diff: python/pyspark/sql/readwriter.py ---
@@ -282,6 +282,45 @@ def csv(self, paths):
:param paths: string, or list of strings, for input
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12817#discussion_r61691196
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ---
@@ -606,6 +606,14 @@ final class DataFrameWriter private[sql](df:
DataFrame)
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12817#discussion_r61691181
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
@@ -393,6 +393,45 @@ class DataFrameReader private[sql](sparkSession:
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12817#discussion_r61691183
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
@@ -393,6 +393,45 @@ class DataFrameReader private[sql](sparkSession:
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216041028
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216041029
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216040991
**[Test build #57479 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57479/consoleFull)**
for PR 12817 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216036699
**[Test build #57479 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57479/consoleFull)**
for PR 12817 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216025414
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216025415
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216025375
**[Test build #57471 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57471/consoleFull)**
for PR 12817 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216021442
**[Test build #57471 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57471/consoleFull)**
for PR 12817 at commit
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/12817#discussion_r61682274
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
@@ -393,6 +393,45 @@ class DataFrameReader private[sql](sparkSession:
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12817#discussion_r61682060
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
@@ -393,6 +393,45 @@ class DataFrameReader private[sql](sparkSession:
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12817#discussion_r61682039
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
@@ -393,6 +393,45 @@ class DataFrameReader private[sql](sparkSession:
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216017668
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216017667
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216017641
**[Test build #57467 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57467/consoleFull)**
for PR 12817 at commit
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/12817#discussion_r61679408
--- Diff: python/pyspark/sql/readwriter.py ---
@@ -663,6 +700,18 @@ def csv(self, path, mode=None, compression=None):
Github user HyukjinKwon commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216015221
@rxin BTW, I found two todos, `TODO: Remove this one in Spark 2.0.` at
`DataFrameReader` and `DataFrameWriter` added in
https://github.com/apache/spark/pull/9945.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216015119
**[Test build #57467 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57467/consoleFull)**
for PR 12817 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216015021
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216015022
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216015018
**[Test build #57466 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57466/consoleFull)**
for PR 12817 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216014991
**[Test build #57466 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57466/consoleFull)**
for PR 12817 at commit
Github user HyukjinKwon commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216014925
cc @rxin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/12817#discussion_r61679234
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
@@ -393,6 +393,45 @@ class DataFrameReader private[sql](sparkSession:
Github user HyukjinKwon commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216014355
Hm.. this gives me a pass of Python style test at local.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user HyukjinKwon commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216013835
Hm.. It passes a local style test for Python and I did not edit
`./python/pyspark/sql/__init__.py`.
---
If your project is set up for it, you can reply to this
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216013493
**[Test build #57464 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57464/consoleFull)**
for PR 12817 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216013507
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216013503
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216012967
**[Test build #57464 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57464/consoleFull)**
for PR 12817 at commit
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/12817#discussion_r61678681
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
@@ -393,6 +393,45 @@ class DataFrameReader private[sql](sparkSession:
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216012741
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216012739
**[Test build #57463 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57463/consoleFull)**
for PR 12817 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216012742
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216012673
**[Test build #57463 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57463/consoleFull)**
for PR 12817 at commit
GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/12817
[SPARK-13425][SQL] Documentation for CSV datasource options
## What changes were proposed in this pull request?
This PR adds the explanation and documentation for CSV options for
56 matches
Mail list logo
