[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...

2018-09-27 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22568#discussion_r220948793 --- Diff: python/pyspark/sql/tests.py --- @@ -5714,24 +5755,31 @@ def test_wrong_args(self): pandas_udf(lambda x, y: x

[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...

2018-09-27 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22568#discussion_r220948302 --- Diff: python/pyspark/sql/tests.py --- @@ -5525,32 +5525,73 @@ def data(self): .withColumn("v", explode(col('vs'))

[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...

2018-09-27 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22568#discussion_r220943714 --- Diff: python/pyspark/sql/tests.py --- @@ -5525,32 +5525,73 @@ def data(self): .withColumn("v", explode(col('vs'))

[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...

2018-09-27 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22568#discussion_r220948939 --- Diff: python/pyspark/sql/tests.py --- @@ -5714,24 +5755,31 @@ def test_wrong_args(self): pandas_udf(lambda x, y: x

[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...

2018-09-27 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22568#discussion_r220944429 --- Diff: python/pyspark/sql/tests.py --- @@ -5525,32 +5525,73 @@ def data(self): .withColumn("v", explode(col('vs'))

[GitHub] spark issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker ...

2018-09-27 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22570 Exactly same opinion with Sean's. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands

[GitHub] spark pull request #22237: [SPARK-25243][SQL] Use FailureSafeParser in from_...

2018-09-27 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22237#discussion_r220908825 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/FailureSafeParser.scala --- @@ -15,50 +15,51 @@ * limitations under

[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...

2018-09-27 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22568 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...

2018-09-27 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22568#discussion_r220880080 --- Diff: python/pyspark/sql/tests.py --- @@ -5715,23 +5752,29 @@ def test_wrong_args(self): def test_unsupported_types(self

[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...

2018-09-27 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22568#discussion_r220879251 --- Diff: python/pyspark/sql/tests.py --- @@ -5525,32 +5525,69 @@ def data(self): .withColumn("v", explode(col('vs'))

[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...

2018-09-27 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22568#discussion_r220878993 --- Diff: python/pyspark/sql/tests.py --- @@ -5525,32 +5525,69 @@ def data(self): .withColumn("v", explode(col('vs'))

[GitHub] spark issue #22237: [SPARK-25243][SQL] Use FailureSafeParser in from_json

2018-09-27 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22237 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21588: [SPARK-24590][BUILD] Make Jenkins tests passed with hado...

2018-09-27 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21588 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...

2018-09-26 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22545 Anyway, merged to master. Let me take #22552 out of branch-2.4 for now but please feel free to get this and that into branch-2.4 (without checking along with me) if you guys feel

[GitHub] spark issue #22237: [SPARK-25243][SQL] Use FailureSafeParser in from_json

2018-09-26 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22237 Seems okay. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22237: [SPARK-25243][SQL] Use FailureSafeParser in from_...

2018-09-26 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22237#discussion_r220789449 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/FailureSafeParser.scala --- @@ -15,50 +15,51 @@ * limitations under

[GitHub] spark pull request #22237: [SPARK-25243][SQL] Use FailureSafeParser in from_...

2018-09-26 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22237#discussion_r220788994 --- Diff: docs/sql-programming-guide.md --- @@ -1879,6 +1879,10 @@ working with timestamps in `pandas_udf`s to get the best performance, see

[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...

2018-09-26 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22545 I think the session support is kind of partially implemented in Python side, and not being very well tested. There are some inconsistency between Python and Scala side (for instance see https

[GitHub] spark pull request #22295: [SPARK-25255][PYTHON]Add getActiveSession to Spar...

2018-09-26 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22295#discussion_r220787563 --- Diff: python/pyspark/sql/session.py --- @@ -252,6 +253,22 @@ def newSession(self): """ return self.__c

[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...

2018-09-26 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22545 @cloud-fan, do we target this 2.4? Looks it might break an existing app, in particular, when a Python shell creates a session and another shell (like Zeppelin) or another session depends

[GitHub] spark pull request #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf fo...

2018-09-26 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22545#discussion_r220784330 --- Diff: python/pyspark/sql/session.py --- @@ -181,17 +181,11 @@ def getOrCreate(self): sparkConf.set(key, value

[GitHub] spark issue #22552: [SPARK-25540][SQL][PYSPARK] Make HiveContext in PySpark ...

2018-09-26 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22552 hm, why did we merge this into 2.4 btw? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22528: [SPARK-25513][SQL] Read zipped CSV and JSON

2018-09-26 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22528 The limitation is quite clear - multi line mode only and single file. We don't have to rush about this - looks @MaxGekk has a clear idea on this. Another concern here is, we have another place

[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()

2018-09-26 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22379 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #22550: [SPARK-25501] Kafka delegation token support

2018-09-26 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22550#discussion_r220590459 --- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala --- @@ -160,6 +160,10 @@ package object config { .doc("

[GitHub] spark issue #22551: [SPARK-25539][BUILD] Upgrade lz4-java to 1.5.0 get speed...

2018-09-26 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22551 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #22453: [SPARK-20937][DOCS] Describe spark.sql.parquet.writeLega...

2018-09-26 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22453 Merged to master and branch-2.4. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands

[GitHub] spark issue #22543: [SPARK-23715][SQL][DOC] improve document for from/to_utc...

2018-09-26 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22543 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark issue #22543: [SPARK-23715][SQL][DOC] improve document for from/to_utc...

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22543 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #22429: [SPARK-25440][SQL] Dumping query execution info t...

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22429#discussion_r220412301 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/package.scala --- @@ -167,6 +170,56 @@ package object util

[GitHub] spark pull request #22543: [SPARK-23715][SQL][DOC] improve document for from...

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22543#discussion_r220411454 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala --- @@ -1018,9 +1018,20 @@ case class

[GitHub] spark pull request #22543: [SPARK-23715][SQL][DOC] improve document for from...

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22543#discussion_r220410681 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala --- @@ -1018,9 +1018,20 @@ case class

[GitHub] spark pull request #22543: [SPARK-23715][SQL][DOC] improve document for from...

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22543#discussion_r220410505 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala --- @@ -1018,9 +1018,20 @@ case class

[GitHub] spark issue #22503: [SPARK-25493][SQL] Use auto-detection for CRLF in CSV da...

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22503 Seems fine but I or someone else should take a closer look before getting this in. --- - To unsubscribe, e-mail: reviews

[GitHub] spark pull request #22503: [SPARK-25493][SQL] Use auto-detection for CRLF in...

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22503#discussion_r220410193 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVOptions.scala --- @@ -212,6 +212,8 @@ class CSVOptions

[GitHub] spark pull request #22379: [SPARK-25393][SQL] Adding new function from_csv()

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22379#discussion_r220409483 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala --- @@ -520,7 +520,10 @@ object FunctionRegistry

[GitHub] spark pull request #22453: [SPARK-20937][DOCS] Describe spark.sql.parquet.wr...

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22453#discussion_r220409331 --- Diff: docs/sql-programming-guide.md --- @@ -1002,6 +1002,21 @@ Configuration of Parquet can be done using the `setConf` method on `SparkSession

[GitHub] spark pull request #22442: [SPARK-25447][SQL] Support JSON options by schema...

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22442#discussion_r220407196 --- Diff: python/pyspark/sql/functions.py --- @@ -2328,11 +2328,14 @@ def to_json(col, options={}): @ignore_unicode_prefix @since(2.4

[GitHub] spark pull request #22528: [SPARK-25513][SQL] Read zipped CSV and JSON

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22528#discussion_r220406916 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/CodecStreams.scala --- @@ -41,7 +42,12 @@ object CodecStreams

[GitHub] spark issue #22534: [SPARK-25514][SQL] Generating pretty JSON by to_json

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22534 LGTM Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands

[GitHub] spark pull request #22534: [SPARK-25514][SQL] Generating pretty JSON by to_j...

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22534#discussion_r220405853 --- Diff: R/pkg/R/functions.R --- @@ -198,8 +198,9 @@ NULL #' } #' @param ... additional argument(s). In \code{to_json} and \code

[GitHub] spark pull request #22534: [SPARK-25514][SQL] Generating pretty JSON by to_j...

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22534#discussion_r220405798 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala --- @@ -3635,6 +3637,8 @@ object functions { * @param e a column

[GitHub] spark issue #22537: [SPARK-21291][R] add R partitionBy API in DataFrame

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22537 Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #22540: [SPARK-24324] [PYTHON] [FOLLOW-UP] Rename the Conf to sp...

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22540 Merged to master and branch-2.4. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands

[GitHub] spark issue #22540: [SPARK-24324] [PYTHON] [FOLLOW-UP] Rename the Conf to sp...

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22540 Let me push this into branch-2.4 as well to address renaming concern by @BryanCutler. --- - To unsubscribe, e-mail: reviews

[GitHub] spark pull request #22540: [SPARK-24324] [PYTHON] [FOLLOW-UP] Rename the Con...

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22540#discussion_r220403432 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowUtils.scala --- @@ -131,11 +131,8 @@ object ArrowUtils { } else

[GitHub] spark issue #22503: [SPARK-25493][SQL] Use auto-detection for CRLF in CSV da...

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22503 Mind explaining what `setLineSeparatorDetectionEnabled` does in the PR description as well? --- - To unsubscribe, e-mail

[GitHub] spark issue #22503: [SPARK-25493][SQL] Use auto-detection for CRLF in CSV da...

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22503 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #22533: [SPARK-18818][PYTHON] Add 'ascending' parameter t...

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22533#discussion_r220218715 --- Diff: python/pyspark/sql/column.py --- @@ -54,6 +54,22 @@ def _to_java_column(col): return jcol +def

[GitHub] spark pull request #22533: [SPARK-18818][PYTHON] Add 'ascending' parameter t...

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22533#discussion_r220218332 --- Diff: python/pyspark/sql/window.py --- @@ -76,12 +79,37 @@ def partitionBy(*cols): @staticmethod @since(1.4) -def

[GitHub] spark issue #22541: [SPARK-23907][SQL] Revert regr_* functions entirely

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22541 Merged to master and branch-2.4. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands

[GitHub] spark issue #22361: Revert [SPARK-10399] [SPARK-23879] [SPARK-23762] [SPARK-...

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22361 It would be great if the information requested https://github.com/apache/spark/pull/22361#issuecomment-419607024 is shared so that other people can check it as well

[GitHub] spark issue #22540: [SPARK-24324] [PYTHON] [FOLLOW-UP] Rename the Conf to sp...

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22540 @gatorsmile, should be good to go after pushing https://github.com/gatorsmile/spark/pull/4 in --- - To unsubscribe, e-mail

[GitHub] spark pull request #22533: [SPARK-18818][PYTHON] Add 'ascending' parameter t...

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22533#discussion_r220122281 --- Diff: python/pyspark/sql/window.py --- @@ -76,12 +79,37 @@ def partitionBy(*cols): @staticmethod @since(1.4) -def

[GitHub] spark pull request #22543: [SPARK-23715][SQL] improve document for from/to_u...

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22543#discussion_r220112968 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala --- @@ -1018,9 +1018,20 @@ case class

[GitHub] spark pull request #22540: [SPARK-24324] [PYTHON] [FOLLOW-UP] Rename the Con...

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22540#discussion_r220111680 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -1295,15 +1295,15 @@ object SQLConf { .booleanConf

[GitHub] spark pull request #22540: [SPARK-24324] [PYTHON] [FOLLOW-UP] Rename the Con...

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22540#discussion_r220111468 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowUtils.scala --- @@ -131,11 +131,8 @@ object ArrowUtils { } else

[GitHub] spark issue #22198: [SPARK-25121][SQL] Supports multi-part table names for b...

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22198 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #22540: [SPARK-24324] [PYTHON] [FOLLOW-UP] Rename the Conf to sp...

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22540 Change itself LGTM except that bool comparison one --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #22540: [SPARK-24324] [PYTHON] [FOLLOW-UP] Rename the Con...

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22540#discussion_r220058070 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowUtils.scala --- @@ -131,11 +131,8 @@ object ArrowUtils { } else

[GitHub] spark issue #22541: [SPARK-23907][SQL] Revert regr_* functions entirely

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22541 I'm supportive of this change. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #22540: [SPARK-24324] [PYTHON] [FOLLOW-UP] Rename the Con...

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22540#discussion_r220054060 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -1295,15 +1295,15 @@ object SQLConf { .booleanConf

[GitHub] spark issue #22539: [SPARK-25517][SQL] Detect/Infer date type in CSV file

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22539 I think this is a duplicate of https://github.com/apache/spark/pull/21363 --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #22537: [SPARK-21291][R] add R partitionBy API in DataFra...

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22537#discussion_r220050102 --- Diff: R/pkg/tests/fulltests/test_sparkSQL.R --- @@ -2713,8 +2713,16 @@ test_that("read/write text files", { expect_equal(colnam

[GitHub] spark pull request #22537: [SPARK-21291][R] add R partitionBy API in DataFra...

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22537#discussion_r220050067 --- Diff: R/pkg/R/DataFrame.R --- @@ -2985,8 +2988,18 @@ setMethod("write.df", if (is.null(source)) {

[GitHub] spark pull request #22534: [SPARK-25514][SQL] Pretty JSON

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22534#discussion_r219903500 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala --- @@ -2515,4 +2515,35 @@ class JsonSuite extends

[GitHub] spark pull request #22534: [SPARK-25514][SQL] Pretty JSON

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22534#discussion_r219903090 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala --- @@ -2515,4 +2515,35 @@ class JsonSuite extends

[GitHub] spark pull request #22533: [SPARK-18818][PYTHON] Add 'ascending' parameter t...

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22533#discussion_r219902083 --- Diff: python/pyspark/sql/window.py --- @@ -76,12 +79,37 @@ def partitionBy(*cols): @staticmethod @since(1.4) -def

[GitHub] spark pull request #22455: [SPARK-24572][SPARKR] "eager execution" for R she...

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22455#discussion_r219897812 --- Diff: docs/sparkr.md --- @@ -450,6 +450,48 @@ print(model.summaries) {% endhighlight %} +### Eager execution

[GitHub] spark pull request #22453: [SPARK-20937][DOCS] Describe spark.sql.parquet.wr...

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22453#discussion_r219895824 --- Diff: docs/sql-programming-guide.md --- @@ -1002,6 +1002,21 @@ Configuration of Parquet can be done using the `setConf` method on `SparkSession

[GitHub] spark pull request #22453: [SPARK-20937][DOCS] Describe spark.sql.parquet.wr...

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22453#discussion_r219895047 --- Diff: docs/sql-programming-guide.md --- @@ -1002,6 +1002,21 @@ Configuration of Parquet can be done using the `setConf` method on `SparkSession

[GitHub] spark pull request #22453: [SPARK-20937][DOCS] Describe spark.sql.parquet.wr...

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22453#discussion_r219892751 --- Diff: docs/sql-programming-guide.md --- @@ -1002,6 +1002,21 @@ Configuration of Parquet can be done using the `setConf` method on `SparkSession

[GitHub] spark issue #22529: [SPARK-25460][BRANCH-2.4][SS] DataSourceV2: SS sources d...

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22529 Thanks, @dongjoon-hyun and @cloud-fan. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22529: [SPARK-25460][BRANCH-2.4][SS] DataSourceV2: SS so...

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon closed the pull request at: https://github.com/apache/spark/pull/22529 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22316: [SPARK-25048][SQL] Pivoting by multiple columns in Scala...

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22316 Can you just investigate if there's behaviour change about decimal precision? If there is, can you add a simple test if that's a better behaviour? If that's not a better behaviour, let's try

[GitHub] spark issue #22534: [SPARK-25514][SQL] Pretty JSON

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22534 Oh I rushed to read. Shall we then document? Let's add simple set of end to end tests for then as well

[GitHub] spark pull request #22379: [SPARK-25393][SQL] Adding new function from_csv()

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22379#discussion_r219815230 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/CsvFunctionsSuite.scala --- @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #22379: [SPARK-25393][SQL] Adding new function from_csv()

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22379#discussion_r219814691 --- Diff: sql/core/src/test/resources/sql-tests/inputs/csv-functions.sql --- @@ -0,0 +1,12 @@ +-- from_csv +describe function from_csv

[GitHub] spark pull request #22379: [SPARK-25393][SQL] Adding new function from_csv()

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22379#discussion_r219814329 --- Diff: sql/core/src/test/resources/sql-tests/inputs/csv-functions.sql --- @@ -0,0 +1,12 @@ +-- from_csv +describe function from_csv

[GitHub] spark pull request #22379: [SPARK-25393][SQL] Adding new function from_csv()

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22379#discussion_r219813519 --- Diff: python/pyspark/sql/functions.py --- @@ -2637,6 +2637,29 @@ def sequence(start, stop, step=None): _to_java_column(start

[GitHub] spark pull request #22533: [SPARK-18818][PYTHON] Add 'ascending' parameter t...

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22533#discussion_r219811169 --- Diff: python/pyspark/sql/window.py --- @@ -76,12 +79,37 @@ def partitionBy(*cols): @staticmethod @since(1.4) -def

[GitHub] spark pull request #22533: [SPARK-18818][PYTHON] Add 'ascending' parameter t...

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22533#discussion_r219812531 --- Diff: python/pyspark/sql/window.py --- @@ -76,12 +79,37 @@ def partitionBy(*cols): @staticmethod @since(1.4) -def

[GitHub] spark issue #22534: [SPARK-25514][SQL] Pretty JSON

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22534 Let's clarify this options is only for json functions in PR title --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #22534: [SPARK-25514][SQL] Pretty JSON

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22534#discussion_r219809938 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala --- @@ -113,6 +113,11 @@ private[sql] class JSONOptions

[GitHub] spark issue #22534: [SPARK-25514][SQL] Pretty JSON

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22534 I'm supportive of this idea. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark issue #22453: [SPARK-20937][DOCS] Describe spark.sql.parquet.writeLega...

2018-09-23 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22453 cc @jaceklaskowski FYI --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #22453: [SPARK-20937][DOCS] Describe spark.sql.parquet.wr...

2018-09-23 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22453#discussion_r219722950 --- Diff: docs/sql-programming-guide.md --- @@ -1002,6 +1002,15 @@ Configuration of Parquet can be done using the `setConf` method on `SparkSession

[GitHub] spark pull request #22529: [SPARK-25460][BRANCH-2.4][SS] DataSourceV2: SS so...

2018-09-23 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22529#discussion_r219722208 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamReader.scala --- @@ -173,12 +173,16 @@ final class DataStreamReader

[GitHub] spark issue #22531: [SPARK-25415][SQL][FOLLOW-UP] Add Locale.ROOT when toUpp...

2018-09-23 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22531 Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #22453: [SPARK-20937][DOCS] Describe spark.sql.parquet.wr...

2018-09-23 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22453#discussion_r219719299 --- Diff: docs/sql-programming-guide.md --- @@ -1002,6 +1002,15 @@ Configuration of Parquet can be done using the `setConf` method on `SparkSession

[GitHub] spark pull request #22453: [SPARK-20937][DOCS] Describe spark.sql.parquet.wr...

2018-09-23 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22453#discussion_r219719166 --- Diff: docs/sql-programming-guide.md --- @@ -1002,6 +1002,15 @@ Configuration of Parquet can be done using the `setConf` method on `SparkSession

[GitHub] spark issue #22529: [SPARK-25460][BRANCH-2.4][SS] DataSourceV2: SS sources d...

2018-09-23 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22529 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #22503: [SPARK-25493] [SQL] Fix multiline crlf

2018-09-22 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22503 Also, please fix the PR title to be more descriptive. For instance, `[SPARK-25493][SQL] Use auto-detection for CRLF in CSV datasource multiline mode

[GitHub] spark pull request #22503: [SPARK-25493] [SQL] Fix multiline crlf

2018-09-22 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22503#discussion_r219688971 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVOptions.scala --- @@ -212,6 +212,7 @@ class CSVOptions

[GitHub] spark issue #22503: [SPARK-25493] [SQL] Fix multiline crlf

2018-09-22 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22503 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22480: [SPARK-25473][PYTHON][SS][TEST] ForeachWriter tests fail...

2018-09-22 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22480 Thanks, @cloud-fan, @BryanCutler and @holdenk --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #22480: [SPARK-25473][PYTHON][SS][TEST] ForeachWriter tests fail...

2018-09-22 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22480 Merged only to master since I assume it's likely we will meet the test failures on master branch specifically more often

[GitHub] spark pull request #22316: [SPARK-25048][SQL] Pivoting by multiple columns i...

2018-09-22 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22316#discussion_r219686833 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/RelationalGroupedDataset.scala --- @@ -416,7 +426,7 @@ class RelationalGroupedDataset protected

[GitHub] spark issue #22227: [SPARK-25202] [SQL] Implements split with limit sql func...

2018-09-22 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/7 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22462: [SPARK-25460][SS] DataSourceV2: SS sources do not respec...

2018-09-22 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22462 The conflicts looks mainly renaming. I opened a backport - https://github.com/apache/spark/pull/22529 --- - To unsubscribe

<    9   10   11   12   13   14   15   16   17   18   >