[GitHub] spark issue #20895: [SPARK-23787][tests] Fix file download test in SparkSubm...

Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/20895 Merging to master branch. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20851: [SPARK-23727][SQL] Support for pushing down filters for ...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20851 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88575/ Test PASSed. ---

[GitHub] spark issue #20851: [SPARK-23727][SQL] Support for pushing down filters for ...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20851 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20851: [SPARK-23727][SQL] Support for pushing down filters for ...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20851 **[Test build #88575 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88575/testReport)** for PR 20851 at commit

[GitHub] spark issue #20887: [SPARK-23774][SQL] `Cast` to CHAR/VARCHAR should truncat...

Github user maropu commented on the issue: https://github.com/apache/spark/pull/20887 Since this is a behaviour change, I think we need to update the migration guide? --- - To unsubscribe, e-mail:

[GitHub] spark pull request #20887: [SPARK-23774][SQL] `Cast` to CHAR/VARCHAR should ...

Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/20887#discussion_r176982595 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala --- @@ -2792,4 +2793,31 @@ class SQLQuerySuite extends QueryTest with

[GitHub] spark issue #20858: [SPARK-23736][SQL] Implementation of the concat_arrays f...

Github user maropu commented on the issue: https://github.com/apache/spark/pull/20858 Also, `postgresql` has the function `array_cat` for concatenating arrays, so it might be better to make the behaviour the same with the `postgresql` one:

[GitHub] spark pull request #20858: [SPARK-23736][SQL] Implementation of the concat_a...

Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/20858#discussion_r176981046 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -287,3 +289,152 @@ case class

[GitHub] spark issue #20893: [SPARK-23785][LAUNCHER] LauncherBackend doesn't check st...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20893 **[Test build #88577 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88577/testReport)** for PR 20893 at commit

[GitHub] spark issue #20893: [SPARK-23785][LAUNCHER] LauncherBackend doesn't check st...

Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20893 Jenkins, ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20893: [SPARK-23785][LAUNCHER] LauncherBackend doesn't check st...

Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20893 The change looks good, cc @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20893: [SPARK-23785][LAUNCHER] LauncherBackend doesn't c...

Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20893#discussion_r176978788 --- Diff: core/src/main/scala/org/apache/spark/launcher/LauncherBackend.scala --- @@ -114,10 +114,10 @@ private[spark] abstract class LauncherBackend {

[GitHub] spark issue #20795: [SPARK-23486]cache the function name from the catalog fo...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20795 **[Test build #88576 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88576/testReport)** for PR 20795 at commit

[GitHub] spark issue #20895: [SPARK-23787][tests] Fix file download test in SparkSubm...

Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20895 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #20900: [SPARK-23645][MINOR][DOCS][PYTHON] Add docs RE `p...

Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20900 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20900: [SPARK-23645][MINOR][DOCS][PYTHON] Add docs RE `pandas_u...

Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20900 Merged to master and branch-2.3 anyway. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20892: [SPARK-23700][PYTHON] Cleanup imports in pyspark....

Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20892 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20892: [SPARK-23700][PYTHON] Cleanup imports in pyspark.sql

Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20892 Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20900: [SPARK-23645][MINOR][DOCS][PYTHON] Add docs RE `pandas_u...

Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20900 I think we should generally make everything works in both Python 2 and Python 3 but I want to know if there are special chases that I am missing too if there are any. ---

[GitHub] spark pull request #20756: [SPARK-23593][SQL] Add interpreted execution for ...

Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20756#discussion_r176973244 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala --- @@ -1261,8 +1261,42 @@ case class

[GitHub] spark pull request #20858: [SPARK-23736][SQL] Implementation of the concat_a...

Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/20858#discussion_r176971799 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -287,3 +289,152 @@ case class

[GitHub] spark pull request #20851: [SPARK-23727][SQL] Support for pushing down filte...

Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20851#discussion_r176971232 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala --- @@ -129,6 +154,10 @@ private[parquet]

[GitHub] spark pull request #20851: [SPARK-23727][SQL] Support for pushing down filte...

Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20851#discussion_r176971146 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala --- @@ -50,6 +59,10 @@ private[parquet] object

[GitHub] spark issue #20858: [SPARK-23736][SQL] Implementation of the concat_arrays f...

Github user maropu commented on the issue: https://github.com/apache/spark/pull/20858 We should handle different (and compatible) typed arrays in this funs? ``` scala> sql("select concat_arrays(array(1L, 2L), array(3, 4))").show org.apache.spark.sql.AnalysisException:

[GitHub] spark issue #20858: [SPARK-23736][SQL] Implementation of the concat_arrays f...

Github user maropu commented on the issue: https://github.com/apache/spark/pull/20858 The current code can't handle inner arrays; ``` scala> sql("select concat_arrays(array(1, 2, array(3, 4)), array(5, 6, 7, 8))").show org.apache.spark.sql.AnalysisException: cannot resolve

[GitHub] spark issue #20842: [SPARK-23162][PySpark][ML] Add r2adj into Python API in ...

Github user kevinyu98 commented on the issue: https://github.com/apache/spark/pull/20842 @tengpeng Thanks, are you using ./dev/lint-python to run the python style test locally? --- - To unsubscribe, e-mail:

[GitHub] spark pull request #20849: [SPARK-23723] New charset option for json datasou...

Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20849#discussion_r176966887 --- Diff: python/pyspark/sql/readwriter.py --- @@ -176,7 +176,7 @@ def json(self, path, schema=None, primitivesAsString=None, prefersDecimal=None,

[GitHub] spark issue #20851: [SPARK-23727][SQL] Support for pushing down filters for ...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20851 **[Test build #88575 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88575/testReport)** for PR 20851 at commit

[GitHub] spark pull request #20851: [SPARK-23727][SQL] Support for pushing down filte...

Github user yucai commented on a diff in the pull request: https://github.com/apache/spark/pull/20851#discussion_r176966288 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -353,6 +353,13 @@ object SQLConf { .booleanConf

[GitHub] spark pull request #20858: [SPARK-23736][SQL] Implementation of the concat_a...

Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/20858#discussion_r176966017 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala --- @@ -3046,6 +3046,14 @@ object functions { ArrayContains(column.expr,

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20877 If this one is merged, I believe it should be easier to review #20885 too. --- - To unsubscribe, e-mail:

[GitHub] spark issue #20849: [SPARK-23723] New charset option for json datasource

Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20849 I am against to this mainly by https://github.com/MaxGekk/spark-1/pull/1#discussion_r175444502 if there isn't better way than rewriting it. Also, I think we should support `charset` option

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20877 There was a discussion about the naming here - https://github.com/apache/spark/pull/20727#discussion_r172341859. I am against to `recordDelimiter`. Both PR deal with a different

[GitHub] spark issue #19222: [SPARK-10399][CORE][SQL] Introduce multiple MemoryBlocks...

Github user kiszk commented on the issue: https://github.com/apache/spark/pull/19222 ping @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20753: [SPARK-23582][SQL] StaticInvoke should support interpret...

Github user kiszk commented on the issue: https://github.com/apache/spark/pull/20753 ping @hvanhovell --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20797: [SPARK-23583][SQL] Invoke should support interpreted exe...

Github user kiszk commented on the issue: https://github.com/apache/spark/pull/20797 ping @hvanhovell --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20850: [SPARK-23713][SQL] Cleanup UnsafeWriter and BufferHolder...

Github user kiszk commented on the issue: https://github.com/apache/spark/pull/20850 ping @hvanhovell --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20636: [SPARK-23415][SQL][TEST] Make behavior of BufferHolderSp...

Github user kiszk commented on the issue: https://github.com/apache/spark/pull/20636 ping @gatorsmile and @hvanhovell --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark pull request #20774: [SPARK-23549][SQL] Cast to timestamp when compari...

Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20774 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20774: [SPARK-23549][SQL] Cast to timestamp when comparing time...

Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20774 LGTM Thanks! Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20849: [SPARK-23723] New charset option for json datasource

Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20849 @MaxGekk @HyukjinKwon What are the status of this PR? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark pull request #20885: [SPARK-23724][SPARK-23765][SQL] Line separator fo...

Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20885#discussion_r176958504 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala --- @@ -85,6 +85,38 @@ private[sql] class JSONOptions(

[GitHub] spark pull request #20885: [SPARK-23724][SPARK-23765][SQL] Line separator fo...

Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20885#discussion_r176958338 --- Diff: python/pyspark/sql/readwriter.py --- @@ -176,7 +176,7 @@ def json(self, path, schema=None, primitivesAsString=None, prefersDecimal=None,

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20877 Since both PRs are ready for review, let us review both and see which one is better --- - To unsubscribe, e-mail:

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20877 Yeah. `recordDelimiter` is better based on the semantics. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #20756: [SPARK-23593][SQL] Add interpreted execution for ...

Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/20756#discussion_r176956857 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ObjectExpressionsSuite.scala --- @@ -68,6 +68,23 @@ class

[GitHub] spark pull request #20756: [SPARK-23593][SQL] Add interpreted execution for ...

Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/20756#discussion_r176956802 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala --- @@ -1261,8 +1261,42 @@ case class

[GitHub] spark pull request #20756: [SPARK-23593][SQL] Add interpreted execution for ...

Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/20756#discussion_r176956651 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala --- @@ -1261,8 +1261,39 @@ case class

[GitHub] spark issue #20835: [HOT-FIX] Fix SparkOutOfMemoryError: Unable to acquire 2...

Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/20835 I have cherry-picked this into branch-2.3 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20774: [SPARK-23549][SQL] Cast to timestamp when comparing time...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20774 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88574/ Test PASSed. ---

[GitHub] spark issue #20774: [SPARK-23549][SQL] Cast to timestamp when comparing time...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20774 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20774: [SPARK-23549][SQL] Cast to timestamp when comparing time...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20774 **[Test build #88574 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88574/testReport)** for PR 20774 at commit

[GitHub] spark issue #20861: [SPARK-23599][SQL] Use RandomUUIDGenerator in Uuid expre...

Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/20861 @viirya I have backported #20817 to 2.3 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20877 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20877 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88572/ Test PASSed. ---

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20877 **[Test build #88572 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88572/testReport)** for PR 20877 at commit

[GitHub] spark issue #20811: [SPARK-23668][K8S] Add config option for passing through...

Github user liyinan926 commented on the issue: https://github.com/apache/spark/pull/20811 @foxish @mccheah can you help merge this? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20894: [SPARK-23786][SQL] Checking column names of csv headers

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20894 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88571/ Test PASSed. ---

[GitHub] spark issue #20894: [SPARK-23786][SQL] Checking column names of csv headers

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20894 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20894: [SPARK-23786][SQL] Checking column names of csv headers

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20894 **[Test build #88571 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88571/testReport)** for PR 20894 at commit

[GitHub] spark issue #20900: [SPARK-23645][MINOR][DOCS][PYTHON] Add docs RE `pandas_u...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20900 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20900: [SPARK-23645][MINOR][DOCS][PYTHON] Add docs RE `pandas_u...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20900 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88573/ Test PASSed. ---

[GitHub] spark issue #20900: [SPARK-23645][MINOR][DOCS][PYTHON] Add docs RE `pandas_u...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20900 **[Test build #88573 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88573/testReport)** for PR 20900 at commit

[GitHub] spark issue #20900: [SPARK-23645][MINOR][DOCS][PYTHON] Add docs RE `pandas_u...

Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20900 > One general question: how do we tend to think about the py2/3 split for api quirks/features? Must everything that is added for py3 also be functional in py2? ideally, is there

[GitHub] spark pull request #20894: [SPARK-23786][SQL] Checking column names of csv h...

Github user MaxGekk commented on a diff in the pull request: https://github.com/apache/spark/pull/20894#discussion_r176949956 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVOptions.scala --- @@ -150,6 +150,12 @@ class CSVOptions(

[GitHub] spark issue #20900: [SPARK-23645][MINOR][DOCS][PYTHON] Add docs RE `pandas_u...

Github user mstewart141 commented on the issue: https://github.com/apache/spark/pull/20900 Many (though not all, I don't think `callable`s are impacted) of the limitations of pandas_udf relative to UDF in this domain are due to the fact that `pandas_udf` doesn't allow for keyword

[GitHub] spark pull request #20894: [SPARK-23786][SQL] Checking column names of csv h...

Github user MaxGekk commented on a diff in the pull request: https://github.com/apache/spark/pull/20894#discussion_r176949718 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/UnivocityParser.scala --- @@ -289,27 +294,52 @@ private[csv] object

[GitHub] spark issue #20774: [SPARK-23549][SQL] Cast to timestamp when comparing time...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20774 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20774: [SPARK-23549][SQL] Cast to timestamp when comparing time...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20774 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1745/

[GitHub] spark pull request #20787: [MINOR][DOCS] Documenting months_between directio...

Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20787#discussion_r176949408 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala --- @@ -1117,11 +1117,23 @@ case class

[GitHub] spark issue #20774: [SPARK-23549][SQL] Cast to timestamp when comparing time...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20774 **[Test build #88574 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88574/testReport)** for PR 20774 at commit

[GitHub] spark issue #20900: [SPARK-23645][MINOR][DOCS][PYTHON] Add docs RE `pandas_u...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20900 **[Test build #88573 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88573/testReport)** for PR 20900 at commit

[GitHub] spark issue #20894: [SPARK-23786][SQL] Checking column names of csv headers

Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/20894 @gatorsmile For example, Spark is case sensitive for jsons: ``` cat ./case_sesitive.json {"FIELD1": 1} ``` If schema is specified with field name "field1": ``` val

[GitHub] spark pull request #20267: [SPARK-23068][BUILD][RELEASE][WIP] doc build erro...

Github user felixcheung closed the pull request at: https://github.com/apache/spark/pull/20267 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #20798: [SPARK-23645][MINOR][DOCS][PYTHON] Add docs RE `p...

Github user mstewart141 closed the pull request at: https://github.com/apache/spark/pull/20798 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20798: [SPARK-23645][MINOR][DOCS][PYTHON] Add docs RE `pandas_u...

Github user mstewart141 commented on the issue: https://github.com/apache/spark/pull/20798 see https://github.com/apache/spark/pull/20900 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark pull request #20851: [SPARK-23727][SQL] Support for pushing down filte...

Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/20851#discussion_r176948516 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -353,6 +353,13 @@ object SQLConf { .booleanConf

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20877 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1744/

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20877 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20877 **[Test build #88572 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88572/testReport)** for PR 20877 at commit

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

Github user rxin commented on the issue: https://github.com/apache/spark/pull/20877 We can also change both if they haven’t been released yet. On Sun, Mar 25, 2018 at 10:37 AM Maxim Gekk wrote: > @gatorsmile

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20877 Correct me if I am wrong. None of renaming or adding more flexible functionality to the line separator blocks this PR, right? Even if we go renaming, we should do it for text datasource

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/20877 @gatorsmile The PR has been already submitted: https://github.com/apache/spark/pull/20885 . Frankly speaking I would prefer another name for the option like we discussed before:

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20877 He submitted this - https://github.com/apache/spark/pull/20885 and I believe we need more feedback and another review iteration. ---

[GitHub] spark pull request #20787: [MINOR][DOCS] Documenting months_between directio...

Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20787#discussion_r176946746 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala --- @@ -1117,11 +1117,23 @@ case class

[GitHub] spark pull request #20851: [SPARK-23727][SQL] Support for pushing down filte...

Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20851#discussion_r176946273 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -353,6 +353,13 @@ object SQLConf { .booleanConf

[GitHub] spark issue #20894: [SPARK-23786][SQL] Checking column names of csv headers

Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20894 I just checked the JIRA https://issues.apache.org/jira/browse/SPARK-23786, I think CSV should follow the other format (e.g., parquet), right? ---

[GitHub] spark pull request #20894: [SPARK-23786][SQL] Checking column names of csv h...

Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20894#discussion_r176946135 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/UnivocityParser.scala --- @@ -289,27 +294,52 @@ private[csv] object

[GitHub] spark pull request #20894: [SPARK-23786][SQL] Checking column names of csv h...

Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20894#discussion_r176946115 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVOptions.scala --- @@ -150,6 +150,12 @@ class CSVOptions(

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20877 @MaxGekk Will you submit a PR for addressing the comment https://github.com/apache/spark/pull/20877#issuecomment-375622342 in the next few weeks? If so, we can hold this PR. ---

[GitHub] spark pull request #20877: [SPARK-23765][SQL] Supports custom line separator...

Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20877#discussion_r176945815 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamReader.scala --- @@ -268,6 +268,8 @@ final class DataStreamReader

[GitHub] spark issue #19616: [SPARK-22404][YARN] Provide an option to use unmanaged A...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19616 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20343: [SPARK-23167][SQL] Add TPCDS queries v2.7 in TPCD...

Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20343 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20343: [SPARK-23167][SQL] Add TPCDS queries v2.7 in TPCDSQueryS...

Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20343 Thanks! Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20894: [SPARK-23786][SQL] Checking column names of csv headers

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20894 **[Test build #88571 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88571/testReport)** for PR 20894 at commit

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/20877 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20877 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88570/ Test PASSed. ---

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20877 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20877 **[Test build #88570 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88570/testReport)** for PR 20877 at commit

[GitHub] spark issue #20897: [MINOR][DOC] Fix a few markdown typos

Github user Lemonjing commented on the issue: https://github.com/apache/spark/pull/20897 @HyukjinKwon The description of ElementwiseProduct is obvious, and i think "`Qu8T948*1#`" is a mistake, if not, "`Qu8T948*1#`" is a input vector and 1 is scaling vector, and this is not

  1   2   >