[GitHub] spark issue #20525: [SPARK-23271[SQL] Parquet output contains only _SUCCESS ...

2018-03-07 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20525 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #20768: [BACKPORT] [SPARK-23525] [SQL] Support ALTER TABLE CHANG...

2018-03-07 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20768 cc @cloud-fan can we merge this to 2.2? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20768: [BACKPORT] [SPARK-23525] [SQL] Support ALTER TABL...

2018-03-07 Thread jiangxb1987
GitHub user jiangxb1987 opened a pull request: https://github.com/apache/spark/pull/20768 [BACKPORT] [SPARK-23525] [SQL] Support ALTER TABLE CHANGE COLUMN COMMENT for external hive table ## What changes were proposed in this pull request? The following query doesn't

[GitHub] spark issue #20045: [Spark-22360][SQL][TEST] Add unit tests for Window Speci...

2018-03-05 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20045 Sorry for the delay, I'll check the results later this week. --- - To unsubscribe, e-mail: reviews-uns

[GitHub] spark pull request #20649: [SPARK-23462][SQL] improve missing field error me...

2018-03-04 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20649#discussion_r172086269 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala --- @@ -284,7 +286,8 @@ case class StructType(fields: Array

[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...

2018-03-04 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20649 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #20649: [SPARK-23462][SQL] improve missing field error me...

2018-03-04 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20649#discussion_r172080837 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala --- @@ -284,7 +286,8 @@ case class StructType(fields: Array

[GitHub] spark pull request #20706: [SPARK-23550][core] Cleanup `Utils`.

2018-03-02 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20706#discussion_r171881215 --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala --- @@ -1876,17 +1822,6 @@ private[spark] object Utils extends Logging

[GitHub] spark pull request #20706: [SPARK-23550][core] Cleanup `Utils`.

2018-03-02 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20706#discussion_r171879418 --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala --- @@ -1018,70 +1016,18 @@ private[spark] object Utils extends Logging

[GitHub] spark pull request #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE CO...

2018-02-28 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20696#discussion_r171439915 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala --- @@ -314,8 +314,8 @@ case class AlterTableChangeColumnCommand

[GitHub] spark issue #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE COLUMN CO...

2018-02-28 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20696 cc @skliarpawlo @gatorsmile @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE CO...

2018-02-28 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20696#discussion_r171285332 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala --- @@ -1597,6 +1597,7 @@ abstract class DDLSuite extends

[GitHub] spark pull request #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE CO...

2018-02-28 Thread jiangxb1987
GitHub user jiangxb1987 opened a pull request: https://github.com/apache/spark/pull/20696 [SPARK-23525] [SQL] Support ALTER TABLE CHANGE COLUMN COMMENT for external hive table ## What changes were proposed in this pull request? The following query doesn't work as exp

[GitHub] spark issue #20693: [SPARK-23523] [SQL] [FOLLOWUP] Minor refactor of Optimiz...

2018-02-28 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20693 cc @gatorsmile @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #20693: [SPARK-23523] [SQL] [FOLLOWUP] Minor refactor of ...

2018-02-28 Thread jiangxb1987
GitHub user jiangxb1987 opened a pull request: https://github.com/apache/spark/pull/20693 [SPARK-23523] [SQL] [FOLLOWUP] Minor refactor of OptimizeMetadataOnlyQuery ## What changes were proposed in this pull request? Inside `OptimizeMetadataOnlyQuery.getPartitionAttrs

[GitHub] spark issue #20670: [SPARK-23405] Generate additional constraints for Join's...

2018-02-28 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20670 LGTM only nits --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #20670: [SPARK-23405] Generate additional constraints for...

2018-02-28 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20670#discussion_r171201415 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/QueryPlanConstraints.scala --- @@ -22,21 +22,30 @@ import

[GitHub] spark issue #20667: [SPARK-23508][CORE] Fix BlockmanagerId in case blockMana...

2018-02-27 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20667 In case the same `BlockManagerId` being created multiple times, this cache will ensure we always use the first one that is created, which make it possible for the rest `BlockManagerId

[GitHub] spark pull request #20667: [SPARK-23508][CORE] Fix BlockmanagerId in case bl...

2018-02-27 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20667#discussion_r171136135 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala --- @@ -132,10 +133,17 @@ private[spark] object BlockManagerId

[GitHub] spark pull request #20667: [SPARK-23508][CORE] Fix BlockmanagerId in case bl...

2018-02-27 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20667#discussion_r170884966 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala --- @@ -132,10 +133,15 @@ private[spark] object BlockManagerId

[GitHub] spark issue #15717: [SPARK-17910][SQL] Allow users to update the comment of ...

2018-02-27 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/15717 Please file a JIRA issue if you think there is unexpected behavior. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #15717: [SPARK-17910][SQL] Allow users to update the comment of ...

2018-02-27 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/15717 Can you also show the result of DESC TABLE before and after the command? --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #20667: [SPARK-23508][CORE] Fix BlockmanagerId in case bl...

2018-02-27 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20667#discussion_r170865897 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala --- @@ -132,10 +133,15 @@ private[spark] object BlockManagerId

[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

2018-02-26 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20664 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #20670: [SPARK-23405] Generate additional constraints for Join's...

2018-02-26 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20670 You shall also add test cases. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark issue #20663: [SPARK-23501][UI] Refactor AllStagesPage in order to avo...

2018-02-26 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20663 @gengliangwang Mind take a look? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands

[GitHub] spark issue #20611: [SPARK-23425][SQL]When wild card is been used in load co...

2018-02-26 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20611 QQ: how does hive behave on the same/similar sql command? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #20670: [SPARK-23405] Add constranits

2018-02-26 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20670 Also, a better title for this PR would be: ``` Generate additional constraints for Join's children ``` --- ---

[GitHub] spark issue #20670: [SPARK-23405] Add constranits

2018-02-26 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20670 Agree with that @cloud-fan proposed to have constraints for a plan and the children. However, that requires a relative wider change as well as a find set of test cases, please don't be hes

[GitHub] spark pull request #19033: [SPARK-21811][SQL]Inconsistency when finding the ...

2018-02-26 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/19033#discussion_r170547020 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -150,9 +150,27 @@ object TypeCoercion

[GitHub] spark issue #20667: [SPARK-23508][CORE] Use softreference for BlockmanagerId...

2018-02-26 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20667 Had a offline chat with @cloud-fan and we feel `com.google.common.cache.Cache` should be used here. You can find a example at `CodeGenerator.cache

[GitHub] spark issue #20673: [SPARK-23515] Use input/output streams for large events ...

2018-02-25 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20673 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #20664: [SPARK-23496][CORE] Locality of coalesced partiti...

2018-02-25 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20664#discussion_r170456391 --- Diff: core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala --- @@ -1129,6 +1129,36 @@ class RDDSuite extends SparkFunSuite with

[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

2018-02-25 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20664 This LGTM overall, just some nits. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20664: [SPARK-23496][CORE] Locality of coalesced partiti...

2018-02-25 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20664#discussion_r170456095 --- Diff: core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala --- @@ -1129,6 +1129,36 @@ class RDDSuite extends SparkFunSuite with

[GitHub] spark pull request #20664: [SPARK-23496][CORE] Locality of coalesced partiti...

2018-02-25 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20664#discussion_r170456039 --- Diff: core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala --- @@ -1129,6 +1129,36 @@ class RDDSuite extends SparkFunSuite with

[GitHub] spark pull request #20664: [SPARK-23496][CORE] Locality of coalesced partiti...

2018-02-25 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20664#discussion_r170455929 --- Diff: core/src/main/scala/org/apache/spark/rdd/CoalescedRDD.scala --- @@ -266,17 +266,15 @@ private class DefaultPartitionCoalescer(val

[GitHub] spark pull request #20649: [SPARK-23462][SQL] improve missing field error me...

2018-02-25 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20649#discussion_r170452170 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/StructTypeSuite.scala --- @@ -0,0 +1,43 @@ +/* + * Licensed to the Apache Software

[GitHub] spark issue #20667: [SPARK-23508][CORE] Use timeStampedHashMap for Blockmana...

2018-02-25 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20667 Can we use WeakReference here to keep cached `BlockManagerId` ? --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #20667: [SPARK-23508][CORE] Use timeStampedHashMap for Bl...

2018-02-25 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20667#discussion_r170451578 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala --- @@ -132,10 +133,15 @@ private[spark] object BlockManagerId

[GitHub] spark issue #20553: [SPARK-23285][K8S] Add a config property for specifying ...

2018-02-23 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20553 IIUC the `spark.kubernetes.executor.cores` here is just a special case for `spark.executor.cores`, for k8s backend, you shall still have to handle float values if you're to read the val

[GitHub] spark issue #20553: [SPARK-23285][K8S] Add a config property for specifying ...

2018-02-23 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20553 also cc @cloud-fan @jerryshao --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark issue #20553: [SPARK-23285][K8S] Add a config property for specifying ...

2018-02-23 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20553 How do we plan to support dynamic allocation with k8s? Should we read `spark.executor.cores` or `spark.kubernetes.executor.cores

[GitHub] spark issue #20628: [SPARK-23449][K8S] Preserve extraJavaOptions ordering

2018-02-22 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20628 also cc @vanzin to verify this change. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20652: [SPARK-23476][CORE] Generate secret in local mode...

2018-02-22 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20652#discussion_r169996395 --- Diff: core/src/test/scala/org/apache/spark/SecurityManagerSuite.scala --- @@ -440,23 +440,41 @@ class SecurityManagerSuite extends SparkFunSuite

[GitHub] spark pull request #20546: [SPARK-20659][Core] Removing sc.getExecutorStorag...

2018-02-14 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20546#discussion_r168147073 --- Diff: core/src/main/java/org/apache/spark/SparkExecutorInfo.java --- @@ -30,4 +30,8 @@ int port(); long cacheSize(); int

[GitHub] spark pull request #20546: [SPARK-20659][Core] Removing sc.getExecutorStorag...

2018-02-14 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20546#discussion_r168144704 --- Diff: core/src/main/java/org/apache/spark/SparkExecutorInfo.java --- @@ -30,4 +30,8 @@ int port(); long cacheSize(); int

[GitHub] spark pull request #20546: [SPARK-20659][Core] Removing sc.getExecutorStorag...

2018-02-14 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20546#discussion_r168142034 --- Diff: core/src/main/java/org/apache/spark/SparkExecutorInfo.java --- @@ -30,4 +30,8 @@ int port(); long cacheSize(); int

[GitHub] spark pull request #20546: [SPARK-20659][Core] Removing sc.getExecutorStorag...

2018-02-14 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20546#discussion_r168136979 --- Diff: core/src/main/java/org/apache/spark/SparkExecutorInfo.java --- @@ -30,4 +30,8 @@ int port(); long cacheSize(); int

[GitHub] spark issue #20555: [SPARK-23366] Improve hot reading path in ReadAheadInput...

2018-02-14 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20555 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #20568: [SPARK-23381][CORE] Murmur3 hash generates a different v...

2018-02-14 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20568 How about add a new config to control whether to use the new Murmur3 hash function and have that default turned off? We also have to document the change explicitly. WDYT @gatorsmile

[GitHub] spark pull request #20601: [SPARK-23413][UI] Fix sorting tasks by Host / Exe...

2018-02-14 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20601#discussion_r168103464 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala --- @@ -963,33 +965,60 @@ private[ui] class TaskPagedTable( private

[GitHub] spark pull request #20601: [SPARK-23413][UI] Fix sorting tasks by Host / Exe...

2018-02-14 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20601#discussion_r168102644 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala --- @@ -963,33 +965,60 @@ private[ui] class TaskPagedTable( private

[GitHub] spark issue #20596: [SPARK-23404][CORE]When the underlying buffers are direc...

2018-02-14 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20596 +1 on knowing the case you want to fix here. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #20589: [SPARK-23394][UI] In RDD storage page show the executor ...

2018-02-14 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20589 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #20589: [SPARK-23394][UI] In RDD storage page show the ex...

2018-02-14 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20589#discussion_r168100501 --- Diff: core/src/main/scala/org/apache/spark/ui/storage/RDDPage.scala --- @@ -198,7 +200,10 @@ private[ui] class BlockDataSource

[GitHub] spark issue #20543: [SPARK-23357][CORE] 'SHOW TABLE EXTENDED LIKE pattern=ST...

2018-02-13 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20543 Personally I don't think it add any value to tell whether a table is partitioned, but I don't feel strong on this. --- --

[GitHub] spark issue #20555: [SPARK-23366] Improve hot reading path in ReadAheadInput...

2018-02-13 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20555 It would also be great if we can add some unit tests on the read ahead stream model. --- - To unsubscribe, e-mail: reviews

[GitHub] spark issue #20532: [SPARK-23353][CORE] Allow ExecutorMetricsUpdate events t...

2018-02-13 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20532 I'm still wondering whether event log is supposed to work this way, that as the source for customized analysis. I really feel we shall need some event/metrics logging framework that serve

[GitHub] spark issue #19077: [SPARK-21860][core]Improve memory reuse for heap memory ...

2018-02-08 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/19077 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #19077: [SPARK-21860][core]Improve memory reuse for heap memory ...

2018-02-08 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/19077 seems jenkins is very tired now, let's try later :) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.or

[GitHub] spark pull request #20449: [SPARK-23040][CORE]: Returns interruptible iterat...

2018-02-08 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20449#discussion_r166857525 --- Diff: core/src/main/scala/org/apache/spark/shuffle/BlockStoreShuffleReader.scala --- @@ -104,9 +104,18 @@ private[spark] class

[GitHub] spark issue #20527: [SPARK-23348][SQL] append data using saveAsTable should ...

2018-02-07 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20527 LGTM only some nits. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #20527: [SPARK-23348][SQL] append data using saveAsTable ...

2018-02-07 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20527#discussion_r166853858 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala --- @@ -132,6 +134,32 @@ class InMemoryCatalogedDDLSuite

[GitHub] spark pull request #20527: [SPARK-23348][SQL] append data using saveAsTable ...

2018-02-07 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20527#discussion_r166852743 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala --- @@ -346,37 +349,11 @@ case class PreprocessTableInsertion

[GitHub] spark pull request #20516: [SPARK-23343][CORE][TEST] Increase the exception ...

2018-02-07 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20516#discussion_r166849307 --- Diff: core/src/test/scala/org/apache/spark/network/netty/NettyBlockTransferServiceSuite.scala --- @@ -77,16 +79,53 @@ class

[GitHub] spark issue #20516: [SPARK-23343][CORE][TEST] Increase the exception test fo...

2018-02-07 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20516 Does this PR add any value or fix any bugs? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #20532: [SPARK-23353][CORE] Allow ExecutorMetricsUpdate events t...

2018-02-07 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20532 I'm also worried that if we want to sample more events in the future, we have to add more configs following this way, which doesn't sound like a perfe

[GitHub] spark pull request #20532: [SPARK-23353][CORE] Allow ExecutorMetricsUpdate e...

2018-02-07 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20532#discussion_r166833429 --- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala --- @@ -53,10 +53,21 @@ package object config { .booleanConf

[GitHub] spark issue #20532: [SPARK-23353][CORE] Allow ExecutorMetricsUpdate events t...

2018-02-07 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20532 Emmm... in case we want to sample more events, does that means we shall add a new config for each event sampling? --- - To

[GitHub] spark pull request #20525: [SPARK-23271[SQL] Parquet output contains only _S...

2018-02-07 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20525#discussion_r166830970 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala --- @@ -190,9 +190,13 @@ object FileFormatWriter

[GitHub] spark issue #20274: [SPARK-20120][SQL][FOLLOW-UP] Better way to support spar...

2018-02-07 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20274 @liufengdb Can you verify this one? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20502: [SPARK-23330][WebUI] Spark UI SQL executions page...

2018-02-04 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20502#discussion_r165890435 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/ui/AllExecutionsPage.scala --- @@ -179,7 +179,7 @@ private[ui] abstract class

[GitHub] spark issue #20502: [SPARK-23330][WebUI] Spark UI SQL executions page throws...

2018-02-04 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20502 cc @vanzin @zsxwing @cloud-fan @sameeragarwal --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark pull request #20502: [SPARK-23330][WebUI] Spark UI SQL executions page...

2018-02-03 Thread jiangxb1987
GitHub user jiangxb1987 opened a pull request: https://github.com/apache/spark/pull/20502 [SPARK-23330][WebUI] Spark UI SQL executions page throws NPE ## What changes were proposed in this pull request? Spark SQL executions page throws the following error and the page

[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...

2018-02-01 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20460 This may require more changes than it appears to be, agree we may need more tests to ensure it don't break anything, so +1 on target it to 2.4 instead of

[GitHub] spark pull request #20481: [SPARK-23307][WEBUI]Sort jobs/stages/tasks/querie...

2018-02-01 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20481#discussion_r165544206 --- Diff: core/src/main/scala/org/apache/spark/status/storeTypes.scala --- @@ -73,6 +73,8 @@ private[spark] class JobDataWrapper( @JsonIgnore

[GitHub] spark pull request #20481: [SPARK-23307][WEBUI]Sort jobs/stages/tasks/querie...

2018-02-01 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20481#discussion_r165544232 --- Diff: core/src/main/scala/org/apache/spark/status/storeTypes.scala --- @@ -90,6 +92,8 @@ private[spark] class StageDataWrapper( @JsonIgnore

[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...

2018-02-01 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20460 Yea, we should either move the check to elsewhere or modify it, to avoid potential failure. --- - To unsubscribe, e-mail

[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...

2018-02-01 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20460 Actually I think this may fail some check (though may not throw exceptions) for instance this one: https://github.com/apache/spark/blob/9678941f54ebc5db935ed8d694e502086e2a31c0/core/src/main

[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...

2018-02-01 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20460 Do we also want to update the comment of `SPARK_EXECUTOR_CORES` in `spark-env.sh` ? --- - To unsubscribe, e-mail: reviews

[GitHub] spark issue #20422: [SPARK-23253][Core][Shuffle]Only write shuffle temporary...

2018-02-01 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20422 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #20422: [SPARK-23253][Core][Shuffle]Only write shuffle te...

2018-01-31 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20422#discussion_r165250968 --- Diff: core/src/test/scala/org/apache/spark/shuffle/sort/IndexShuffleBlockResolverSuite.scala --- @@ -89,26 +96,39 @@ class

[GitHub] spark pull request #20422: [SPARK-23253][Core][Shuffle]Only write shuffle te...

2018-01-31 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20422#discussion_r165250833 --- Diff: core/src/test/scala/org/apache/spark/shuffle/sort/IndexShuffleBlockResolverSuite.scala --- @@ -89,26 +96,39 @@ class

[GitHub] spark pull request #20422: [SPARK-23253][Core][Shuffle]Only write shuffle te...

2018-01-31 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20422#discussion_r165249708 --- Diff: core/src/test/scala/org/apache/spark/shuffle/sort/IndexShuffleBlockResolverSuite.scala --- @@ -17,7 +17,7 @@ package

[GitHub] spark issue #20450: [SPARK-23280][SQL] add map type support to ColumnVector

2018-01-31 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20450 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #20463: [SQL][MINOR] Inline SpecifiedWindowFrame.defaultW...

2018-01-31 Thread jiangxb1987
GitHub user jiangxb1987 opened a pull request: https://github.com/apache/spark/pull/20463 [SQL][MINOR] Inline SpecifiedWindowFrame.defaultWindowFrame(). ## What changes were proposed in this pull request? SpecifiedWindowFrame.defaultWindowFrame(hasOrderSpecification

[GitHub] spark pull request #20361: [SPARK-23188][SQL] Make vectorized columar reader...

2018-01-31 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20361#discussion_r165234841 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.java --- @@ -49,8 +49,9 @@ * After

[GitHub] spark pull request #20433: [SPARK-23264][SQL] Support interval values withou...

2018-01-31 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20433#discussion_r165233944 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala --- @@ -561,8 +561,11 @@ class

[GitHub] spark issue #20450: [SPARK-23280][SQL] add map type support to ColumnVector

2018-01-31 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20450 LGTM only some nits and naming issues. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20450: [SPARK-23280][SQL] add map type support to Column...

2018-01-31 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20450#discussion_r165231647 --- Diff: sql/core/src/main/java/org/apache/spark/sql/vectorized/ColumnarMap.java --- @@ -0,0 +1,53 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #20450: [SPARK-23280][SQL] add map type support to Column...

2018-01-31 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20450#discussion_r165231132 --- Diff: sql/core/src/main/java/org/apache/spark/sql/vectorized/ColumnVector.java --- @@ -215,10 +215,18 @@ public final ColumnarRow getStruct(int

[GitHub] spark pull request #20450: [SPARK-23280][SQL] add map type support to Column...

2018-01-31 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20450#discussion_r165230542 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OffHeapColumnVector.java --- @@ -530,7 +530,7 @@ public int putByteArray

[GitHub] spark issue #20450: [SPARK-23280][SQL] add map type support to ColumnVector

2018-01-31 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20450 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #20422: [SPARK-23253][Core][Shuffle]Only write shuffle te...

2018-01-31 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20422#discussion_r165226127 --- Diff: core/src/test/scala/org/apache/spark/shuffle/sort/IndexShuffleBlockResolverSuite.scala --- @@ -133,4 +133,65 @@ class

[GitHub] spark pull request #20422: [SPARK-23253][Core][Shuffle]Only write shuffle te...

2018-01-31 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20422#discussion_r165225733 --- Diff: core/src/test/scala/org/apache/spark/shuffle/sort/IndexShuffleBlockResolverSuite.scala --- @@ -123,7 +123,7 @@ class

[GitHub] spark issue #20327: [SPARK-12963][CORE] NM host for driver end points

2018-01-31 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20327 cc @vanzin since you are expert on this topic. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark pull request #20327: [SPARK-12963][CORE] NM host for driver end points

2018-01-31 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20327#discussion_r165193521 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -744,7 +744,9 @@ object SparkSubmit extends CommandLineUtils with

[GitHub] spark issue #20400: [SPARK-23084][PYTHON]Add unboundedPreceding(), unbounded...

2018-01-30 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20400 LGTM only one nit --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #20400: [SPARK-23084][PYTHON]Add unboundedPreceding(), un...

2018-01-30 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20400#discussion_r164950347 --- Diff: python/pyspark/sql/window.py --- @@ -124,16 +126,20 @@ def rangeBetween(start, end): values directly. :param

<    1   2   3   4   5   6   7   8   9   10   >