[jira] [Created] (SPARK-37101) In class ShuffleBlockPusher, use config instead of key

2021-10-23 Thread jinhai (Jira)
jinhai created SPARK-37101: -- Summary: In class ShuffleBlockPusher, use config instead of key Key: SPARK-37101 URL: https://issues.apache.org/jira/browse/SPARK-37101 Project: Spark Issue Type:

[jira] [Commented] (SPARK-37006) MapStatus adds localDirs to avoid the rpc request by method getHostLocalDirs when shuffle reading

2021-10-22 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17432914#comment-17432914 ] jinhai commented on SPARK-37006: hi [~Ngone51], can you review this issue for me? > MapStatus adds

[jira] [Issue Comment Deleted] (SPARK-37006) MapStatus adds localDirs to avoid the rpc request by method getHostLocalDirs when shuffle reading

2021-10-22 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jinhai updated SPARK-37006: --- Comment: was deleted (was: hi [~Ngone51], can you review this issue for me?) > MapStatus adds localDirs to

[jira] [Comment Edited] (SPARK-37006) MapStatus adds localDirs to avoid the rpc request by method getHostLocalDirs when shuffle reading

2021-10-22 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17429079#comment-17429079 ] jinhai edited comment on SPARK-37006 at 10/22/21, 11:01 AM: Or whether we

[jira] [Updated] (SPARK-37006) MapStatus adds localDirs to avoid the rpc request by method getHostLocalDirs when shuffle reading

2021-10-22 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jinhai updated SPARK-37006: --- Description: When executing the ShuffleBlockFetcherIterator.fetchHostLocalBlocks method, in order to

[jira] [Commented] (SPARK-37006) MapStatus adds localDirs to avoid the rpc request by method getHostLocalDirs when shuffle reading

2021-10-20 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17430994#comment-17430994 ] jinhai commented on SPARK-37006: hi [~Ngone51], can you review this issue for me? > MapStatus adds

[jira] [Resolved] (SPARK-37029) Modify the assignment logic of dirFetchRequests variables

2021-10-20 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jinhai resolved SPARK-37029. Resolution: Won't Fix > Modify the assignment logic of dirFetchRequests variables >

[jira] [Created] (SPARK-37029) Modify the assignment logic of dirFetchRequests variables

2021-10-17 Thread jinhai (Jira)
jinhai created SPARK-37029: -- Summary: Modify the assignment logic of dirFetchRequests variables Key: SPARK-37029 URL: https://issues.apache.org/jira/browse/SPARK-37029 Project: Spark Issue Type:

[jira] [Commented] (SPARK-37006) MapStatus adds localDirs to avoid the rpc request by method getHostLocalDirs when shuffle reading

2021-10-14 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17429079#comment-17429079 ] jinhai commented on SPARK-37006: Or whether we can generate localDirs based on appId and execId, just

[jira] [Commented] (SPARK-37006) MapStatus adds localDirs to avoid the rpc request by method getHostLocalDirs when shuffle reading

2021-10-14 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17428714#comment-17428714 ] jinhai commented on SPARK-37006: hi [~cloud_fan]，can you review this issue for me? > MapStatus adds

[jira] [Created] (SPARK-37006) MapStatus adds localDirs to avoid the rpc request by method getHostLocalDirs when shuffle reading

2021-10-14 Thread jinhai (Jira)
jinhai created SPARK-37006: -- Summary: MapStatus adds localDirs to avoid the rpc request by method getHostLocalDirs when shuffle reading Key: SPARK-37006 URL: https://issues.apache.org/jira/browse/SPARK-37006

[jira] [Resolved] (SPARK-36855) Uniformly execute the mergeContinuousShuffleBlockIdsIfNeeded method first

2021-10-13 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jinhai resolved SPARK-36855. Resolution: Not A Problem > Uniformly execute the mergeContinuousShuffleBlockIdsIfNeeded method first >

[jira] [Updated] (SPARK-36855) Uniformly execute the mergeContinuousShuffleBlockIdsIfNeeded method first

2021-09-26 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jinhai updated SPARK-36855: --- Description: In the ShuffleBlockFetcherIterator.partitionBlocksByFetchMode method, both local and

[jira] [Updated] (SPARK-36855) Uniformly execute the mergeContinuousShuffleBlockIdsIfNeeded method first

2021-09-26 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jinhai updated SPARK-36855: --- Description: In the ShuffleBlockFetcherIterator.partitionBlocksByFetchMode method, both local and

[jira] [Created] (SPARK-36855) Uniformly execute the mergeContinuousShuffleBlockIdsIfNeeded method first

2021-09-26 Thread jinhai (Jira)
jinhai created SPARK-36855: -- Summary: Uniformly execute the mergeContinuousShuffleBlockIdsIfNeeded method first Key: SPARK-36855 URL: https://issues.apache.org/jira/browse/SPARK-36855 Project: Spark

[jira] [Commented] (SPARK-24194) HadoopFsRelation cannot overwrite a path that is also being read from

2020-12-03 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243056#comment-17243056 ] jinhai commented on SPARK-24194: Similar issue: https://issues.apache.org/jira/browse/SPARK-32838 >

[jira] [Updated] (SPARK-33628) Use the Hive.getPartitionsByNames method instead of Hive.getPartitions in the HiveClientImpl

2020-12-02 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jinhai updated SPARK-33628: --- Description: When partitions are tracked by the catalog, that will compute all custom partition locations,

[jira] [Updated] (SPARK-33628) Use the Hive.getPartitionsByNames method instead of Hive.getPartitions in the HiveClientImpl

2020-12-02 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jinhai updated SPARK-33628: --- Description: When partitions are tracked by the catalog, that will compute all custom partition locations,

[jira] [Comment Edited] (SPARK-33628) Use the Hive.getPartitionsByNames method instead of Hive.getPartitions in the HiveClientImpl

2020-12-02 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17242951#comment-17242951 ] jinhai edited comment on SPARK-33628 at 12/3/20, 6:40 AM: -- And,

[jira] [Comment Edited] (SPARK-33628) Use the Hive.getPartitionsByNames method instead of Hive.getPartitions in the HiveClientImpl

2020-12-02 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17242951#comment-17242951 ] jinhai edited comment on SPARK-33628 at 12/3/20, 6:38 AM: -- And,

[jira] [Commented] (SPARK-33628) Use the Hive.getPartitionsByNames method instead of Hive.getPartitions in the HiveClientImpl

2020-12-02 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17242951#comment-17242951 ] jinhai commented on SPARK-33628: And, Hive.getPartitions will cause MetaStoreClient close connection.

[jira] [Updated] (SPARK-33628) Use the Hive.getPartitionsByNames method instead of Hive.getPartitions in the HiveClientImpl

2020-12-02 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jinhai updated SPARK-33628: --- Description: When partitions are tracked by the catalog, that will compute all custom partition locations,

[jira] [Updated] (SPARK-33628) Use the Hive.getPartitionsByNames method instead of Hive.getPartitions in the HiveClientImpl

2020-12-02 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jinhai updated SPARK-33628: --- Description: When partitions are tracked by the catalog, will compute all custom partition locations that

[jira] [Updated] (SPARK-33628) Use the Hive.getPartitionsByNames method instead of Hive.getPartitions in the HiveClientImpl

2020-12-02 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jinhai updated SPARK-33628: --- Description: When partitions are tracked by the catalog, will compute all custom partition locations that

[jira] [Commented] (SPARK-33628) Use the Hive.getPartitionsByNames method instead of Hive.getPartitions in the HiveClientImpl

2020-12-02 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17242206#comment-17242206 ] jinhai commented on SPARK-33628: Hi, [~yumwang] Can you review this issue for me? > Use the

[jira] [Updated] (SPARK-33628) Use the Hive.getPartitionsByNames method instead of Hive.getPartitions in the HiveClientImpl

2020-12-02 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jinhai updated SPARK-33628: --- Attachment: (was: image-2020-12-02-16-56-36-274.png) > Use the Hive.getPartitionsByNames method instead

[jira] [Updated] (SPARK-33628) Use the Hive.getPartitionsByNames method instead of Hive.getPartitions in the HiveClientImpl

2020-12-02 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jinhai updated SPARK-33628: --- Description: When partitions are tracked by the catalog, compute all custom partition locations that may

[jira] [Updated] (SPARK-33628) Use the Hive.getPartitionsByNames method instead of Hive.getPartitions in the HiveClientImpl

2020-12-02 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jinhai updated SPARK-33628: --- Description: When partitions are tracked by the catalog, compute all custom partition locations that may

[jira] [Updated] (SPARK-33628) Use the Hive.getPartitionsByNames method instead of Hive.getPartitions in the HiveClientImpl

2020-12-02 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jinhai updated SPARK-33628: --- Description: When partitions are tracked by the catalog, compute all custom partition locations that may

[jira] [Updated] (SPARK-33628) Use the Hive.getPartitionsByNames method instead of Hive.getPartitions in the HiveClientImpl

2020-12-02 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jinhai updated SPARK-33628: --- Attachment: image-2020-12-02-16-57-43-619.png > Use the Hive.getPartitionsByNames method instead of

[jira] [Created] (SPARK-33628) Use the Hive.getPartitionsByNames method instead of Hive.getPartitions in the HiveClientImpl

2020-12-02 Thread jinhai (Jira)
jinhai created SPARK-33628: -- Summary: Use the Hive.getPartitionsByNames method instead of Hive.getPartitions in the HiveClientImpl Key: SPARK-33628 URL: https://issues.apache.org/jira/browse/SPARK-33628

[jira] [Updated] (SPARK-33628) Use the Hive.getPartitionsByNames method instead of Hive.getPartitions in the HiveClientImpl

2020-12-02 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jinhai updated SPARK-33628: --- Description: When partitions are tracked by the catalog, compute all custom partition locations that may

[jira] [Updated] (SPARK-33628) Use the Hive.getPartitionsByNames method instead of Hive.getPartitions in the HiveClientImpl

2020-12-02 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jinhai updated SPARK-33628: --- Attachment: image-2020-12-02-16-56-36-274.png > Use the Hive.getPartitionsByNames method instead of

[jira] [Created] (SPARK-33478) Allow overwrite a path that is also being read under dynamic partition overwrite

2020-11-18 Thread jinhai (Jira)
jinhai created SPARK-33478: -- Summary: Allow overwrite a path that is also being read under dynamic partition overwrite Key: SPARK-33478 URL: https://issues.apache.org/jira/browse/SPARK-33478 Project: Spark

[jira] [Updated] (SPARK-33187) Add a check on the number of returned partitions in the HiveShim#getPartitionsByFilter method

2020-11-02 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jinhai updated SPARK-33187: --- Description: In the method Shim#getPartitionsByFilter, when filter is empty or when the hive table has a

[jira] [Updated] (SPARK-33187) Add a check on the number of returned partitions in the HiveShim#getPartitionsByFilter method

2020-11-02 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jinhai updated SPARK-33187: --- Description: In the method Shim#getPartitionsByFilter, when filter is empty or when the hive table has a

[jira] [Updated] (SPARK-33187) Add a check on the number of returned partitions in the HiveShim#getPartitionsByFilter method

2020-10-19 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jinhai updated SPARK-33187: --- Description: In the method HiveShim#getPartitionsByFilter, when filter is empty or when the hive table has

[jira] [Created] (SPARK-33187) Add a check on the number of returned partitions in the HiveShim#getPartitionsByFilter method

2020-10-19 Thread jinhai (Jira)
jinhai created SPARK-33187: -- Summary: Add a check on the number of returned partitions in the HiveShim#getPartitionsByFilter method Key: SPARK-33187 URL: https://issues.apache.org/jira/browse/SPARK-33187

[jira] [Resolved] (SPARK-32475) java.lang.NoSuchMethodError: java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;

2020-07-30 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jinhai resolved SPARK-32475. Resolution: Not A Problem > java.lang.NoSuchMethodError: java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;

[jira] [Commented] (SPARK-32475) java.lang.NoSuchMethodError: java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;

2020-07-30 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17167690#comment-17167690 ] jinhai commented on SPARK-32475: My mistake. The default jdk version of maven is 14 not jdk8 >

[jira] [Updated] (SPARK-32475) java.lang.NoSuchMethodError: java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;

2020-07-30 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jinhai updated SPARK-32475: --- Attachment: image-2020-07-30-15-15-42-319.png > java.lang.NoSuchMethodError:

[jira] [Commented] (SPARK-32475) java.lang.NoSuchMethodError: java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;

2020-07-30 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17167689#comment-17167689 ] jinhai commented on SPARK-32475: !image-2020-07-30-15-15-42-319.png! > java.lang.NoSuchMethodError:

[jira] [Created] (SPARK-32475) java.lang.NoSuchMethodError: java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;

2020-07-28 Thread jinhai (Jira)
jinhai created SPARK-32475: -- Summary: java.lang.NoSuchMethodError: java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer; Key: SPARK-32475 URL: https://issues.apache.org/jira/browse/SPARK-32475 Project: Spark