[jira] [Commented] (SPARK-27192) spark.task.cpus should be less or equal than spark.task.cpus when use static executor allocation

2019-03-31 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806208#comment-16806208 ] Lijia Liu commented on SPARK-27192: --- https://github.com/apache/spark/pull/24261 > spark.task.cpus

[jira] [Updated] (SPARK-27192) spark.task.cpus should be less or equal than spark.task.cpus when use static executor allocation

2019-03-18 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijia Liu updated SPARK-27192: -- Description: When use dynamic executor allocation, if we set spark.executor.cores small than 

[jira] [Created] (SPARK-27192) spark.task.cpus should be less or equal than spark.task.cpus when use static executor allocation

2019-03-18 Thread Lijia Liu (JIRA)
Lijia Liu created SPARK-27192: - Summary: spark.task.cpus should be less or equal than spark.task.cpus when use static executor allocation Key: SPARK-27192 URL: https://issues.apache.org/jira/browse/SPARK-27192

[jira] [Updated] (SPARK-26705) UnsafeHashedRelation changed after serialization

2019-01-23 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijia Liu updated SPARK-26705: -- Summary: UnsafeHashedRelation changed after serialization (was: UnsafeHashedRelation ) >

[jira] [Commented] (SPARK-26705) UnsafeHashedRelation

2019-01-23 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749951#comment-16749951 ] Lijia Liu commented on SPARK-26705: --- cc [~cloud_fan] cc [~davies] Do you have any ideas? >

[jira] [Updated] (SPARK-26705) UnsafeHashedRelation changed after broadcasted

2019-01-23 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijia Liu updated SPARK-26705: -- Summary: UnsafeHashedRelation changed after broadcasted (was: UnsafeHashedRelation changed after

[jira] [Created] (SPARK-26705) UnsafeHashedRelation

2019-01-23 Thread Lijia Liu (JIRA)
Lijia Liu created SPARK-26705: - Summary: UnsafeHashedRelation Key: SPARK-26705 URL: https://issues.apache.org/jira/browse/SPARK-26705 Project: Spark Issue Type: Bug Components: SQL

[jira] [Commented] (SPARK-25211) speculation and fetch failed result in hang of job

2018-08-23 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590267#comment-16590267 ] Lijia Liu commented on SPARK-25211: --- In https://issues.apache.org/jira/browse/SPARK-23948, This issue

[jira] [Updated] (SPARK-25211) speculation and fetch failed result in hang of job

2018-08-23 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijia Liu updated SPARK-25211: -- Description: In current `DAGScheduler.handleTaskCompletion` code, when a shuffleMapStage with job

[jira] [Created] (SPARK-25211) speculation and fetch failed result in hang of job

2018-08-23 Thread Lijia Liu (JIRA)
Lijia Liu created SPARK-25211: - Summary: speculation and fetch failed result in hang of job Key: SPARK-25211 URL: https://issues.apache.org/jira/browse/SPARK-25211 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-24809) Serializing LongHashedRelation in executor may result in data error

2018-07-15 Thread Lijia Liu (JIRA)
Lijia Liu created SPARK-24809: - Summary: Serializing LongHashedRelation in executor may result in data error Key: SPARK-24809 URL: https://issues.apache.org/jira/browse/SPARK-24809 Project: Spark

[jira] [Created] (SPARK-24098) ScriptTransformationExec should wait process exiting before output iterator finish

2018-04-26 Thread Lijia Liu (JIRA)
Lijia Liu created SPARK-24098: - Summary: ScriptTransformationExec should wait process exiting before output iterator finish Key: SPARK-24098 URL: https://issues.apache.org/jira/browse/SPARK-24098

[jira] [Updated] (SPARK-24012) Union of map and other compatible column

2018-04-18 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijia Liu updated SPARK-24012: -- Description: Union of map and other compatible column result in unresolved operator 'Union; exception

[jira] [Updated] (SPARK-24012) Union of map and other compatible column

2018-04-18 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijia Liu updated SPARK-24012: -- Description: Union of map and other compatible column result in unresolved operator 'Union; exception

[jira] [Created] (SPARK-24012) Union of map and other compatible column

2018-04-18 Thread Lijia Liu (JIRA)
Lijia Liu created SPARK-24012: - Summary: Union of map and other compatible column Key: SPARK-24012 URL: https://issues.apache.org/jira/browse/SPARK-24012 Project: Spark Issue Type: Bug

[jira] [Comment Edited] (SPARK-5498) [SPARK-SQL]when the partition schema does not match table schema,it throws java.lang.ClassCastException and so on

2018-03-16 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401701#comment-16401701 ] Lijia Liu edited comment on SPARK-5498 at 3/16/18 10:10 AM: In our cluster, we

[jira] [Reopened] (SPARK-5498) [SPARK-SQL]when the partition schema does not match table schema,it throws java.lang.ClassCastException and so on

2018-03-16 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijia Liu reopened SPARK-5498: -- In our cluster, we use hive 1.2, spark 2.2, hadoop 2.7. When we read hive table use spark, we will get the

[jira] [Updated] (SPARK-5498) [SPARK-SQL]when the partition schema does not match table schema,it throws java.lang.ClassCastException and so on

2018-03-16 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijia Liu updated SPARK-5498: - Affects Version/s: 2.2.0 Fix Version/s: 3.0.0 > [SPARK-SQL]when the partition schema does not

[jira] [Created] (SPARK-22987) UnsafeExternalSorter cases OOM when invoking `getIterator` function.

2018-01-07 Thread Lijia Liu (JIRA)
Lijia Liu created SPARK-22987: - Summary: UnsafeExternalSorter cases OOM when invoking `getIterator` function. Key: SPARK-22987 URL: https://issues.apache.org/jira/browse/SPARK-22987 Project: Spark

[jira] [Issue Comment Deleted] (SPARK-22598) ExecutorAllocationManager does not requests new executors when executor fail and target has not change

2017-12-13 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijia Liu updated SPARK-22598: -- Comment: was deleted (was: duplicate with SPARK-22598) > ExecutorAllocationManager does not requests

[jira] [Comment Edited] (SPARK-22598) ExecutorAllocationManager does not requests new executors when executor fail and target has not change

2017-12-13 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16288966#comment-16288966 ] Lijia Liu edited comment on SPARK-22598 at 12/13/17 9:34 AM: - duplicate with

[jira] [Updated] (SPARK-22598) ExecutorAllocationManager does not requests new executors when executor fail and target has not change

2017-12-13 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijia Liu updated SPARK-22598: -- Fix Version/s: 2.2.1 duplicate with SPARK22598 > ExecutorAllocationManager does not requests new

[jira] [Closed] (SPARK-22598) ExecutorAllocationManager does not requests new executors when executor fail and target has not change

2017-11-29 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijia Liu closed SPARK-22598. - > ExecutorAllocationManager does not requests new executors when executor fail > and target has not change

[jira] [Resolved] (SPARK-22598) ExecutorAllocationManager does not requests new executors when executor fail and target has not change

2017-11-29 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijia Liu resolved SPARK-22598. --- Resolution: Unresolved > ExecutorAllocationManager does not requests new executors when executor

[jira] [Updated] (SPARK-22598) ExecutorAllocationManager does not requests new executors when executor fail and target has not change

2017-11-24 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijia Liu updated SPARK-22598: -- Summary: ExecutorAllocationManager does not requests new executors when executor fail and target has

[jira] [Updated] (SPARK-22598) ExecutorAllocationManager hang when executor fail and target has not change

2017-11-24 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijia Liu updated SPARK-22598: -- Summary: ExecutorAllocationManager hang when executor fail and target has not change (was:

[jira] [Created] (SPARK-22598) ExecutorAllocationManager hang when executor fail and numExecutorsTarget has not change

2017-11-23 Thread Lijia Liu (JIRA)
Lijia Liu created SPARK-22598: - Summary: ExecutorAllocationManager hang when executor fail and numExecutorsTarget has not change Key: SPARK-22598 URL: https://issues.apache.org/jira/browse/SPARK-22598

[jira] [Created] (SPARK-22469) Accuracy problem in comparison with string and numeric

2017-11-08 Thread Lijia Liu (JIRA)
Lijia Liu created SPARK-22469: - Summary: Accuracy problem in comparison with string and numeric Key: SPARK-22469 URL: https://issues.apache.org/jira/browse/SPARK-22469 Project: Spark Issue

[jira] [Updated] (SPARK-22233) filter out empty InputSplit in HadoopRDD

2017-10-10 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijia Liu updated SPARK-22233: -- Description: Sometimes, Hive will create an empty table with many empty files, Spark use the

[jira] [Updated] (SPARK-22233) filter out empty InputSplit in HadoopRDD

2017-10-10 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijia Liu updated SPARK-22233: -- Description: Sometimes, Hive will create an empty table with many empty files, Spark use the

[jira] [Created] (SPARK-22233) filter out empty InputSplit in HadoopRDD

2017-10-09 Thread Lijia Liu (JIRA)
Lijia Liu created SPARK-22233: - Summary: filter out empty InputSplit in HadoopRDD Key: SPARK-22233 URL: https://issues.apache.org/jira/browse/SPARK-22233 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-22155) Generates a simple default alias for the function in select

2017-09-28 Thread Lijia Liu (JIRA)
Lijia Liu created SPARK-22155: - Summary: Generates a simple default alias for the function in select Key: SPARK-22155 URL: https://issues.apache.org/jira/browse/SPARK-22155 Project: Spark Issue

[jira] [Comment Edited] (SPARK-22144) ExchangeCoordinator will not combine the partitions of an 0 sized pre-shuffle

2017-09-28 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183884#comment-16183884 ] Lijia Liu edited comment on SPARK-22144 at 9/28/17 10:11 AM: - [~yhuai] Would

[jira] [Updated] (SPARK-22144) ExchangeCoordinator will not combine the partitions of an 0 sized pre-shuffle

2017-09-28 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijia Liu updated SPARK-22144: -- Component/s: (was: Spark Core) SQL > ExchangeCoordinator will not combine the

[jira] [Commented] (SPARK-22144) ExchangeCoordinator will not combine the partitions of an 0 sized pre-shuffle

2017-09-28 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183884#comment-16183884 ] Lijia Liu commented on SPARK-22144: --- [~yhuai]: Please look at this issue! > ExchangeCoordinator will

[jira] [Commented] (SPARK-22144) ExchangeCoordinator will not combine the partitions of an 0 sized pre-shuffle

2017-09-28 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183883#comment-16183883 ] Lijia Liu commented on SPARK-22144: --- cc: Yin Huai Please take a look at this issue. >

[jira] [Issue Comment Deleted] (SPARK-22144) ExchangeCoordinator will not combine the partitions of an 0 sized pre-shuffle

2017-09-28 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijia Liu updated SPARK-22144: -- Comment: was deleted (was: cc: Yin Huai Please take a look at this issue.) > ExchangeCoordinator

[jira] [Issue Comment Deleted] (SPARK-22144) ExchangeCoordinator will not combine the partitions of an 0 sized pre-shuffle

2017-09-27 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijia Liu updated SPARK-22144: -- Comment: was deleted (was: cc Yin Huai) > ExchangeCoordinator will not combine the partitions of an 0

[jira] [Commented] (SPARK-22144) ExchangeCoordinator will not combine the partitions of an 0 sized pre-shuffle

2017-09-27 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16182317#comment-16182317 ] Lijia Liu commented on SPARK-22144: --- cc Yin Huai > ExchangeCoordinator will not combine the partitions

[jira] [Created] (SPARK-22144) ExchangeCoordinator will not combine the partitions of an 0 sized pre-shuffle

2017-09-27 Thread Lijia Liu (JIRA)
Lijia Liu created SPARK-22144: - Summary: ExchangeCoordinator will not combine the partitions of an 0 sized pre-shuffle Key: SPARK-22144 URL: https://issues.apache.org/jira/browse/SPARK-22144 Project:

[jira] [Closed] (SPARK-20475) Whether use "broadcast join" depends on hive configuration

2017-04-26 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijia Liu closed SPARK-20475. - Resolution: Fixed Fix Version/s: 2.0.3 > Whether use "broadcast join" depends on hive

[jira] [Commented] (SPARK-20475) Whether use "broadcast join" depends on hive configuration

2017-04-26 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985995#comment-15985995 ] Lijia Liu commented on SPARK-20475: --- [~ZenWzh] This issue is duplicated to

[jira] [Created] (SPARK-20475) Whether use "broadcast join" depends on hive configuration

2017-04-26 Thread Lijia Liu (JIRA)
Lijia Liu created SPARK-20475: - Summary: Whether use "broadcast join" depends on hive configuration Key: SPARK-20475 URL: https://issues.apache.org/jira/browse/SPARK-20475 Project: Spark Issue