[jira] [Comment Edited] (SPARK-11583) Make MapStatus use less memory uage

2015-11-11 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000187#comment-15000187 ] Kent Yao edited comment on SPARK-11583 at 11/11/15 10:21 AM: - [~imranr]

[jira] [Comment Edited] (SPARK-11583) Make MapStatus use less memory uage

2015-11-11 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000187#comment-15000187 ] Kent Yao edited comment on SPARK-11583 at 11/11/15 10:20 AM: - @lemire

[jira] [Commented] (SPARK-11583) Make MapStatus use less memory uage

2015-11-11 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000187#comment-15000187 ] Kent Yao commented on SPARK-11583: -- @lemire [~imranr] 1. test cases 1.1 sparse case: for each task

[jira] [Created] (SPARK-11583) Make MapStatus use less memory uage

2015-11-08 Thread Kent Yao (JIRA)
Kent Yao created SPARK-11583: Summary: Make MapStatus use less memory uage Key: SPARK-11583 URL: https://issues.apache.org/jira/browse/SPARK-11583 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-11583) Make MapStatus use less memory uage

2015-11-08 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14996147#comment-14996147 ] Kent Yao commented on SPARK-11583: -- [~srowen] MapStatus stores BlockManagerId where it locates and

[jira] [Comment Edited] (SPARK-11583) Make MapStatus use less memory uage

2015-11-11 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15001420#comment-15001420 ] Kent Yao edited comment on SPARK-11583 at 11/12/15 12:48 AM: - Are these

[jira] [Commented] (SPARK-11583) Make MapStatus use less memory uage

2015-11-11 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15001420#comment-15001420 ] Kent Yao commented on SPARK-11583: -- Are these methods {noformat} r.add(0,20) {noformat} and

[jira] [Commented] (SPARK-11583) Make MapStatus use less memory uage

2015-11-10 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999835#comment-14999835 ] Kent Yao commented on SPARK-11583: -- [~imranr] [~dlemire] The JVM I use : -Xmx256M -Xms32M bq. scala>

[jira] [Comment Edited] (SPARK-11583) Make MapStatus use less memory uage

2015-11-10 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999835#comment-14999835 ] Kent Yao edited comment on SPARK-11583 at 11/11/15 2:32 AM: [~imranr]

[jira] [Comment Edited] (SPARK-11583) Make MapStatus use less memory uage

2015-11-10 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999835#comment-14999835 ] Kent Yao edited comment on SPARK-11583 at 11/11/15 2:33 AM: [~imranr]

[jira] [Commented] (SPARK-11583) Make MapStatus use less memory uage

2015-11-09 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14996206#comment-14996206 ] Kent Yao commented on SPARK-11583: -- 1. The _Compressed_ in MapStatus means *Array[Long]* ->

[jira] [Commented] (SPARK-11583) Make MapStatus use less memory uage

2015-11-09 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14996415#comment-14996415 ] Kent Yao commented on SPARK-11583: -- I have reviewed my code and merged two classes. And I get confused

[jira] [Comment Edited] (SPARK-11583) Make MapStatus use less memory uage

2015-11-09 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14996415#comment-14996415 ] Kent Yao edited comment on SPARK-11583 at 11/9/15 12:00 PM: I have reviewed

[jira] [Commented] (SPARK-10399) Off Heap Memory Access for non-JVM libraries (C++)

2016-02-02 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15129928#comment-15129928 ] Kent Yao commented on SPARK-10399: -- How does this work go? > Off Heap Memory Access for non-JVM

[jira] [Updated] (SPARK-19626) Configuration `spark.yarn.credentials.updateTime` takes no effect

2017-02-16 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-19626: - Description: In [SPARK-14743|https://github.com/apache/spark/pull/14065], we introduced a configurable

[jira] [Created] (SPARK-19626) Configuration `spark.yarn.credentials.updateTime` takes no effect

2017-02-16 Thread Kent Yao (JIRA)
Kent Yao created SPARK-19626: Summary: Configuration `spark.yarn.credentials.updateTime` takes no effect Key: SPARK-19626 URL: https://issues.apache.org/jira/browse/SPARK-19626 Project: Spark

[jira] [Updated] (SPARK-17517) Improve generated Code for BroadcastHashJoinExec

2016-09-13 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-17517: - Description: For current `BroadcastHashJoinExec`, we generate join code for key is not unique like

[jira] [Created] (SPARK-17517) Improve generated Code for BroadcastHashJoinExec

2016-09-12 Thread Kent Yao (JIRA)
Kent Yao created SPARK-17517: Summary: Improve generated Code for BroadcastHashJoinExec Key: SPARK-17517 URL: https://issues.apache.org/jira/browse/SPARK-17517 Project: Spark Issue Type:

[jira] [Updated] (SPARK-17517) Improve generated Code for BroadcastHashJoinExec

2016-09-12 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-17517: - Description: For current `BroadcastHashJoinExec`, we generate join code for key is not unique like

[jira] [Updated] (SPARK-17517) Improve generated Code for BroadcastHashJoinExec

2016-09-12 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-17517: - Target Version/s: (was: 2.1.0) External issue ID: (was: 12795) Fix Version/s: (was:

[jira] [Created] (SPARK-17655) [SQL]Remove unused variables declarations and definations in a WholeStageCodeGened stage

2016-09-23 Thread Kent Yao (JIRA)
Kent Yao created SPARK-17655: Summary: [SQL]Remove unused variables declarations and definations in a WholeStageCodeGened stage Key: SPARK-17655 URL: https://issues.apache.org/jira/browse/SPARK-17655

[jira] [Updated] (SPARK-17655) Remove unused variables declarations and definations in a WholeStageCodeGened stage

2016-09-23 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-17655: - Summary: Remove unused variables declarations and definations in a WholeStageCodeGened stage (was:

[jira] [Commented] (SPARK-11150) Dynamic partition pruning

2016-08-24 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434458#comment-15434458 ] Kent Yao commented on SPARK-11150: -- [~davies] Any progress on this issue? Thanks. Any problem with this

[jira] [Created] (SPARK-19043) Make SparkSQLSessionManager more configurable

2016-12-31 Thread Kent Yao (JIRA)
Kent Yao created SPARK-19043: Summary: Make SparkSQLSessionManager more configurable Key: SPARK-19043 URL: https://issues.apache.org/jira/browse/SPARK-19043 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-20096) Expose the real queue name not null while using --verbose

2017-03-25 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941712#comment-15941712 ] Kent Yao commented on SPARK-20096: -- [~sowen] example added > Expose the real queue name not null while

[jira] [Updated] (SPARK-20096) Expose the real queue name not null while using --verbose

2017-03-25 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-20096: - Summary: Expose the real queue name not null while using --verbose (was: Expose the real queue name not

[jira] [Created] (SPARK-20096) Expose the real queue name not null using --verbose

2017-03-25 Thread Kent Yao (JIRA)
Kent Yao created SPARK-20096: Summary: Expose the real queue name not null using --verbose Key: SPARK-20096 URL: https://issues.apache.org/jira/browse/SPARK-20096 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-20096) Expose the real queue name not null while using --verbose

2017-03-25 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-20096: - Description: while submit apps with -v or --verbose, we can print the right queue name, but if we set a

[jira] [Updated] (SPARK-20060) Support Standalone visiting secured HDFS

2017-03-25 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-20060: - Description: h1. Brief design h2. Introductions The basic issue for Standalone mode to visit kerberos

[jira] [Created] (SPARK-19997) proxy-user failed connecting to a kerberos configured metastore

2017-03-17 Thread Kent Yao (JIRA)
Kent Yao created SPARK-19997: Summary: proxy-user failed connecting to a kerberos configured metastore Key: SPARK-19997 URL: https://issues.apache.org/jira/browse/SPARK-19997 Project: Spark

[jira] [Updated] (SPARK-19997) proxy-user failed connecting to a kerberos configured metastore

2017-03-17 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-19997: - Description: Start runing spark-sql via proxy-user on a kerberos configured hadoop cluster and

[jira] [Updated] (SPARK-19997) proxy-user failed connecting to a kerberos configured metastore

2017-03-17 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-19997: - Description: Start runing spark-sql via proxy-user on a kerberos configured hadoop cluster and

[jira] [Created] (SPARK-20060) Spark On Non-Yarn Mode with Kerberized HDFS ProxyUser Fails Talking to Hive MetaStore

2017-03-22 Thread Kent Yao (JIRA)
Kent Yao created SPARK-20060: Summary: Spark On Non-Yarn Mode with Kerberized HDFS ProxyUser Fails Talking to Hive MetaStore Key: SPARK-20060 URL: https://issues.apache.org/jira/browse/SPARK-20060

[jira] [Created] (SPARK-21637) `hive.metastore.warehouse` in --hiveconf is not respected

2017-08-04 Thread Kent Yao (JIRA)
Kent Yao created SPARK-21637: Summary: `hive.metastore.warehouse` in --hiveconf is not respected Key: SPARK-21637 URL: https://issues.apache.org/jira/browse/SPARK-21637 Project: Spark Issue

[jira] [Created] (SPARK-21428) CliSessionState never be recognized because of IsolatedClientLoader

2017-07-16 Thread Kent Yao (JIRA)
Kent Yao created SPARK-21428: Summary: CliSessionState never be recognized because of IsolatedClientLoader Key: SPARK-21428 URL: https://issues.apache.org/jira/browse/SPARK-21428 Project: Spark

[jira] [Created] (SPARK-21449) Hive client's SessionState was not closed properly in HiveExternalCatalog

2017-07-17 Thread Kent Yao (JIRA)
Kent Yao created SPARK-21449: Summary: Hive client's SessionState was not closed properly in HiveExternalCatalog Key: SPARK-21449 URL: https://issues.apache.org/jira/browse/SPARK-21449 Project: Spark

[jira] [Created] (SPARK-21451) HiveConf in SparkSQLCLIDriver doesn't respect spark.hadoop.some.hive.variables

2017-07-18 Thread Kent Yao (JIRA)
Kent Yao created SPARK-21451: Summary: HiveConf in SparkSQLCLIDriver doesn't respect spark.hadoop.some.hive.variables Key: SPARK-21451 URL: https://issues.apache.org/jira/browse/SPARK-21451 Project:

[jira] [Created] (SPARK-21675) Add a navigation bar at the bottom of the Details for Stage Page

2017-08-09 Thread Kent Yao (JIRA)
Kent Yao created SPARK-21675: Summary: Add a navigation bar at the bottom of the Details for Stage Page Key: SPARK-21675 URL: https://issues.apache.org/jira/browse/SPARK-21675 Project: Spark

[jira] [Commented] (SPARK-21771) SparkSQLEnv creates a useless meta hive client

2017-08-18 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131828#comment-16131828 ] Kent Yao commented on SPARK-21771: -- https://github.com/apache/spark/pull/18983 git webhook doesn't work,

[jira] [Comment Edited] (SPARK-21771) SparkSQLEnv creates a useless meta hive client

2017-08-18 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131828#comment-16131828 ] Kent Yao edited comment on SPARK-21771 at 8/18/17 7:05 AM: ---

[jira] [Created] (SPARK-21771) SparkSQLEnv creates a useless meta hive client

2017-08-17 Thread Kent Yao (JIRA)
Kent Yao created SPARK-21771: Summary: SparkSQLEnv creates a useless meta hive client Key: SPARK-21771 URL: https://issues.apache.org/jira/browse/SPARK-21771 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-21906) No need to runAsSparkUser to switch UserGroupInformation in YARN mode

2017-09-04 Thread Kent Yao (JIRA)
Kent Yao created SPARK-21906: Summary: No need to runAsSparkUser to switch UserGroupInformation in YARN mode Key: SPARK-21906 URL: https://issues.apache.org/jira/browse/SPARK-21906 Project: Spark

[jira] [Updated] (SPARK-21906) No need to runAsSparkUser to switch UserGroupInformation in YARN mode

2017-09-04 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-21906: - Description: 1、The Yarn application‘s ugi is determined by the ugi launching it 2、 runAsSparkUser is

[jira] [Updated] (SPARK-21906) No need to runAsSparkUser to switch UserGroupInformation in YARN mode

2017-09-04 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-21906: - Description: 1、The Yarn application‘s ugi is determined by the ugi launching it 2、 runAsSparkUser is

[jira] [Updated] (SPARK-21906) No need to runAsSparkUser to switch UserGroupInformation in YARN mode

2017-09-04 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-21906: - Description: 1、The Yarn application‘s ugi is determined by the ugi launching it 2、 runAsSparkUser is

[jira] [Created] (SPARK-22224) Override toString of KeyValueGroupedDataset & RelationalGroupedDataset

2017-10-09 Thread Kent Yao (JIRA)
Kent Yao created SPARK-4: Summary: Override toString of KeyValueGroupedDataset & RelationalGroupedDataset Key: SPARK-4 URL: https://issues.apache.org/jira/browse/SPARK-4 Project: Spark

[jira] [Commented] (SPARK-22640) Can not switch python exec in executor side

2017-11-28 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270208#comment-16270208 ] Kent Yao commented on SPARK-22640: -- [~Jeff_Z] spark.yarn.appMasterEnv specified > Can not switch python

[jira] [Created] (SPARK-22640) Can not switch python exec in executor side

2017-11-28 Thread Kent Yao (JIRA)
Kent Yao created SPARK-22640: Summary: Can not switch python exec in executor side Key: SPARK-22640 URL: https://issues.apache.org/jira/browse/SPARK-22640 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-22463) Missing hadoop/hive/hbase/etc configuration files in SPARK_CONF_DIR to distributed archive

2017-11-06 Thread Kent Yao (JIRA)
Kent Yao created SPARK-22463: Summary: Missing hadoop/hive/hbase/etc configuration files in SPARK_CONF_DIR to distributed archive Key: SPARK-22463 URL: https://issues.apache.org/jira/browse/SPARK-22463

[jira] [Created] (SPARK-22466) SPARK_CONF_DIR is not is set by Spark's launch scripts with default value

2017-11-07 Thread Kent Yao (JIRA)
Kent Yao created SPARK-22466: Summary: SPARK_CONF_DIR is not is set by Spark's launch scripts with default value Key: SPARK-22466 URL: https://issues.apache.org/jira/browse/SPARK-22466 Project: Spark

[jira] [Created] (SPARK-22487) No usages of HIVE_EXECUTION_VERSION found in whole spark project

2017-11-09 Thread Kent Yao (JIRA)
Kent Yao created SPARK-22487: Summary: No usages of HIVE_EXECUTION_VERSION found in whole spark project Key: SPARK-22487 URL: https://issues.apache.org/jira/browse/SPARK-22487 Project: Spark

[jira] [Updated] (SPARK-24241) Do not fail fast when dynamic resource allocation enabled with 0 executor

2018-05-10 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-24241: - Summary: Do not fail fast when dynamic resource allocation enabled with 0 executor (was: Do not fail

[jira] [Created] (SPARK-24241) Do not fail fast when dynamic resource allocation enabled

2018-05-10 Thread Kent Yao (JIRA)
Kent Yao created SPARK-24241: Summary: Do not fail fast when dynamic resource allocation enabled Key: SPARK-24241 URL: https://issues.apache.org/jira/browse/SPARK-24241 Project: Spark Issue

[jira] [Created] (SPARK-23253) Only write shuffle temporary index file when there is not an existing one

2018-01-29 Thread Kent Yao (JIRA)
Kent Yao created SPARK-23253: Summary: Only write shuffle temporary index file when there is not an existing one Key: SPARK-23253 URL: https://issues.apache.org/jira/browse/SPARK-23253 Project: Spark

[jira] [Created] (SPARK-23295) Exclude Waring message when generating versions in make-distribution.sh

2018-01-31 Thread Kent Yao (JIRA)
Kent Yao created SPARK-23295: Summary: Exclude Waring message when generating versions in make-distribution.sh Key: SPARK-23295 URL: https://issues.apache.org/jira/browse/SPARK-23295 Project: Spark

[jira] [Updated] (SPARK-23295) Exclude Waring message when generating versions in make-distribution.sh

2018-01-31 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-23295: - Description: When we specified a wrong profile to make a spark distribution, such as -Phadoop1000, we

[jira] [Updated] (SPARK-23383) Make a distribution should exit with usage while detecting wrong options

2018-02-10 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-23383: - Description: {code:java} ./dev/make-distribution.sh --name ne-1.0.0-SNAPSHOT xyz --tgz -Phadoop-2.7

[jira] [Created] (SPARK-23383) Make a distribution should exit with usage while detecting wrong options

2018-02-10 Thread Kent Yao (JIRA)
Kent Yao created SPARK-23383: Summary: Make a distribution should exit with usage while detecting wrong options Key: SPARK-23383 URL: https://issues.apache.org/jira/browse/SPARK-23383 Project: Spark

[jira] [Created] (SPARK-22950) user classpath first cause no class found exception

2018-01-03 Thread Kent Yao (JIRA)
Kent Yao created SPARK-22950: Summary: user classpath first cause no class found exception Key: SPARK-22950 URL: https://issues.apache.org/jira/browse/SPARK-22950 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-22950) user classpath first cause no class found exception

2018-01-03 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-22950: - Description: {code:java} bin/spark-submit --class some.class.enbaled.hive.Support --master yarn

[jira] [Updated] (SPARK-25174) ApplicationMaster suspends when unregistering itself from RM with extreme large diagnostic message

2018-08-21 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-25174: - Description: We recently ran into SPARK-18016 which has been fixed in v2.3.0. This JIRA is not about

[jira] [Updated] (SPARK-25174) ApplicationMaster suspends when unregistering itself from RM with extreme large diagnostic message

2018-08-21 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-25174: - Description: We recently ran into SPARK-18016 which has been fixed in v2.3.0. This JIRA is not about

[jira] [Created] (SPARK-25174) ApplicationMaster suspends when unregistering itself from RM with extreme large diagnostic message

2018-08-21 Thread Kent Yao (JIRA)
Kent Yao created SPARK-25174: Summary: ApplicationMaster suspends when unregistering itself from RM with extreme large diagnostic message Key: SPARK-25174 URL: https://issues.apache.org/jira/browse/SPARK-25174

[jira] [Updated] (SPARK-25174) ApplicationMaster suspends when unregistering itself from RM with extreme large diagnostic message

2018-08-21 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-25174: - Description: We recently ran into SPARK-18016 which has been fixed in v2.3.0. This JIRA is not about

[jira] [Created] (SPARK-23639) SparkSQL CLI fails talk to Kerberized metastore when use proxy user

2018-03-09 Thread Kent Yao (JIRA)
Kent Yao created SPARK-23639: Summary: SparkSQL CLI fails talk to Kerberized metastore when use proxy user Key: SPARK-23639 URL: https://issues.apache.org/jira/browse/SPARK-23639 Project: Spark

[jira] [Created] (SPARK-27301) DStreamCheckpointData failed to clean up because it's fileSystem cached

2019-03-28 Thread Kent Yao (JIRA)
Kent Yao created SPARK-27301: Summary: DStreamCheckpointData failed to clean up because it's fileSystem cached Key: SPARK-27301 URL: https://issues.apache.org/jira/browse/SPARK-27301 Project: Spark

[jira] [Updated] (SPARK-27301) DStreamCheckpointData failed to clean up because it's fileSystem cached

2019-03-28 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-27301: - Description: The cached FileSystem's token will expire if no tokens explicitly are add into it.

[jira] [Created] (SPARK-27183) YarnConfiguration in ApplicationMaster should be cluster-specific

2019-03-17 Thread Kent Yao (JIRA)
Kent Yao created SPARK-27183: Summary: YarnConfiguration in ApplicationMaster should be cluster-specific Key: SPARK-27183 URL: https://issues.apache.org/jira/browse/SPARK-27183 Project: Spark

[jira] [Updated] (SPARK-26794) SparkSession enableHiveSupport does not point to hive but in-memory while the SparkContext exists

2019-01-31 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-26794: - Description: {code:java} public class SqlDemo { public static void main(final String[] args)

[jira] [Created] (SPARK-26794) SparkSession enableHiveSupport does not point to hive but in-memory while the SparkContext exists

2019-01-31 Thread Kent Yao (JIRA)
Kent Yao created SPARK-26794: Summary: SparkSession enableHiveSupport does not point to hive but in-memory while the SparkContext exists Key: SPARK-26794 URL: https://issues.apache.org/jira/browse/SPARK-26794

[jira] [Created] (SPARK-28143) Missing attribute name in `IN` expressions throws java.lang.UnsupportedOperationException: empty.init

2019-06-23 Thread Kent Yao (JIRA)
Kent Yao created SPARK-28143: Summary: Missing attribute name in `IN` expressions throws java.lang.UnsupportedOperationException: empty.init Key: SPARK-28143 URL: https://issues.apache.org/jira/browse/SPARK-28143

[jira] [Created] (SPARK-28895) Spark client process is unable to upload jars to hdfs while using ConfigMap not HADOOP_CONF_DIR

2019-08-28 Thread Kent Yao (Jira)
Kent Yao created SPARK-28895: Summary: Spark client process is unable to upload jars to hdfs while using ConfigMap not HADOOP_CONF_DIR Key: SPARK-28895 URL: https://issues.apache.org/jira/browse/SPARK-28895

[jira] [Created] (SPARK-28896) Spark client process is unable to upload jars to hdfs while using ConfigMap not HADOOP_CONF_DIR

2019-08-28 Thread Kent Yao (Jira)
Kent Yao created SPARK-28896: Summary: Spark client process is unable to upload jars to hdfs while using ConfigMap not HADOOP_CONF_DIR Key: SPARK-28896 URL: https://issues.apache.org/jira/browse/SPARK-28896

[jira] [Created] (SPARK-28947) Status logging occurs on every state change but not at an interval for liveness.

2019-09-02 Thread Kent Yao (Jira)
Kent Yao created SPARK-28947: Summary: Status logging occurs on every state change but not at an interval for liveness. Key: SPARK-28947 URL: https://issues.apache.org/jira/browse/SPARK-28947 Project:

[jira] [Updated] (SPARK-28949) Kubernetes CGroup leaking leads to Spark Pods hang in Pending status

2019-09-02 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-28949: - Description: After running Spark on k8s for a few days, some kubelet fails to create pod caused by

[jira] [Created] (SPARK-28949) Kubernetes CGroup leaking leads to Spark Pods hang in Pending status

2019-09-02 Thread Kent Yao (Jira)
Kent Yao created SPARK-28949: Summary: Kubernetes CGroup leaking leads to Spark Pods hang in Pending status Key: SPARK-28949 URL: https://issues.apache.org/jira/browse/SPARK-28949 Project: Spark

[jira] [Updated] (SPARK-28949) Kubernetes CGroup leaking leads to Spark Pods hang in Pending status

2019-09-02 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-28949: - Attachment: describe-driver-pod.txt > Kubernetes CGroup leaking leads to Spark Pods hang in Pending

[jira] [Updated] (SPARK-28949) Kubernetes CGroup leaking leads to Spark Pods hang in Pending status

2019-09-02 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-28949: - Attachment: describe-executor-pod.txt > Kubernetes CGroup leaking leads to Spark Pods hang in Pending

[jira] [Created] (SPARK-28992) Support update dependencies from hdfs when task run on executor pods

2019-09-05 Thread Kent Yao (Jira)
Kent Yao created SPARK-28992: Summary: Support update dependencies from hdfs when task run on executor pods Key: SPARK-28992 URL: https://issues.apache.org/jira/browse/SPARK-28992 Project: Spark

[jira] [Created] (SPARK-29021) NoSuchElementException: key not found: hostPath.spark-local-dir-5.options.path

2019-09-09 Thread Kent Yao (Jira)
Kent Yao created SPARK-29021: Summary: NoSuchElementException: key not found: hostPath.spark-local-dir-5.options.path Key: SPARK-29021 URL: https://issues.apache.org/jira/browse/SPARK-29021 Project:

[jira] [Updated] (SPARK-29021) NoSuchElementException: key not found: hostPath.spark-local-dir-5.options.path

2019-09-09 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-29021: - Description: Mount hostPath has an issue: {code:java} Exception in thread "main"

[jira] [Commented] (SPARK-29021) NoSuchElementException: key not found: hostPath.spark-local-dir-5.options.path

2019-09-09 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925603#comment-16925603 ] Kent Yao commented on SPARK-29021: -- it seems that need to configure options.path to an actual  host

[jira] [Updated] (SPARK-29257) All Task attempts scheduled to the same executor inevitably access the same bad disk

2019-09-26 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-29257: - Description: We have an HDFS/YARN cluster with about 2k~3k nodes, each node has 8T * 12 local disks

[jira] [Updated] (SPARK-29257) All Task attempts scheduled to the same executor inevitably access the same bad disk

2019-09-26 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-29257: - Attachment: image-2019-09-26-16-44-48-554.png > All Task attempts scheduled to the same executor

[jira] [Created] (SPARK-29257) All Task attempts scheduled to the same executor inevitably access the same bad disk

2019-09-26 Thread Kent Yao (Jira)
Kent Yao created SPARK-29257: Summary: All Task attempts scheduled to the same executor inevitably access the same bad disk Key: SPARK-29257 URL: https://issues.apache.org/jira/browse/SPARK-29257

[jira] [Created] (SPARK-29285) Temporary shuffle and local block should be able to handle disk failures

2019-09-29 Thread Kent Yao (Jira)
Kent Yao created SPARK-29285: Summary: Temporary shuffle and local block should be able to handle disk failures Key: SPARK-29285 URL: https://issues.apache.org/jira/browse/SPARK-29285 Project: Spark

[jira] [Commented] (SPARK-29257) All Task attempts scheduled to the same executor inevitably access the same bad disk

2019-09-26 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939075#comment-16939075 ] Kent Yao commented on SPARK-29257: -- this issue should be not a problem anymore after this fix - 

[jira] [Created] (SPARK-29287) Executors should not receive any offers before they are actually constructed

2019-09-29 Thread Kent Yao (Jira)
Kent Yao created SPARK-29287: Summary: Executors should not receive any offers before they are actually constructed Key: SPARK-29287 URL: https://issues.apache.org/jira/browse/SPARK-29287 Project: Spark

[jira] [Created] (SPARK-29653) Fix MICROS_PER_MONTH in IntervalUtils

2019-10-30 Thread Kent Yao (Jira)
Kent Yao created SPARK-29653: Summary: Fix MICROS_PER_MONTH in IntervalUtils Key: SPARK-29653 URL: https://issues.apache.org/jira/browse/SPARK-29653 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-29371) Support interval field values with fractional parts

2019-10-30 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962798#comment-16962798 ] Kent Yao commented on SPARK-29371: -- {code:java} postgres=# select interval '1.41 years 2.51 months 2.21

[jira] [Commented] (SPARK-29371) Support interval field values with fractional parts

2019-10-30 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962797#comment-16962797 ] Kent Yao commented on SPARK-29371: -- I am working on this > Support interval field values with

[jira] [Created] (SPARK-29722) Non reversed keywords should be able to be used in high order functions

2019-11-02 Thread Kent Yao (Jira)
Kent Yao created SPARK-29722: Summary: Non reversed keywords should be able to be used in high order functions Key: SPARK-29722 URL: https://issues.apache.org/jira/browse/SPARK-29722 Project: Spark

[jira] [Created] (SPARK-29737) Concat for array in Spark SQL is not the one in PostgreSQL but array_cat

2019-11-03 Thread Kent Yao (Jira)
Kent Yao created SPARK-29737: Summary: Concat for array in Spark SQL is not the one in PostgreSQL but array_cat Key: SPARK-29737 URL: https://issues.apache.org/jira/browse/SPARK-29737 Project: Spark

[jira] [Commented] (SPARK-28307) Support Interval comparison

2019-11-10 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971325#comment-16971325 ] Kent Yao commented on SPARK-28307: -- Resolved by the PR: [https://github.com/apache/spark/pull/26337] >

[jira] [Created] (SPARK-29774) Date and Timestamp type +/- null should be null as Postgres

2019-11-06 Thread Kent Yao (Jira)
Kent Yao created SPARK-29774: Summary: Date and Timestamp type +/- null should be null as Postgres Key: SPARK-29774 URL: https://issues.apache.org/jira/browse/SPARK-29774 Project: Spark Issue

[jira] [Created] (SPARK-29688) Support average with interval type values

2019-10-31 Thread Kent Yao (Jira)
Kent Yao created SPARK-29688: Summary: Support average with interval type values Key: SPARK-29688 URL: https://issues.apache.org/jira/browse/SPARK-29688 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-29663) Support sum with interval type values

2019-10-30 Thread Kent Yao (Jira)
Kent Yao created SPARK-29663: Summary: Support sum with interval type values Key: SPARK-29663 URL: https://issues.apache.org/jira/browse/SPARK-29663 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-29713) Support Interval Unit Abbreviations in Interval Literals

2019-11-01 Thread Kent Yao (Jira)
Kent Yao created SPARK-29713: Summary: Support Interval Unit Abbreviations in Interval Literals Key: SPARK-29713 URL: https://issues.apache.org/jira/browse/SPARK-29713 Project: Spark Issue Type:

[jira] [Created] (SPARK-29679) Make interval type camparable and orderable

2019-10-30 Thread Kent Yao (Jira)
Kent Yao created SPARK-29679: Summary: Make interval type camparable and orderable Key: SPARK-29679 URL: https://issues.apache.org/jira/browse/SPARK-29679 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-29888) New interval string parser parse '.111 seconds' to null

2019-11-13 Thread Kent Yao (Jira)
Kent Yao created SPARK-29888: Summary: New interval string parser parse '.111 seconds' to null Key: SPARK-29888 URL: https://issues.apache.org/jira/browse/SPARK-29888 Project: Spark Issue Type:

  1   2   3   4   5   6   7   8   9   10   >