[jira] [Updated] (SPARK-20869) Master should clear failed apps when worker down

[ https://issues.apache.org/jira/browse/SPARK-20869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lyc updated SPARK-20869: Description: In `Master.removeWorker`, master clears executor and driver state, but does not clear app state. App

[jira] [Updated] (SPARK-20869) Master should clear failed apps when worker down

[ https://issues.apache.org/jira/browse/SPARK-20869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lyc updated SPARK-20869: Description: In `Master.removeWorker`, master clears executor and driver state, but does not clear app state. App

[jira] [Commented] (SPARK-19900) [Standalone] Master registers application again when driver relaunched

[ https://issues.apache.org/jira/browse/SPARK-19900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049942#comment-16049942 ] lyc commented on SPARK-19900: - Hi, what's the meaning of JIRA id? I only know that my user name (and JIRA

[jira] [Commented] (SPARK-19878) Add hive configuration when initialize hive serde in InsertIntoHiveTable.scala

[ https://issues.apache.org/jira/browse/SPARK-19878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16038648#comment-16038648 ] lyc commented on SPARK-19878: - See

[jira] [Commented] (SPARK-20662) Block jobs that have greater than a configured number of tasks

[ https://issues.apache.org/jira/browse/SPARK-20662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16034282#comment-16034282 ] lyc commented on SPARK-20662: - Do you mean `mapreduce.job.running.map.limit`? The conf means `The maximum

[jira] [Comment Edited] (SPARK-20662) Block jobs that have greater than a configured number of tasks

[ https://issues.apache.org/jira/browse/SPARK-20662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16034282#comment-16034282 ] lyc edited comment on SPARK-20662 at 6/2/17 7:36 AM: - Do you mean

[jira] [Commented] (SPARK-20675) Support Index to skip when retrieval disk structure in CoGroupedRDD

[ https://issues.apache.org/jira/browse/SPARK-20675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16034256#comment-16034256 ] lyc commented on SPARK-20675: - What do you mean `StreamBuffer`? In commit `6d05c1` (at Jun 1/17), there is

[jira] [Commented] (SPARK-20365) Not so accurate classpath format for AM and Containers

[ https://issues.apache.org/jira/browse/SPARK-20365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16034176#comment-16034176 ] lyc commented on SPARK-20365: - Thanks for reviewing. > Not so accurate classpath format for AM and

[jira] [Commented] (SPARK-15682) Hive ORC partition write looks for root hdfs folder for existence

[ https://issues.apache.org/jira/browse/SPARK-15682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16034076#comment-16034076 ] lyc commented on SPARK-15682: - Hi, I tried this both for `orc` and `parquet`, and they both throws `path

[jira] [Updated] (SPARK-20869) Master should clear failed apps when worker down

[ https://issues.apache.org/jira/browse/SPARK-20869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lyc updated SPARK-20869: Summary: Master should clear failed apps when worker down (was: Master may clear failed apps when worker down) >

[jira] [Commented] (SPARK-20320) AnalysisException: Columns of grouping_id (count(value#17L)) does not match grouping columns (count(value#17L))

[ https://issues.apache.org/jira/browse/SPARK-20320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16027384#comment-16027384 ] lyc commented on SPARK-20320: - It seems `count("value")` should not be in `cube`, there should only be column

[jira] [Comment Edited] (SPARK-20660) Not able to merge Dataframes with different column orders

[ https://issues.apache.org/jira/browse/SPARK-20660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023230#comment-16023230 ] lyc edited comment on SPARK-20660 at 5/24/17 5:02 PM: -- Do you expect spark to fail

[jira] [Commented] (SPARK-20660) Not able to merge Dataframes with different column orders

[ https://issues.apache.org/jira/browse/SPARK-20660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023230#comment-16023230 ] lyc commented on SPARK-20660: - Do you expect spark to fail the query? > Not able to merge Dataframes with

[jira] [Updated] (SPARK-20869) Master may clear failed apps when worker down

[ https://issues.apache.org/jira/browse/SPARK-20869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lyc updated SPARK-20869: Summary: Master may clear failed apps when worker down (was: Master should clear failed apps when worker down) >

[jira] [Updated] (SPARK-20869) Master should clear failed apps when worker down

[ https://issues.apache.org/jira/browse/SPARK-20869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lyc updated SPARK-20869: Remaining Estimate: 2h (was: 24h) Original Estimate: 2h (was: 24h) > Master should clear failed apps when

[jira] [Updated] (SPARK-20869) Master should clear failed apps when worker down

[ https://issues.apache.org/jira/browse/SPARK-20869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lyc updated SPARK-20869: Description: In `Master.removeWorker`, master clears executor and driver state, but does not clear app state. App

[jira] [Updated] (SPARK-20869) Master should clear failed apps when worker down

[ https://issues.apache.org/jira/browse/SPARK-20869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lyc updated SPARK-20869: Description: In `Master.removeWorker`, master clears executor and driver state, but does not clear app state. App

[jira] [Created] (SPARK-20869) Master should clear failed apps when worker down

lyc created SPARK-20869: --- Summary: Master should clear failed apps when worker down Key: SPARK-20869 URL: https://issues.apache.org/jira/browse/SPARK-20869 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-20681) DataFram.Drop doesn't take effective, neither does error

[ https://issues.apache.org/jira/browse/SPARK-20681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021093#comment-16021093 ] lyc commented on SPARK-20681: - As said in spark source code, `drop` can only be used to drop top level

[jira] [Comment Edited] (SPARK-20744) Predicates with multiple columns do not work

[ https://issues.apache.org/jira/browse/SPARK-20744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019366#comment-16019366 ] lyc edited comment on SPARK-20744 at 5/22/17 9:32 AM: -- Why should {code}

[jira] [Commented] (SPARK-20744) Predicates with multiple columns do not work

[ https://issues.apache.org/jira/browse/SPARK-20744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019366#comment-16019366 ] lyc commented on SPARK-20744: - Why should `spark.range(10).selectExpr("id as a", "id as b").where("(a,b) in

[jira] [Commented] (SPARK-20808) External Table unnecessarily not created in Hive-compatible way

[ https://issues.apache.org/jira/browse/SPARK-20808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018366#comment-16018366 ] lyc commented on SPARK-20808: - Since 2.2.0, `createExternalTable` is deprecated and you should use

[jira] [Commented] (SPARK-20816) MetricsConfig doen't trim the properties file cause the exception very confused

[ https://issues.apache.org/jira/browse/SPARK-20816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018364#comment-16018364 ] lyc commented on SPARK-20816: - It seems hard for spark to change, because spark uses

[jira] [Comment Edited] (SPARK-19900) [Standalone] Master registers application again when driver relaunched

[ https://issues.apache.org/jira/browse/SPARK-19900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15999741#comment-15999741 ] lyc edited comment on SPARK-19900 at 5/7/17 9:19 AM: - I can successfully reproduce in

[jira] [Comment Edited] (SPARK-19900) [Standalone] Master registers application again when driver relaunched

[ https://issues.apache.org/jira/browse/SPARK-19900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15999741#comment-15999741 ] lyc edited comment on SPARK-19900 at 5/7/17 9:18 AM: - I can successfully reproduce in

[jira] [Commented] (SPARK-19900) [Standalone] Master registers application again when driver relaunched

[ https://issues.apache.org/jira/browse/SPARK-19900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15999741#comment-15999741 ] lyc commented on SPARK-19900: - I can successfully reproduce in spark master ( commit 63d90e7d, and spark