[jira] [Created] (SPARK-15480) Some InputInfo missed with window operation applied

2016-05-23 Thread Mao, Wei (JIRA)
Mao, Wei created SPARK-15480: Summary: Some InputInfo missed with window operation applied Key: SPARK-15480 URL: https://issues.apache.org/jira/browse/SPARK-15480 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-15220) Add hyperlink to "running application" and "completed application"

2016-05-09 Thread Mao, Wei (JIRA)
Mao, Wei created SPARK-15220: Summary: Add hyperlink to "running application" and "completed application" Key: SPARK-15220 URL: https://issues.apache.org/jira/browse/SPARK-15220 Project: Spark

[jira] [Commented] (SPARK-14688) pyspark textFileStream gzipped

2016-04-28 Thread Mao, Wei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15262436#comment-15262436 ] Mao, Wei commented on SPARK-14688: -- Actually, I used the script you provided in previous comment. I

[jira] [Commented] (SPARK-14688) pyspark textFileStream gzipped

2016-04-28 Thread Mao, Wei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15261747#comment-15261747 ] Mao, Wei commented on SPARK-14688: -- As I tested, gz file is supported in StreamingContext. About

[jira] [Created] (SPARK-14976) make StreamingContext.textFileStream support wildcard

2016-04-28 Thread Mao, Wei (JIRA)
Mao, Wei created SPARK-14976: Summary: make StreamingContext.textFileStream support wildcard Key: SPARK-14976 URL: https://issues.apache.org/jira/browse/SPARK-14976 Project: Spark Issue Type:

[jira] [Created] (SPARK-13758) Error message is misleading when RDD refer to null spark context

2016-03-08 Thread Mao, Wei (JIRA)
Mao, Wei created SPARK-13758: Summary: Error message is misleading when RDD refer to null spark context Key: SPARK-13758 URL: https://issues.apache.org/jira/browse/SPARK-13758 Project: Spark

[jira] [Closed] (SPARK-13737) Add getOrCreate method for HiveContext

2016-03-07 Thread Mao, Wei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mao, Wei closed SPARK-13737. Resolution: Won't Fix > Add getOrCreate method for HiveContext > -- >

[jira] [Commented] (SPARK-13737) Add getOrCreate method for HiveContext

2016-03-07 Thread Mao, Wei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15184263#comment-15184263 ] Mao, Wei commented on SPARK-13737: -- According to Reynold, databricks is actually going to deprecate

[jira] [Commented] (SPARK-13737) Add getOrCreate method for HiveContext

2016-03-07 Thread Mao, Wei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15184221#comment-15184221 ] Mao, Wei commented on SPARK-13737: -- HiveContext is heavily used by many users now, and enen many of them

[jira] [Comment Edited] (SPARK-13737) Add getOrCreate method for HiveContext

2016-03-07 Thread Mao, Wei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15184221#comment-15184221 ] Mao, Wei edited comment on SPARK-13737 at 3/8/16 2:07 AM: -- HiveContext is

[jira] [Created] (SPARK-13737) Add getOrCreate method for HiveContext

2016-03-07 Thread Mao, Wei (JIRA)
Mao, Wei created SPARK-13737: Summary: Add getOrCreate method for HiveContext Key: SPARK-13737 URL: https://issues.apache.org/jira/browse/SPARK-13737 Project: Spark Issue Type: New Feature

[jira] [Updated] (SPARK-13374) make it possible to create recoverable accumulator for streaming application

2016-02-18 Thread Mao, Wei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mao, Wei updated SPARK-13374: - Issue Type: New Feature (was: Bug) > make it possible to create recoverable accumulator for streaming

[jira] [Created] (SPARK-13374) make it possible to create recoverable accumulator for streaming application

2016-02-18 Thread Mao, Wei (JIRA)
Mao, Wei created SPARK-13374: Summary: make it possible to create recoverable accumulator for streaming application Key: SPARK-13374 URL: https://issues.apache.org/jira/browse/SPARK-13374 Project: Spark

[jira] [Updated] (SPARK-13222) checkpoint latest stateful RDD on graceful shutdown

2016-02-05 Thread Mao, Wei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mao, Wei updated SPARK-13222: - Description: In case the input source is not repeatable, and user don't want to enable WAL because of

[jira] [Updated] (SPARK-13222) checkpoint latest stateful RDD on graceful shutdown

2016-02-05 Thread Mao, Wei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mao, Wei updated SPARK-13222: - Component/s: Streaming > checkpoint latest stateful RDD on graceful shutdown >

[jira] [Updated] (SPARK-13222) checkpoint latest stateful RDD on graceful shutdown

2016-02-05 Thread Mao, Wei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mao, Wei updated SPARK-13222: - Description: The CheckpointInterval of DStream could be customized to multiple of BatchInteveal. So when

[jira] [Created] (SPARK-13222) checkpoint latest stateful RDD on graceful shutdown

2016-02-05 Thread Mao, Wei (JIRA)
Mao, Wei created SPARK-13222: Summary: checkpoint latest stateful RDD on graceful shutdown Key: SPARK-13222 URL: https://issues.apache.org/jira/browse/SPARK-13222 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-12864) initialize executorIdCounter after ApplicationMaster killed for max number of executor failures reached

2016-01-19 Thread Mao, Wei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107812#comment-15107812 ] Mao, Wei commented on SPARK-12864: -- I don't quite get your point here. With your new executor id idea,

[jira] [Created] (SPARK-12710) Create local CoGroup operator

2016-01-07 Thread Mao, Wei (JIRA)
Mao, Wei created SPARK-12710: Summary: Create local CoGroup operator Key: SPARK-12710 URL: https://issues.apache.org/jira/browse/SPARK-12710 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-12709) Create local Except operator

2016-01-07 Thread Mao, Wei (JIRA)
Mao, Wei created SPARK-12709: Summary: Create local Except operator Key: SPARK-12709 URL: https://issues.apache.org/jira/browse/SPARK-12709 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-12260) Graceful Shutdown with In-Memory State

2015-12-10 Thread Mao, Wei (JIRA)
Mao, Wei created SPARK-12260: Summary: Graceful Shutdown with In-Memory State Key: SPARK-12260 URL: https://issues.apache.org/jira/browse/SPARK-12260 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-12260) Graceful Shutdown with In-Memory State

2015-12-10 Thread Mao, Wei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050282#comment-15050282 ] Mao, Wei commented on SPARK-12260: -- Here is the design doc:

[jira] [Commented] (SPARK-12260) Graceful Shutdown with In-Memory State

2015-12-10 Thread Mao, Wei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051949#comment-15051949 ] Mao, Wei commented on SPARK-12260: -- In order to recover from process restarting, there are several

[jira] [Commented] (SPARK-8426) Add blacklist mechanism for YARN container allocation

2015-08-31 Thread Mao, Wei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724616#comment-14724616 ] Mao, Wei commented on SPARK-8426: - post the design doc under related JIRA (both 8425 and 8426)

[jira] [Comment Edited] (SPARK-8426) Add blacklist mechanism for YARN container allocation

2015-08-31 Thread Mao, Wei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724616#comment-14724616 ] Mao, Wei edited comment on SPARK-8426 at 9/1/15 2:29 AM: - post the design doc

[jira] [Commented] (SPARK-8425) Add blacklist mechanism for task scheduling

2015-08-28 Thread Mao, Wei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14718204#comment-14718204 ] Mao, Wei commented on SPARK-8425: - Here is the new design doc based on Saisai's previous