[jira] [Commented] (AMBARI-13980) When Spark Thrift server is selected, ensure Hive is deployed

2015-11-25 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/AMBARI-13980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028308#comment-15028308 ] Saisai Shao commented on AMBARI-13980: -- [~sumitmohanty], patch is available, please

[jira] [Assigned] (AMBARI-13980) When Spark Thrift server is selected, ensure Hive is deployed

2015-11-25 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/AMBARI-13980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned AMBARI-13980: Assignee: Saisai Shao (was: Sumit Mohanty) > When Spark Thrift server is selected, ensur

[jira] [Commented] (SPARK-11909) Spark Standalone's master URL accepts URLs without port (assuming default 7077)

2015-11-22 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15021669#comment-15021669 ] Saisai Shao commented on SPARK-11909: - The master will print the master url in web UI

[jira] [Commented] (AMBARI-13983) Correctly deploy and start Spark thrift server both in secure and non-secure environment

2015-11-22 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/AMBARI-13983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15020918#comment-15020918 ] Saisai Shao commented on AMBARI-13983: -- [~sumitmohanty], please help to review this

[jira] [Updated] (AMBARI-13983) Correctly deploy and start Spark thrift server both in secure and non-secure environment

2015-11-19 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/AMBARI-13983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated AMBARI-13983: - Description: Changes to make Spark TS work correctly both in secure and non-secure environment.

[jira] [Created] (AMBARI-13983) Correctly deploy and start Spark thrift server both in secure and non-secure environment

2015-11-19 Thread Saisai Shao (JIRA)
Saisai Shao created AMBARI-13983: Summary: Correctly deploy and start Spark thrift server both in secure and non-secure environment Key: AMBARI-13983 URL: https://issues.apache.org/jira/browse/AMBARI-13983

[jira] [Created] (AMBARI-13980) When Spark Thrift server is selected, ensure Hive is deployed

2015-11-19 Thread Saisai Shao (JIRA)
Saisai Shao created AMBARI-13980: Summary: When Spark Thrift server is selected, ensure Hive is deployed Key: AMBARI-13980 URL: https://issues.apache.org/jira/browse/AMBARI-13980 Project: Ambari

[jira] [Updated] (AMBARI-13964) Spark TS start should wait till HiveServer has started

2015-11-18 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/AMBARI-13964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated AMBARI-13964: - Attachment: AMBARI-13964.patch > Spark TS start should wait till HiveServer has started > --

[jira] [Commented] (AMBARI-13899) Remove hdp.version configuration in Ambari Spark definition

2015-11-15 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/AMBARI-13899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006193#comment-15006193 ] Saisai Shao commented on AMBARI-13899: -- Please review. > Remove hdp.version config

[jira] [Created] (AMBARI-13899) Remove hdp.version configuration in Ambari Spark definition

2015-11-15 Thread Saisai Shao (JIRA)
Saisai Shao created AMBARI-13899: Summary: Remove hdp.version configuration in Ambari Spark definition Key: AMBARI-13899 URL: https://issues.apache.org/jira/browse/AMBARI-13899 Project: Ambari

[jira] [Commented] (SPARK-11744) bin/pyspark --version doesn't return version and exit

2015-11-15 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005862#comment-15005862 ] Saisai Shao commented on SPARK-11744: - Yes, it is actually a small bug in Launcher, l

[jira] [Commented] (SPARK-11701) YARN - dynamic allocation and speculation active task accounting wrong

2015-11-13 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15003996#comment-15003996 ] Saisai Shao commented on SPARK-11701: - Hi [~tgraves] I think I'm referring to your la

[jira] [Commented] (AMBARI-13830) Bump the Ambari Spark definition to Spark 1.5.2

2015-11-13 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/AMBARI-13830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15003735#comment-15003735 ] Saisai Shao commented on AMBARI-13830: -- Is it getting merged, looks like not. Would

[jira] [Commented] (SPARK-11701) YARN - dynamic allocation and speculation active task accounting wrong

2015-11-12 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15003601#comment-15003601 ] Saisai Shao commented on SPARK-11701: - Hi [~tgraves], I think I also hit this issue,

[jira] [Commented] (SPARK-2960) Spark executables fail to start via symlinks

2015-11-12 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15003598#comment-15003598 ] Saisai Shao commented on SPARK-2960: Hi [~yinxusen], from script's view point, it cann

[jira] [Created] (SPARK-11718) Explicit killing executor dies silent without get response information

2015-11-12 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-11718: --- Summary: Explicit killing executor dies silent without get response information Key: SPARK-11718 URL: https://issues.apache.org/jira/browse/SPARK-11718 Project: Spark

[jira] [Commented] (AMBARI-13830) Bump the Ambari Spark definition to Spark 1.5.2

2015-11-10 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/AMBARI-13830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1465#comment-1465 ] Saisai Shao commented on AMBARI-13830: -- Spark code already update to version 1.5.2,

[jira] [Updated] (AMBARI-13830) Bump the Ambari Spark definition to Spark 1.5.2

2015-11-10 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/AMBARI-13830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated AMBARI-13830: - Attachment: AMBARI-13830.patch > Bump the Ambari Spark definition to Spark 1.5.2 > -

[jira] [Created] (AMBARI-13830) Bump the Ambari Spark definition to Spark 1.5.2

2015-11-10 Thread Saisai Shao (JIRA)
Saisai Shao created AMBARI-13830: Summary: Bump the Ambari Spark definition to Spark 1.5.2 Key: AMBARI-13830 URL: https://issues.apache.org/jira/browse/AMBARI-13830 Project: Ambari Issue Type

[jira] [Created] (SPARK-11632) Filter out empty partition for KafkaRDD

2015-11-10 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-11632: --- Summary: Filter out empty partition for KafkaRDD Key: SPARK-11632 URL: https://issues.apache.org/jira/browse/SPARK-11632 Project: Spark Issue Type: Improvement

[jira] [Updated] (AMBARI-13784) Update the Spark env definitions

2015-11-09 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/AMBARI-13784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated AMBARI-13784: - Attachment: AMBARI-13784.patch > Update the Spark env definitions >

[jira] [Created] (AMBARI-13784) Update the Spark env definitions

2015-11-08 Thread Saisai Shao (JIRA)
Saisai Shao created AMBARI-13784: Summary: Update the Spark env definitions Key: AMBARI-13784 URL: https://issues.apache.org/jira/browse/AMBARI-13784 Project: Ambari Issue Type: Bug

[jira] [Commented] (SPARK-5934) DStreamGraph.clearMetadata attempts to unpersist the same RDD multiple times

2015-11-05 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14991353#comment-14991353 ] Saisai Shao commented on SPARK-5934: I think here in your code: {code} val output = i

[jira] [Updated] (AMBARI-13729) Change the Spark thrift server security configurations

2015-11-04 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/AMBARI-13729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated AMBARI-13729: - Attachment: AMBARI-13729.patch > Change the Spark thrift server security configurations > --

[jira] [Created] (AMBARI-13729) Change the Spark thrift server security configurations

2015-11-04 Thread Saisai Shao (JIRA)
Saisai Shao created AMBARI-13729: Summary: Change the Spark thrift server security configurations Key: AMBARI-13729 URL: https://issues.apache.org/jira/browse/AMBARI-13729 Project: Ambari Iss

[jira] [Commented] (AMBARI-13728) Fix Spark History server not work problem

2015-11-04 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/AMBARI-13728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14990971#comment-14990971 ] Saisai Shao commented on AMBARI-13728: -- Please help to review, this blocks the devd

[jira] [Updated] (AMBARI-13728) Fix Spark History server not work problem

2015-11-04 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/AMBARI-13728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated AMBARI-13728: - Attachment: AMBARI-13728.patch > Fix Spark History server not work problem > ---

[jira] [Created] (AMBARI-13728) Fix Spark History server not work problem

2015-11-04 Thread Saisai Shao (JIRA)
Saisai Shao created AMBARI-13728: Summary: Fix Spark History server not work problem Key: AMBARI-13728 URL: https://issues.apache.org/jira/browse/AMBARI-13728 Project: Ambari Issue Type: Bug

[jira] [Commented] (AMBARI-13650) Bump the Ambari Spark definition to Spark 1.5.1

2015-11-04 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/AMBARI-13650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14990860#comment-14990860 ] Saisai Shao commented on AMBARI-13650: -- Now the HDP trunk repo has bumped to Spark

[jira] [Commented] (SPARK-11472) SparkContext creation error after sc.stop() when Spark is compiled for Hive

2015-11-03 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14987314#comment-14987314 ] Saisai Shao commented on SPARK-11472: - Yes, agree with [~srowen], there might be a lo

[jira] [Created] (SPARK-11457) Yarn AM proxy filter configuration should be reloaded when recovered from checkpoint

2015-11-02 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-11457: --- Summary: Yarn AM proxy filter configuration should be reloaded when recovered from checkpoint Key: SPARK-11457 URL: https://issues.apache.org/jira/browse/SPARK-11457 Pr

[jira] [Updated] (AMBARI-13648) Fix incorrect SPARK_CONF_DIR in Ambari stack definition

2015-11-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/AMBARI-13648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated AMBARI-13648: - Attachment: AMBARI-13648.patch > Fix incorrect SPARK_CONF_DIR in Ambari stack definition > -

[jira] [Updated] (AMBARI-13648) Fix incorrect SPARK_CONF_DIR in Ambari stack definition

2015-11-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/AMBARI-13648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated AMBARI-13648: - Attachment: (was: AMBARI-13648.patch.1) > Fix incorrect SPARK_CONF_DIR in Ambari stack defin

[jira] [Updated] (AMBARI-13648) Fix incorrect SPARK_CONF_DIR in Ambari stack definition

2015-11-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/AMBARI-13648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated AMBARI-13648: - Attachment: (was: AMBARI-13648.patch) > Fix incorrect SPARK_CONF_DIR in Ambari stack definit

[jira] [Updated] (AMBARI-13648) Fix incorrect SPARK_CONF_DIR in Ambari stack definition

2015-11-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/AMBARI-13648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated AMBARI-13648: - Attachment: AMBARI-13648.patch.1 > Fix incorrect SPARK_CONF_DIR in Ambari stack definition > ---

[jira] [Updated] (AMBARI-13650) Bump the Ambari Spark definition to Spark 1.5.1

2015-10-30 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/AMBARI-13650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated AMBARI-13650: - Attachment: AMBARI-13650.patch > Bump the Ambari Spark definition to Spark 1.5.1 > -

[jira] [Updated] (AMBARI-13650) Bump the Ambari Spark definition to Spark 1.5.1

2015-10-30 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/AMBARI-13650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated AMBARI-13650: - Attachment: (was: 0001-Bump-to-1.5.1-support.patch) > Bump the Ambari Spark definition to Sp

[jira] [Updated] (AMBARI-13648) Fix incorrect SPARK_CONF_DIR in Ambari stack definition

2015-10-30 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/AMBARI-13648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated AMBARI-13648: - Attachment: AMBARI-13648.patch > Fix incorrect SPARK_CONF_DIR in Ambari stack definition > -

[jira] [Updated] (AMBARI-13648) Fix incorrect SPARK_CONF_DIR in Ambari stack definition

2015-10-30 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/AMBARI-13648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated AMBARI-13648: - Attachment: (was: 0001-Fix-incorrect-SPARK_CONF_DIR-definition.patch) > Fix incorrect SPARK_

[jira] [Commented] (AMBARI-13650) Bump the Ambari Spark definition to Spark 1.5.1

2015-10-30 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/AMBARI-13650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14982060#comment-14982060 ] Saisai Shao commented on AMBARI-13650: -- This Patch is to bump the version of Spark

[jira] [Updated] (AMBARI-13650) Bump the Ambari Spark definition to Spark 1.5.1

2015-10-30 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/AMBARI-13650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated AMBARI-13650: - Attachment: 0001-Bump-to-1.5.1-support.patch > Bump the Ambari Spark definition to Spark 1.5.1 >

[jira] [Created] (AMBARI-13650) Bump the Ambari Spark definition to Spark 1.5.1

2015-10-30 Thread Saisai Shao (JIRA)
Saisai Shao created AMBARI-13650: Summary: Bump the Ambari Spark definition to Spark 1.5.1 Key: AMBARI-13650 URL: https://issues.apache.org/jira/browse/AMBARI-13650 Project: Ambari Issue Type

[jira] [Commented] (AMBARI-13648) Fix incorrect SPARK_CONF_DIR in Ambari stack definition

2015-10-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/AMBARI-13648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14981792#comment-14981792 ] Saisai Shao commented on AMBARI-13648: -- [~venkateshrin], [~jluniya], please help to

[jira] [Updated] (AMBARI-13648) Fix incorrect SPARK_CONF_DIR in Ambari stack definition

2015-10-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/AMBARI-13648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated AMBARI-13648: - Attachment: 0001-Fix-incorrect-SPARK_CONF_DIR-definition.patch > Fix incorrect SPARK_CONF_DIR in

[jira] [Created] (AMBARI-13648) Fix incorrect SPARK_CONF_DIR in Ambari stack definition

2015-10-29 Thread Saisai Shao (JIRA)
Saisai Shao created AMBARI-13648: Summary: Fix incorrect SPARK_CONF_DIR in Ambari stack definition Key: AMBARI-13648 URL: https://issues.apache.org/jira/browse/AMBARI-13648 Project: Ambari Is

[jira] [Commented] (SPARK-2089) With YARN, preferredNodeLocalityData isn't honored

2015-10-28 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14979586#comment-14979586 ] Saisai Shao commented on SPARK-2089: Hi [~sandyr] and [~tgraves], thank a lot for your

[jira] [Commented] (SPARK-2089) With YARN, preferredNodeLocalityData isn't honored

2015-10-28 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14977969#comment-14977969 ] Saisai Shao commented on SPARK-2089: Hi [~pwendell], [~mridulm80], [~sandyr] and [~lia

[jira] [Updated] (SPARK-11272) Support importing and exporting event logs from HistoryServer web portal

2015-10-22 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-11272: Summary: Support importing and exporting event logs from HistoryServer web portal (was: Support im

[jira] [Updated] (SPARK-11272) Support importing and exporting event logs from History web portal

2015-10-22 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-11272: Description: Many users who met the problems when running Spark applications will send the logs to

[jira] [Updated] (SPARK-11272) Support importing and exporting event logs from History web portal

2015-10-22 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-11272: Attachment: screenshot-1.png > Support importing and exporting event logs from History web portal >

[jira] [Created] (SPARK-11272) Support importing and exporting event logs from History web portal

2015-10-22 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-11272: --- Summary: Support importing and exporting event logs from History web portal Key: SPARK-11272 URL: https://issues.apache.org/jira/browse/SPARK-11272 Project: Spark

[jira] [Commented] (SPARK-11181) Spark Yarn : Spark reducing total executors count even when Dynamic Allocation is disabled.

2015-10-21 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14968649#comment-14968649 ] Saisai Shao commented on SPARK-11181: - I think you'd better backport to branch 1.3, n

[jira] [Commented] (SPARK-11181) Spark Yarn : Spark reducing total executors count even when Dynamic Allocation is disabled.

2015-10-21 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14968648#comment-14968648 ] Saisai Shao commented on SPARK-11181: - OK, get it. > Spark Yarn : Spark reducing tot

[jira] [Commented] (SPARK-11181) Spark Yarn : Spark reducing total executors count even when Dynamic Allocation is disabled.

2015-10-21 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14968453#comment-14968453 ] Saisai Shao commented on SPARK-11181: - Hi [~prakhar088], As discussed in the mailist

[jira] [Commented] (SPARK-11175) Concurrent execution of JobSet within a batch in Spark streaming

2015-10-18 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14962708#comment-14962708 ] Saisai Shao commented on SPARK-11175: - There's a configuration "spark.streaming.concu

[jira] [Closed] (SPARK-11082) Cores per executor is wrong when response vcore number is less than requested number

2015-10-13 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-11082. --- Resolution: Invalid > Cores per executor is wrong when response vcore number is less than requested

[jira] [Created] (SPARK-11082) Cores per executor is wrong when response vcore number is less than requested number

2015-10-13 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-11082: --- Summary: Cores per executor is wrong when response vcore number is less than requested number Key: SPARK-11082 URL: https://issues.apache.org/jira/browse/SPARK-11082 Pr

[jira] [Commented] (SPARK-11045) Contributing Receiver based Low Level Kafka Consumer from Spark-Packages to Apache Spark Project

2015-10-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14952594#comment-14952594 ] Saisai Shao commented on SPARK-11045: - Hi Dib, To get better performance and fix the

[jira] [Commented] (SPARK-11045) Contributing Receiver based Low Level Kafka Consumer from Spark-Packages to Apache Spark Project

2015-10-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14952560#comment-14952560 ] Saisai Shao commented on SPARK-11045: - {quote} Unfortunately, when customer fall back

[jira] [Created] (SPARK-11060) Fix some potential NPEs in DStream transformation

2015-10-11 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-11060: --- Summary: Fix some potential NPEs in DStream transformation Key: SPARK-11060 URL: https://issues.apache.org/jira/browse/SPARK-11060 Project: Spark Issue Type: B

[jira] [Comment Edited] (SPARK-10858) YARN: archives/jar/files rename with # doesn't work unless scheme given

2015-09-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14936317#comment-14936317 ] Saisai Shao edited comment on SPARK-10858 at 9/30/15 5:49 AM: -

[jira] [Comment Edited] (SPARK-10858) YARN: archives/jar/files rename with # doesn't work unless scheme given

2015-09-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14936313#comment-14936313 ] Saisai Shao edited comment on SPARK-10858 at 9/30/15 3:50 AM: -

[jira] [Commented] (SPARK-10858) YARN: archives/jar/files rename with # doesn't work unless scheme given

2015-09-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14936317#comment-14936317 ] Saisai Shao commented on SPARK-10858: - So basically I think the problem is do we need

[jira] [Commented] (SPARK-10858) YARN: archives/jar/files rename with # doesn't work unless scheme given

2015-09-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14936313#comment-14936313 ] Saisai Shao commented on SPARK-10858: - Hi [~tgraves], I tested again with Mac and Lin

[jira] [Issue Comment Deleted] (SPARK-10858) YARN: archives/jar/files rename with # doesn't work unless scheme given

2015-09-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-10858: Comment: was deleted (was: The error I got in the failed case is the same as you mentioned above,

[jira] [Issue Comment Deleted] (SPARK-10858) YARN: archives/jar/files rename with # doesn't work unless scheme given

2015-09-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-10858: Comment: was deleted (was: The error I got in the failed case is the same as you mentioned above,

[jira] [Commented] (SPARK-10858) YARN: archives/jar/files rename with # doesn't work unless scheme given

2015-09-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14936175#comment-14936175 ] Saisai Shao commented on SPARK-10858: - The error I got in the failed case is the same

[jira] [Commented] (SPARK-10858) YARN: archives/jar/files rename with # doesn't work unless scheme given

2015-09-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14936174#comment-14936174 ] Saisai Shao commented on SPARK-10858: - The error I got in the failed case is the same

[jira] [Commented] (SPARK-10858) YARN: archives/jar/files rename with # doesn't work unless scheme given

2015-09-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14936176#comment-14936176 ] Saisai Shao commented on SPARK-10858: - The error I got in the failed case is the same

[jira] [Commented] (SPARK-10858) YARN: archives/jar/files rename with # doesn't work unless scheme given

2015-09-28 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14934724#comment-14934724 ] Saisai Shao commented on SPARK-10858: - [~tgraves], I assume you're using yarn-cluster

[jira] [Commented] (SPARK-10790) Dynamic Allocation does not request any executors if first stage needs less than or equal to spark.dynamicAllocation.initialExecutors

2015-09-24 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14906969#comment-14906969 ] Saisai Shao commented on SPARK-10790: - Hi [~jonathak], I think I reproduced the probl

[jira] [Commented] (SPARK-10790) Dynamic Allocation does not request any executors if first stage needs less than or equal to spark.dynamicAllocation.initialExecutors

2015-09-24 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14906787#comment-14906787 ] Saisai Shao commented on SPARK-10790: - Hi [~jonathak], let me trying to understand yo

[jira] [Commented] (SPARK-10790) Dynamic Allocation does not request any executors if first stage needs less than or equal to spark.dynamicAllocation.initialExecutors

2015-09-24 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14906696#comment-14906696 ] Saisai Shao commented on SPARK-10790: - Thanks [~srowen], let me check it. > Dynamic

[jira] [Commented] (SPARK-10739) Add attempt window for long running Spark application on Yarn

2015-09-22 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14902996#comment-14902996 ] Saisai Shao commented on SPARK-10739: - Yes, as Sandy mentioned about, SPARK-6735 is f

[jira] [Commented] (SPARK-10739) Add attempt window for long running Spark application on Yarn

2015-09-21 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901944#comment-14901944 ] Saisai Shao commented on SPARK-10739: - [~srowen] I'm not sure what you mentioned abou

[jira] [Updated] (SPARK-10739) Add attempt window for long running Spark application on Yarn

2015-09-21 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-10739: Description: Currently Spark on Yarn uses max attempts to control the failure number, if applicati

[jira] [Created] (SPARK-10739) Add attempt window for long running Spark application on Yarn

2015-09-21 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-10739: --- Summary: Add attempt window for long running Spark application on Yarn Key: SPARK-10739 URL: https://issues.apache.org/jira/browse/SPARK-10739 Project: Spark

[jira] [Commented] (SPARK-10644) Applications wait even if free executors are available

2015-09-16 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14791439#comment-14791439 ] Saisai Shao commented on SPARK-10644: - So what's the cluster manager you use, standal

[jira] [Commented] (SPARK-10644) Applications wait even if free executors are available

2015-09-16 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14791370#comment-14791370 ] Saisai Shao commented on SPARK-10644: - Does you jobs have dependencies? That is sayin

[jira] [Created] (SPARK-10558) Wrong executor state in standalone master because of wrong state transition

2015-09-10 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-10558: --- Summary: Wrong executor state in standalone master because of wrong state transition Key: SPARK-10558 URL: https://issues.apache.org/jira/browse/SPARK-10558 Project: Sp

[jira] [Commented] (SPARK-10288) Add a rest client for Spark on Yarn

2015-09-07 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14734161#comment-14734161 ] Saisai Shao commented on SPARK-10288: - Hi Marcelo, Yes I agree with you that rest cl

[jira] [Commented] (SPARK-10288) Add a rest client for Spark on Yarn

2015-09-06 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14733223#comment-14733223 ] Saisai Shao commented on SPARK-10288: - Thanks a lot Marcelo for your comments, indeed

[jira] [Commented] (SPARK-7173) Support YARN node label expressions for the application master

2015-09-03 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14730358#comment-14730358 ] Saisai Shao commented on SPARK-7173: Hi guys, is it still meaningful to support label

[jira] [Commented] (SPARK-10399) Off Heap Memory Access for non-JVM libraries (C++)

2015-09-02 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14726886#comment-14726886 ] Saisai Shao commented on SPARK-10399: - Hi [~paulweiss], a simple question about C++ l

[jira] [Updated] (SPARK-10288) Add a rest client for Spark on Yarn

2015-09-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-10288: Description: This is a proposal to add rest client for Spark on Yarn. Rest API offers a convenient

[jira] [Commented] (SPARK-10288) Add a rest client for Spark on Yarn

2015-09-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14726814#comment-14726814 ] Saisai Shao commented on SPARK-10288: - Hi [~vanzin], thanks a lot for your comments.

[jira] [Created] (SPARK-10288) Add a rest client for Spark on Yarn

2015-08-25 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-10288: --- Summary: Add a rest client for Spark on Yarn Key: SPARK-10288 URL: https://issues.apache.org/jira/browse/SPARK-10288 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-10122) AttributeError: 'RDD' object has no attribute 'offsetRanges'

2015-08-20 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706306#comment-14706306 ] Saisai Shao commented on SPARK-10122: - Thanks :). > AttributeError: 'RDD' object has

[jira] [Commented] (SPARK-10122) AttributeError: 'RDD' object has no attribute 'offsetRanges'

2015-08-20 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706186#comment-14706186 ] Saisai Shao commented on SPARK-10122: - Hi [~aramesh], thanks a lot for pointing this

[jira] [Commented] (SPARK-10122) AttributeError: 'RDD' object has no attribute 'offsetRanges'

2015-08-20 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14704832#comment-14704832 ] Saisai Shao commented on SPARK-10122: - Thanks [~am...@hemtech.co.za] and [~srowen], a

[jira] [Created] (SPARK-9969) Remove old Yarn MR classpath api support for Spark Yarn client

2015-08-14 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-9969: -- Summary: Remove old Yarn MR classpath api support for Spark Yarn client Key: SPARK-9969 URL: https://issues.apache.org/jira/browse/SPARK-9969 Project: Spark Iss

[jira] [Created] (SPARK-9877) StandaloneRestServer NPE when submitting application

2015-08-12 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-9877: -- Summary: StandaloneRestServer NPE when submitting application Key: SPARK-9877 URL: https://issues.apache.org/jira/browse/SPARK-9877 Project: Spark Issue Type: Bu

[jira] [Created] (SPARK-9817) Improve the container placement strategy by considering the localities of pending container requests

2015-08-11 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-9817: -- Summary: Improve the container placement strategy by considering the localities of pending container requests Key: SPARK-9817 URL: https://issues.apache.org/jira/browse/SPARK-9817

[jira] [Commented] (SPARK-1022) Add unit tests for kafka streaming

2015-08-05 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659246#comment-14659246 ] Saisai Shao commented on SPARK-1022: Hi [~anujojha], {{KafkaTestUtils}} is used for Ka

[jira] [Created] (SPARK-9468) Avoid scheduling tasks on preemption executors

2015-07-29 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-9468: -- Summary: Avoid scheduling tasks on preemption executors Key: SPARK-9468 URL: https://issues.apache.org/jira/browse/SPARK-9468 Project: Spark Issue Type: Improvem

[jira] [Commented] (SPARK-4352) Incorporate locality preferences in dynamic allocation requests

2015-07-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14647104#comment-14647104 ] Saisai Shao commented on SPARK-4352: Hi [~tgraves], location is got from DAGScheduler

[jira] [Commented] (SPARK-4352) Incorporate locality preferences in dynamic allocation requests

2015-07-28 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14645333#comment-14645333 ] Saisai Shao commented on SPARK-4352: Hi [~tgraves], to be clear the original way of se

[jira] [Commented] (SPARK-9059) Update Direct Kafka Word count examples to show the use of HasOffsetRanges

2015-07-16 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14630663#comment-14630663 ] Saisai Shao commented on SPARK-9059: I think it would be better also update the relate

[jira] [Commented] (SPARK-8337) KafkaUtils.createDirectStream for python is lacking API/feature parity with the Scala/Java version

2015-07-15 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14627728#comment-14627728 ] Saisai Shao commented on SPARK-8337: OK, thanks TD. > KafkaUtils.createDirectStream f

[jira] [Commented] (SPARK-8425) Add blacklist mechanism for task scheduling

2015-07-14 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14627365#comment-14627365 ] Saisai Shao commented on SPARK-8425: Repost the design doc here: https://docs.google.

<    8   9   10   11   12   13   14   15   16   17   >