[jira] [Updated] (SPARK-13447) Fix AM failure situation for dynamic allocation disabled situation

[ https://issues.apache.org/jira/browse/SPARK-13447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-13447: Summary: Fix AM failure situation for dynamic allocation disabled situation (was: Fix AM failure

[jira] [Updated] (SPARK-13447) Fix AM failure situation for dynamic allocation diabled situation

[ https://issues.apache.org/jira/browse/SPARK-13447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-13447: Summary: Fix AM failure situation for dynamic allocation diabled situation (was: Fix AM failure

[jira] [Created] (SPARK-13447) Fix AM failure situation for dynamic allocation diabled sitation

Saisai Shao created SPARK-13447: --- Summary: Fix AM failure situation for dynamic allocation diabled sitation Key: SPARK-13447 URL: https://issues.apache.org/jira/browse/SPARK-13447 Project: Spark

Re: Kafka streaming receiver approach - new topic not read from beginning

You could set this configuration "auto.offset.reset" through parameter "kafkaParams" which is provided in some other overloaded APIs of createStream. By default Kafka will pick data from latest offset unless you explicitly set it, this is the behavior Kafka, not Spark. Thanks Saisai On Mon, Feb

[jira] [Updated] (SPARK-13426) Remove the support of SIMR cluster manager

[ https://issues.apache.org/jira/browse/SPARK-13426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-13426: Description: Currently there's not so many users who will use SIMR to run Spark, especially

[jira] [Updated] (SPARK-13426) Remove the support of SIMR cluster manager

[ https://issues.apache.org/jira/browse/SPARK-13426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-13426: Issue Type: Sub-task (was: Bug) Parent: SPARK-11806 > Remove the support of SIMR clus

[jira] [Created] (SPARK-13426) Remove the support of SIMR cluster manager

Saisai Shao created SPARK-13426: --- Summary: Remove the support of SIMR cluster manager Key: SPARK-13426 URL: https://issues.apache.org/jira/browse/SPARK-13426 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-12343) Remove YARN Client / ClientArguments

[ https://issues.apache.org/jira/browse/SPARK-12343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153910#comment-15153910 ] Saisai Shao commented on SPARK-12343: - Hi guys, Do we still want to support users who directly

Re: Yarn client mode: Setting environment variables

IIUC for example you want to set environment FOO=bar in executor side, you could use "spark.executor.Env.FOO=bar" in conf file, AM will pick this configuration and set as environment variable through container launching. Just list all the envs you want to set in executor side like

[jira] [Commented] (SPARK-13275) With dynamic allocation, executors appear to be added before job starts

[ https://issues.apache.org/jira/browse/SPARK-13275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149939#comment-15149939 ] Saisai Shao commented on SPARK-13275: - would you please clarify the specific problem you mentioned

[jira] [Commented] (SPARK-13220) Deprecate "yarn-client" and "yarn-cluster"

[ https://issues.apache.org/jira/browse/SPARK-13220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147022#comment-15147022 ] Saisai Shao commented on SPARK-13220: - [~andrewor14] mind me taking a crack on this? > Deprec

Re: IllegalStateException : When use --executor-cores option in YARN

Hi Divya, Would you please provide full stack of exception? From my understanding --executor-cores should be worked, we could know better if you provide the full stack trace. The performance relies on many different aspects, I'd recommend you to check the spark web UI to know the application

Re: sbt publish-local fails with 2.0.0-SNAPSHOT

I think it is due to our recent changes to override the external resolvers in sbt building profile, I just created a JIRA ( https://issues.apache.org/jira/browse/SPARK-13109) to track this. On Mon, Feb 1, 2016 at 3:01 PM, Mike Hynes <91m...@gmail.com> wrote: > Hi devs, > > I used to be able to

[jira] [Created] (SPARK-13109) SBT publishLocal failed to publish to local ivy repo

Saisai Shao created SPARK-13109: --- Summary: SBT publishLocal failed to publish to local ivy repo Key: SPARK-13109 URL: https://issues.apache.org/jira/browse/SPARK-13109 Project: Spark Issue

[jira] [Commented] (SPARK-13104) Spark Metrics currently does not return executors hostname

[ https://issues.apache.org/jira/browse/SPARK-13104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125655#comment-15125655 ] Saisai Shao commented on SPARK-13104: - I think it should be Codahale metrics. > Spark Metr

[jira] [Commented] (SPARK-13106) KafkaUtils.createDirectStream method with messageHandler and topics

[ https://issues.apache.org/jira/browse/SPARK-13106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125598#comment-15125598 ] Saisai Shao commented on SPARK-13106: - IIUC, creating direct stream also supports passing

[jira] [Commented] (SPARK-3374) Spark on Yarn remove deprecated configs for 2.0

[ https://issues.apache.org/jira/browse/SPARK-3374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123043#comment-15123043 ] Saisai Shao commented on SPARK-3374: +1 to address this in the upcoming 2.0 release. 1. Currently

Re: Programmatically launching spark on yarn-client mode no longer works in spark 1.5.2

saying creating sparkcontext manually in your application > still works then I'll investigate more on my side. It just before I dig > more I wanted to know if it was still supported. > > Nir > > On Thu, Jan 28, 2016 at 7:47 PM, Saisai Shao <sai.sai.s...@gmail.com> > wrote:

Re: Programmatically launching spark on yarn-client mode no longer works in spark 1.5.2

I think I met this problem before, this problem might be due to some race conditions in exit period. The way you mentioned is still valid, this problem only occurs when stopping the application. Thanks Saisai On Fri, Jan 29, 2016 at 10:22 AM, Nirav Patel wrote: > Hi, we

Re: How data locality is honored when spark is running on yarn

Hi Todd, There're two levels of locality based scheduling when you run Spark on Yarn if dynamic allocation enabled: 1. Container allocation is based on the locality ratio of pending tasks, this is Yarn specific and only works with dynamic allocation enabled. 2. Task scheduling is locality

Re: streaming textFileStream problem - got only ONE line

Any possibility that this file is still written by other application, so what Spark Streaming processed is an incomplete file. On Tue, Jan 26, 2016 at 5:30 AM, Shixiong(Ryan) Zhu wrote: > Did you move the file into "hdfs://helmhdfs/user/patcharee/cerdata/", or > write

[jira] [Updated] (SPARK-12977) Factoring out StreamingListener and UI to support history UI

[ https://issues.apache.org/jira/browse/SPARK-12977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-12977: Attachment: screenshot-1.png > Factoring out StreamingListener and UI to support history

[jira] [Commented] (SPARK-12977) Factoring out StreamingListener and UI to support history UI

[ https://issues.apache.org/jira/browse/SPARK-12977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15116844#comment-15116844 ] Saisai Shao commented on SPARK-12977: - Attach the current working progress, still some problems

[jira] [Commented] (SPARK-12973) Support to set priority when submit spark application to YARN

[ https://issues.apache.org/jira/browse/SPARK-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15114807#comment-15114807 ] Saisai Shao commented on SPARK-12973: - I think there's a similar JIRA SPARK-10879 about this issue

[jira] [Created] (SPARK-12977) Factoring out StreamingListener and UI to support history UI

Saisai Shao created SPARK-12977: --- Summary: Factoring out StreamingListener and UI to support history UI Key: SPARK-12977 URL: https://issues.apache.org/jira/browse/SPARK-12977 Project: Spark

[jira] [Commented] (SPARK-11045) Contributing Receiver based Low Level Kafka Consumer from Spark-Packages to Apache Spark Project

[ https://issues.apache.org/jira/browse/SPARK-11045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111243#comment-15111243 ] Saisai Shao commented on SPARK-11045: - Hi [~dibbhatt], I'm afraid I could not agree with your comment

[jira] [Commented] (SPARK-12140) Support Streaming UI in HistoryServer

[ https://issues.apache.org/jira/browse/SPARK-12140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111656#comment-15111656 ] Saisai Shao commented on SPARK-12140: - Hi guys, I though a bit on this feature, besides this one big

[jira] [Comment Edited] (SPARK-12140) Support Streaming UI in HistoryServer

[ https://issues.apache.org/jira/browse/SPARK-12140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111656#comment-15111656 ] Saisai Shao edited comment on SPARK-12140 at 1/22/16 1:29 AM: -- Hi guys, I

[jira] [Comment Edited] (SPARK-12140) Support Streaming UI in HistoryServer

[ https://issues.apache.org/jira/browse/SPARK-12140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111656#comment-15111656 ] Saisai Shao edited comment on SPARK-12140 at 1/22/16 1:29 AM: -- Hi guys, I

Re: OOM on yarn-cluster mode

You could try increase the driver memory by "--driver-memory", looks like the OOM is came from driver side, so the simple solution is to increase the memory of driver. On Tue, Jan 19, 2016 at 1:15 PM, Julio Antonio Soto wrote: > Hi, > > I'm having trouble when uploadig spark

[jira] [Commented] (SPARK-12883) 1.6 Dynamic allocation document for removing executors with cached data differs in different sections

[ https://issues.apache.org/jira/browse/SPARK-12883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107085#comment-15107085 ] Saisai Shao commented on SPARK-12883: - I get your point now. But I think these two descriptions

[jira] [Commented] (SPARK-12864) initialize executorIdCounter after ApplicationMaster killed for max number of executor failures reached

[ https://issues.apache.org/jira/browse/SPARK-12864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105671#comment-15105671 ] Saisai Shao commented on SPARK-12864: - What's Spark version are you using? I remember I fixed

[jira] [Comment Edited] (SPARK-12864) initialize executorIdCounter after ApplicationMaster killed for max number of executor failures reached

[ https://issues.apache.org/jira/browse/SPARK-12864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105671#comment-15105671 ] Saisai Shao edited comment on SPARK-12864 at 1/18/16 7:03 PM: -- What Spark

[jira] [Updated] (SPARK-12893) RM redirects to incorrect URL in Spark HistoryServer for yarn-cluster mode

[ https://issues.apache.org/jira/browse/SPARK-12893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-12893: Attachment: Screen Shot 2016-01-18 at 3.47.24 PM.png > RM redirects to incorrect URL in Sp

[jira] [Created] (SPARK-12893) RM redirects to incorrect URL in Spark HistoryServer for yarn-cluster mode

Saisai Shao created SPARK-12893: --- Summary: RM redirects to incorrect URL in Spark HistoryServer for yarn-cluster mode Key: SPARK-12893 URL: https://issues.apache.org/jira/browse/SPARK-12893 Project

[jira] [Updated] (SPARK-12893) RM redirects to incorrect URL in Spark HistoryServer for yarn-cluster mode

[ https://issues.apache.org/jira/browse/SPARK-12893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-12893: Description: This will cause application not found error, the screenshot shows below: !https

[jira] [Commented] (SPARK-12883) 1.6 Dynamic allocation doc still refers to 1.2

[ https://issues.apache.org/jira/browse/SPARK-12883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106005#comment-15106005 ] Saisai Shao commented on SPARK-12883: - I think this doc is still valid, current way of setting

[jira] [Commented] (SPARK-12864) initialize executorIdCounter after ApplicationMaster killed for max number of executor failures reached

[ https://issues.apache.org/jira/browse/SPARK-12864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106214#comment-15106214 ] Saisai Shao commented on SPARK-12864: - So the problem should be that: {{BlockManager}} should

[jira] [Updated] (SPARK-12673) Prepending base URI of job description is missing

[ https://issues.apache.org/jira/browse/SPARK-12673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-12673: Description: The base URI of job description is not prepending in the current code, which makes

[jira] [Updated] (SPARK-12673) Prepending base URI of job description is missing

[ https://issues.apache.org/jira/browse/SPARK-12673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-12673: Attachment: screenshot-1.png > Prepending base URI of job description is miss

[jira] [Created] (SPARK-12673) Prepending base URI of job description is missing

Saisai Shao created SPARK-12673: --- Summary: Prepending base URI of job description is missing Key: SPARK-12673 URL: https://issues.apache.org/jira/browse/SPARK-12673 Project: Spark Issue Type

[jira] [Updated] (SPARK-12673) Prepending base URI of job description is missing

[ https://issues.apache.org/jira/browse/SPARK-12673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-12673: Description: The base URI of job description is not prepending in the current code, which makes

[jira] [Commented] (SPARK-12650) No means to specify Xmx settings for SparkSubmit in yarn-cluster mode

[ https://issues.apache.org/jira/browse/SPARK-12650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15084497#comment-15084497 ] Saisai Shao commented on SPARK-12650: - [~vines], what is your meaning of "SparkSubmit does no

[jira] [Commented] (SPARK-12516) Properly handle NM failure situation for Spark on Yarn

[ https://issues.apache.org/jira/browse/SPARK-12516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15082085#comment-15082085 ] Saisai Shao commented on SPARK-12516: - Thanks a lot [~vanzin] for your reply. Looks like work

[jira] [Commented] (SPARK-12516) Properly handle NM failure situation for Spark on Yarn

[ https://issues.apache.org/jira/browse/SPARK-12516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15075693#comment-15075693 ] Saisai Shao commented on SPARK-12516: - Hi [~vanzin], what is your suggestion of this issue? I'm

[jira] [Commented] (SPARK-12554) Standalone app scheduler will hang when app.coreToAssign < minCoresPerExecutor

[ https://issues.apache.org/jira/browse/SPARK-12554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15074588#comment-15074588 ] Saisai Shao commented on SPARK-12554: - For case 2, I think it is really a misconfiguration problem

[jira] [Created] (SPARK-12552) Recovered driver's resource is not counted in the Master

Saisai Shao created SPARK-12552: --- Summary: Recovered driver's resource is not counted in the Master Key: SPARK-12552 URL: https://issues.apache.org/jira/browse/SPARK-12552 Project: Spark Issue

[jira] [Commented] (SPARK-12554) Standalone app scheduler will hang when app.coreToAssign < minCoresPerExecutor

[ https://issues.apache.org/jira/browse/SPARK-12554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15073954#comment-15073954 ] Saisai Shao commented on SPARK-12554: - First from my understanding tasks can be scheduled after

Re: Problem About Worker System.out

Stdout will not be sent back to driver, no matter you use Scala or Java. You must do something wrongly that makes you think it is an expected behavior. On Mon, Dec 28, 2015 at 5:33 PM, David John wrote: > I have used Spark *1.4* for 6 months. Thanks all the

[jira] [Commented] (SPARK-11782) Master Web UI should link to correct Application UI in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-11782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15073430#comment-15073430 ] Saisai Shao commented on SPARK-11782: - I just verified with latest master branch, seems no such issue

Re: Opening Dynamic Scaling Executors on Yarn

ark-1.6.0 on one yarn > cluster? > > > > *From:* Saisai Shao [mailto:sai.sai.s...@gmail.com] > *Sent:* Monday, December 28, 2015 2:29 PM > *To:* Jeff Zhang > *Cc:* 顾亮亮; user@spark.apache.org; 刘骋昺 > *Subject:* Re: Opening Dynamic Scaling Executors on Yarn > > &g

Re: Opening Dynamic Scaling Executors on Yarn

Replace all the shuffle jars and restart the NodeManager is enough, no need to restart NN. On Mon, Dec 28, 2015 at 2:05 PM, Jeff Zhang wrote: > See > http://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation > > > > On Mon, Dec 28, 2015 at 2:00 PM,

Re: Job Error:Actor not found for: ActorSelection[Anchor(akka.tcp://sparkDriver@130.1.10.108:23600/)

I think SparkContext is thread-safe, you could concurrently submit jobs from different threads, the problem you hit might not relate to this. Can you reproduce this issue each time when you concurrently submit jobs, or is it happened occasionally? BTW, I guess you're using the old version of

Re: Job Error:Actor not found for: ActorSelection[Anchor(akka.tcp://sparkDriver@130.1.10.108:23600/)

might be one potential cause, you'd better increase the vm resource to try again, just to verify your assumption. On Fri, Dec 25, 2015 at 4:28 PM, donhoff_h <165612...@qq.com> wrote: > Hi, Saisai Shao > > Many thanks for your reply. I used spark v1.3. Unfortunately I can not > chang

[jira] [Updated] (SPARK-12516) Properly handle NM failure situation for Spark on Yarn

[ https://issues.apache.org/jira/browse/SPARK-12516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-12516: Description: Failure of NodeManager will make all the executors belong to that NM exit silently

[jira] [Updated] (SPARK-12516) Properly handle NM failure situation for Spark on Yarn

[ https://issues.apache.org/jira/browse/SPARK-12516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-12516: Description: Failure of NodeManager will make all the executors belong to that NM exit silently

[jira] [Updated] (SPARK-12516) Properly handle NM failure situation for Spark on Yarn

[ https://issues.apache.org/jira/browse/SPARK-12516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-12516: Description: Failure of NodeManager will make all the executors belong to that NM exit silently

[jira] [Updated] (SPARK-12447) Only update AM's internal state when executor is successfully launched by NM

[ https://issues.apache.org/jira/browse/SPARK-12447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-12447: Description: Currently {{YarnAllocator}} will update its managed states like

[jira] [Commented] (SPARK-12514) Spark MetricsSystem can fill disks/cause OOMs when using GangliaSink

[ https://issues.apache.org/jira/browse/SPARK-12514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070717#comment-15070717 ] Saisai Shao commented on SPARK-12514: - Since we need to differentiate the metrics between

[jira] [Created] (SPARK-12516) Properly handle NM failure situation for Spark on Yarn

Saisai Shao created SPARK-12516: --- Summary: Properly handle NM failure situation for Spark on Yarn Key: SPARK-12516 URL: https://issues.apache.org/jira/browse/SPARK-12516 Project: Spark Issue

Re: Spark Streaming - Number of RDDs in Dstream

Yes, basically from the currently implementation it should be. On Mon, Dec 21, 2015 at 6:39 PM, Arun Patel <arunp.bigd...@gmail.com> wrote: > So, Does that mean only one RDD is created by all receivers? > > > > On Sun, Dec 20, 2015 at 10:23 PM, Saisai Shao <sai.sai

[jira] [Created] (SPARK-12447) Only update AM's internal state when executor is successfully launched by NM

Saisai Shao created SPARK-12447: --- Summary: Only update AM's internal state when executor is successfully launched by NM Key: SPARK-12447 URL: https://issues.apache.org/jira/browse/SPARK-12447 Project

Re: spark-submit is ignoring "--executor-cores"

Hi Siva, How did you know that --executor-cores is ignored and where did you see that only 1 Vcore is allocated? Thanks Saisai On Tue, Dec 22, 2015 at 9:08 AM, Siva wrote: > Hi Everyone, > > Observing a strange problem while submitting spark streaming job in >

Re: spark-submit is ignoring "--executor-cores"

on web UI. > > Thanks, > Sivakumar Bhavanari. > > On Mon, Dec 21, 2015 at 5:21 PM, Saisai Shao <sai.sai.s...@gmail.com> > wrote: > >> Hi Siva, >> >> How did you know that --executor-cores is ignored and where did you see >> that only 1 Vcore is alloc

Re: Spark Streaming - Number of RDDs in Dstream

Normally there will be one RDD in each batch. You could refer to the implementation of DStream#getOrCompute. On Mon, Dec 21, 2015 at 11:04 AM, Arun Patel wrote: > It may be simple question...But, I am struggling to understand this > > DStream is a sequence of RDDs

[jira] [Commented] (SPARK-10500) sparkr.zip cannot be created if $SPARK_HOME/R/lib is unwritable

[ https://issues.apache.org/jira/browse/SPARK-10500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15061730#comment-15061730 ] Saisai Shao commented on SPARK-10500: - [~sunrui] It would be better to back port to 1.5 if possible

[jira] [Commented] (SPARK-12400) Avoid writing a shuffle file if a partition has no output (empty)

[ https://issues.apache.org/jira/browse/SPARK-12400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15063359#comment-15063359 ] Saisai Shao commented on SPARK-12400: - So from my understanding this will only be existed when

[jira] [Commented] (SPARK-12400) Avoid writing a shuffle file if a partition has no output (empty)

[ https://issues.apache.org/jira/browse/SPARK-12400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15063372#comment-15063372 ] Saisai Shao commented on SPARK-12400: - [~rxin], would you mind me taking a crack at this issue

Re: Can't run spark on yarn

Please check the Yarn AM log to see why AM is failed to start. That's the reason why using `sc` will get such complaint. On Fri, Dec 18, 2015 at 4:25 AM, Eran Witkon wrote: > Hi, > I am trying to install spark 1.5.2 on Apache hadoop 2.6 and Hive and yarn > > spark-env.sh >

[jira] [Commented] (SPARK-12384) Allow -Xms to be set differently then -Xmx

[ https://issues.apache.org/jira/browse/SPARK-12384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15061639#comment-15061639 ] Saisai Shao commented on SPARK-12384: - IIUC, there's also another limitation in container level

[jira] [Commented] (SPARK-12345) Mesos cluster mode is broken

[ https://issues.apache.org/jira/browse/SPARK-12345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059937#comment-15059937 ] Saisai Shao commented on SPARK-12345: - I think by default Spark Mesos implementation will ship all

[jira] [Commented] (SPARK-12345) Mesos cluster mode is broken

[ https://issues.apache.org/jira/browse/SPARK-12345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059954#comment-15059954 ] Saisai Shao commented on SPARK-12345: - Having a quick test by not exporting {{SPARK_HOME

[jira] [Commented] (SPARK-12345) Mesos cluster mode is broken

[ https://issues.apache.org/jira/browse/SPARK-12345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15060035#comment-15060035 ] Saisai Shao commented on SPARK-12345: - Here is the one solution (https://github.com/apache/spark

[jira] [Commented] (SPARK-12345) Mesos cluster mode is broken

[ https://issues.apache.org/jira/browse/SPARK-12345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059778#comment-15059778 ] Saisai Shao commented on SPARK-12345: - A simple solution is to change the scripts to not expose

Re: spark with label nodes in yarn

SPARK-6470 only supports node label expression for executors. SPARK-7173 supports node label expression for AM (will be in 1.6). If you want to schedule your whole application through label expression, you have to configure both am and executor label expression. If you only want to schedule

Re: spark with label nodes in yarn

zzq98...@alibaba-inc.com] > *发送时间:* 2015年12月16日 9:21 > *收件人:* 'Ted Yu' > *抄送:* 'Saisai Shao'; 'dev' > *主题:* Re: spark with label nodes in yarn > > > > Oops... > > > > I do use spark 1.5.0 and apache hadoop 2.6.0 (spark 1.4.1 + apache hadoop > 2.6.0 is

[jira] [Commented] (SPARK-12345) Mesos cluster mode is broken

[ https://issues.apache.org/jira/browse/SPARK-12345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059569#comment-15059569 ] Saisai Shao commented on SPARK-12345: - It is OK in my local test when I followed the step one by one

[jira] [Commented] (SPARK-12176) SparkLauncher's setConf() does not support configs containing spaces

[ https://issues.apache.org/jira/browse/SPARK-12176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15055508#comment-15055508 ] Saisai Shao commented on SPARK-12176: - It is OK in my local test against latest master branch, seems

[jira] [Commented] (SPARK-9059) Update Python Direct Kafka Word count examples to show the use of HasOffsetRanges

[ https://issues.apache.org/jira/browse/SPARK-9059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049804#comment-15049804 ] Saisai Shao commented on SPARK-9059: HasOffsetRanges also has python version, which is added in SPARK

[jira] [Commented] (SPARK-6735) Provide options to make maximum executor failure count ( which kills the application ) relative to a window duration or disable it.

[ https://issues.apache.org/jira/browse/SPARK-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049958#comment-15049958 ] Saisai Shao commented on SPARK-6735: I've submitted a patch to continue this work (https://github.com

Re: Spark Stream Monitoring with Kafka Direct API

I think this is the right JIRA to fix this issue ( https://issues.apache.org/jira/browse/SPARK-7111). It should be in Spark 1.4. On Thu, Dec 10, 2015 at 12:32 AM, Cody Koeninger wrote: > Looks like probably > > https://issues.apache.org/jira/browse/SPARK-8701 > > so 1.5.0 >

[jira] [Commented] (SPARK-12178) Expose reporting of StreamInputInfo for custom made streams

[ https://issues.apache.org/jira/browse/SPARK-12178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15046602#comment-15046602 ] Saisai Shao commented on SPARK-12178: - This is a good idea to make it generic if there's more direct

Re: set up spark 1.4.1 as default spark engine in HDP 2.2/2.3

Please make sure the spark shell script you're running is pointed to /bin/spark-shell Just follow the instructions to correctly configure your spark 1.4.1 and execute correct script are enough. On Wed, Dec 9, 2015 at 11:28 AM, Divya Gehlot wrote: > Hi, > As per

[jira] [Commented] (SPARK-10123) Cannot set "--deploy-mode" in default configuration

[ https://issues.apache.org/jira/browse/SPARK-10123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15041260#comment-15041260 ] Saisai Shao commented on SPARK-10123: - Hi [~vanzin], would you mind letting me take a crack

[jira] [Commented] (SPARK-12103) KafkaUtils createStream with multiple topics -- does not work as expected

[ https://issues.apache.org/jira/browse/SPARK-12103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15041301#comment-15041301 ] Saisai Shao commented on SPARK-12103: - I think I had a proposal of message handler (receiver

[jira] [Commented] (SPARK-10123) Cannot set "--deploy-mode" in default configuration

[ https://issues.apache.org/jira/browse/SPARK-10123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15042598#comment-15042598 ] Saisai Shao commented on SPARK-10123: - Just confirm if it is on your plan, in case duplicated

[jira] [Comment Edited] (SPARK-12059) Standalone Master assertion error

[ https://issues.apache.org/jira/browse/SPARK-12059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034997#comment-15034997 ] Saisai Shao edited comment on SPARK-12059 at 12/2/15 12:47 AM: --- A simple

[jira] [Commented] (SPARK-12059) Standalone Master assertion error

[ https://issues.apache.org/jira/browse/SPARK-12059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034997#comment-15034997 ] Saisai Shao commented on SPARK-12059: - A simple solution is to loose the condition or remove

[jira] [Commented] (SPARK-12059) Standalone Master assertion error

[ https://issues.apache.org/jira/browse/SPARK-12059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035350#comment-15035350 ] Saisai Shao commented on SPARK-12059: - I see, so I will relax the condition to avoid exception from

[jira] [Commented] (SPARK-12059) Standalone Master assertion error

[ https://issues.apache.org/jira/browse/SPARK-12059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15032806#comment-15032806 ] Saisai Shao commented on SPARK-12059: - Thanks a lot [~andrewor14], I will look into this issue

[jira] [Commented] (SPARK-12059) Standalone Master assertion error

[ https://issues.apache.org/jira/browse/SPARK-12059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15033268#comment-15033268 ] Saisai Shao commented on SPARK-12059: - Hi [~andrewor14], when will this be happened? I suppose state

[jira] [Commented] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15031308#comment-15031308 ] Saisai Shao commented on SPARK-12009: - OK, I see. So how about setting {{YarnAllocator

Re: tests blocked at "don't call ssc.stop in listener"

Might be related to this JIRA ( https://issues.apache.org/jira/browse/SPARK-11761), not very sure about it. On Fri, Nov 27, 2015 at 10:22 AM, Nan Zhu wrote: > Hi, all > > Anyone noticed that some of the tests just blocked at the test case “don't > call ssc.stop in

[jira] [Commented] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029425#comment-15029425 ] Saisai Shao commented on SPARK-12009: - So I guess your problem is that after you call {{sc.stop

[jira] [Commented] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029416#comment-15029416 ] Saisai Shao commented on SPARK-12009: - So what actually version of Spark you're running? 1.4.0

[jira] [Comment Edited] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029416#comment-15029416 ] Saisai Shao edited comment on SPARK-12009 at 11/27/15 3:30 AM: --- So what

[jira] [Commented] (SPARK-12002) offsetRanges attribute missing in Kafka RDD when resuming from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-12002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029421#comment-15029421 ] Saisai Shao commented on SPARK-12002: - Looks like because Python `KafkaTransformDStream` specific

[jira] [Comment Edited] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029431#comment-15029431 ] Saisai Shao edited comment on SPARK-12009 at 11/27/15 3:50 AM: --- Alright, my

[jira] [Commented] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029431#comment-15029431 ] Saisai Shao commented on SPARK-12009: - Alright, my code is master branch. Anyway I understood your

[jira] [Commented] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15028340#comment-15028340 ] Saisai Shao commented on SPARK-12009: - Looking at the code again, {{onDisconnected

<    10   11   12   13   14   15   16   17   18   19   >