Get broadcast (set in one method) in another method

Hi Maybe I am overthinking. I’d like to set broadcast in object A method y and get it in object A method x. In example: object A { def main (args: Array[String]) { y() x() } def x() : Unit = { val a = bcA.value ... } def y(): String = { val bcA =

Re: run spark job in yarn cluster mode as specified user

Hi org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor requires user in each node and right permissions set in necessary directories. Br Margus > On 22 Jan 2018, at 13:41, sd wang wrote: > >

Re: run spark job in yarn cluster mode as specified user

Hi One way to get it is use YARN configuration parameter - yarn.nodemanager.container-executor.class. By default it is org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor - gives you user who executes script. Br

Re: spark job paused(active stages finished)

You have to deal with failed jobs. In example try catch in your code. Br Margus Roo > On 9 Nov 2017, at 05:37, bing...@iflytek.com wrote: > > Dear,All > I have a simple spark job, as below, all tasks in the stage 2(sth failed, > retry) already finished. But the next stage never run. > >

[no subject]

unsubscribe - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org

Exception failure: java.lang.ClassNotFoundException: org.apache.spark.streaming.kafka.KafkaReceiver

]# Any ideas? -- Best regards, Margus (Margusja) Roo +372 51 48 780 http://margus.roo.ee http://ee.linkedin.com/in/margusroo skype: margusja ldapsearch -x -h ldap.sk.ee -b c=EE (serialNumber=37303140314)

Re: Announcing Spark 1.0.0

Now I can download. Thanks. Best regards, Margus (Margusja) Roo +372 51 48 780 http://margus.roo.ee http://ee.linkedin.com/in/margusroo skype: margusja ldapsearch -x -h ldap.sk.ee -b c=EE (serialNumber=37303140314) On 30/05/14 13:48, Patrick Wendell wrote: It is updated - try holding Shift