Re: SPARK STREAMING PROBLEM

lit(" ")(2)::res > }} } >}} >res > } > > g.foreach(x => {println(x); println("")}) > > However when I run the code, nothing gets printed on the console apart > from the logs. Am I doin

Re: SPARK STREAMING PROBLEM

t; } > > g.foreach(x => {println(x); println("")}) > > However when I run the code, nothing gets printed on the console apart > from the logs. Am I doing something wrong? > And is there any

Re: Best practices on testing Spark jobs

s the recommended way >> to test Spark jobs? Should I manually create mocks by e.g. extending all >> the classes I'd normally mock and changing the implementation of some >> methods? I don't like this idea but I can't really see any other options >> now. >

Re: Spark Streaming updatyeStateByKey throws OutOfMemory Error

*bump* On Thu, Apr 23, 2015 at 3:46 PM, Sourav Chandra < sourav.chan...@livestream.com> wrote: > HI TD, > > Some observations: > > 1. If I submit the application using spark-submit tool with *client as > deploy mode* it works fine with single master and worker (driver,

Re: Spark Streaming updatyeStateByKey throws OutOfMemory Error

asing the number of > partitions (specify number of partitions in updateStateByKey) ? > > On Wed, Apr 22, 2015 at 2:34 AM, Sourav Chandra < > sourav.chan...@livestream.com> wrote: > >> Anyone? >> >> On Wed, Apr 22, 2015 at 12:29 PM, Sourav Chandra < >>

Re: Spark Streaming updatyeStateByKey throws OutOfMemory Error

Anyone? On Wed, Apr 22, 2015 at 12:29 PM, Sourav Chandra < sourav.chan...@livestream.com> wrote: > Hi Olivier, > > *the update function is as below*: > > *val updateFunc = (values: Seq[IConcurrentUsers], state: Option[(Long, > Long)]) => {* > * val previousC

Re: Spark Streaming updatyeStateByKey throws OutOfMemory Error

aught exception in thread Thread[Executor task launch worker-1,5,main]* On Wed, Apr 22, 2015 at 1:32 AM, Olivier Girardot wrote: > Hi Sourav, > Can you post your updateFunc as well please ? > > Regards, > > Olivier. > > Le mar. 21 avr. 2015 à 12:48, Sourav Chandra

Spark Streaming updatyeStateByKey throws OutOfMemory Error

* * }* We tried different combinations like - with spark 1.1.0 and 1.1.1. - by increasing executor memory - by changing the serialization strategy (switching between kryo and normal java) - by changing broadcast strategy (switching between http and torrent broadcast) Can anyone give any insigh

spark logging issue

, the log files (i.e. stderr) file is not rolled over. What am I missing here? -- Sourav Chandra Senior Software Engineer · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · sourav.chan...@livestream.com o: +91 80 4121 8723 m: +91 988 699 3746 skype: sourav.chandra Livestream

Spark streaming for v1.1.1 - unable to start application

)* *at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)* *at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)* Could you please let me know the reason and fix for this? Spark version is 1.1.1 -- Sourav Chandra Senior Software

Spark Streaming with Kafka is failing with Error

(ThreadPoolExecutor.java:1110)* * at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)* * at java.lang.Thread.run(Thread.java:722)* Can you guys please help me out here? -- Sourav Chandra Senior Software Engineer

[Spark/ Spark Streaming] Spark 1.1.0 fails working with akka 2.3.6

(ApplicationContext.scala:63) ~[analytics-engine.jar:1.0.0]* * ... 13 common frames omitted* Thanks, -- Sourav Chandra Senior Software Engineer · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · sourav.chan...@livestream.com o: +91 80 4121 8723 m: +91 988 699 3746 skype

Re: What's wrong with my spark filter? I get "org.apache.spark.SparkException: Task not serializable"

a.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) > ... > > > > best, > /Shahab > > -- Sourav Chandra Senior Software Engineer · · · · · · · · · · · · · ·

Re: Debugging "Task not serializable"

gt;>> not serializable: java.io.NotSerializableException" errors, apart from >>>> commenting and uncommenting parts of the program, or just turning >>>> everything into Serializable. I find this kind of error very hard to debug, >>>> as these are origi

Questions regarding different spark pre-built packages

, -- Sourav Chandra Senior Software Engineer · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · sourav.chan...@livestream.com o: +91 80 4121 8723 m: +91 988 699 3746 skype: sourav.chandra Livestream "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main,

Re: hi

ost:7077: > akka.remote.EndpointAssociationException: Association failed with > [akka.tcp://sparkMaster@localhost:7077] > 14/06/23 10:45:36 WARN client.AppClient$ClientActor: Could not connect to > akka.tcp://sparkMaster@localhost:7077: > akka.remote.EndpointAssociati

Re: broadcast in spark streaming

variable in spark streaming? > > thanks > -- Sourav Chandra Senior Software Engineer · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · sourav.chan...@livestream.com o: +91 80 4121 8723 m: +91 988 699 3746 skype: sourav.chandra Livestream "Ajmera Summit&qu

Spark streaming issue

ROCESS_LOCAL ls230-127-p.nyc0.ls.local 2014/05/27 07:22:3754 ms Thanks, -- Sourav Chandra Senior Software Engineer · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · sourav.chan...@livestream.com o: +91 80 4121 8723 m: +91 988 699 3746 skype: sourav.chandra Livestream &q

Spark Streaming Error: SparkException: Error sending message to BlockManagerMaster

/BlockManagerMaster#1305432112]] had already been terminated. at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134) at org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:161) ... 39 more Thanks, -- Sourav Chandra Senior Software Engineer

Re: what does broadcast_0 stand for

36.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > -- Sourav Chandra Senior Software Engineer · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · sourav.chan...@livestream.com o: +91 80 4121 8723 m: +91 988 699 3746 skype: sourav.cha

Re: what does broadcast_0 stand for

ist.1001560.n3.nabble.com/what-does-broadcast-0-stand-for-tp4934.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > -- Sourav Chandra Senior Software Engineer · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · sourav.chan...@livestream.com o:

Re: Access Last Element of RDD

Also same thing can be done using rdd.top(1)(reverseOrdering) On Thu, Apr 24, 2014 at 11:28 AM, Sourav Chandra < sourav.chan...@livestream.com> wrote: > You can use rdd.takeOrdered(1)(reverseOrdrering) > > reverseOrdering is you Ordering[T] instance where you define the orderin

Re: Access Last Element of RDD

4, 2014 at 9:51 AM, Sai Prasanna >>>> wrote: >>>> >>>>> Hi All, Some help ! >>>>> RDD.first or RDD.take(1) gives the first item, is there a straight >>>>> forward way to access the last element in a similar way ? >>>>>

Re: about rdd.filter()

> > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/about-rdd-filter-tp4657.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > -- Sourav Chandra Senior Software Engineer · · · · · · · · · · ·

Re: Change print() in JavaNetworkWordCount

ollect().foreach(println) > 594 //if (first100.size > 99) println("(...)") > 595 println() > 596 } > 597 new ForEachDStream(this, > context.sparkContext.clean(foreachFunc)).register() > 598 } > > I would like to print whole the RDD, I ha

Re: How to set environment variable for a spark job

ot;, ldpath) > > Any idea how to set it using java.library.path ? > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/How-to-set-environment-variable-for-a-spark-job-tp3180p3241.html > Sent from the Apache Spark User List mailing l

Re: Change print() in JavaNetworkWordCount

r if anyone > has answered me. I would like changing in the function print() the quantity > of words and the frequency number that are sent to driver's screen. The > default value is 10. > > Anyone could help me with this? > > Best Regards > > -- > Informativa sulla Pri

Re: How to set environment variable for a spark job

1) ORACLE_HOME > 2) LD_LIBRARY_PATH > > thanks > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/How-to-set-environment-variable-for-a-spark-job-tp3180.html > Sent from the Apache Spark User List mailing list archive at Nabbl

Re: spark executor/driver log files management

Mon, Mar 24, 2014 at 12:20 AM, Sourav Chandra < > sourav.chan...@livestream.com> wrote: > >> Hi, >> >> I have few questions regarding log file management in spark: >> >> 1. Currently I did not find any way to modify the lof file name for >> executor/

Re: Java API - Serialization Issue

ndent jars as > well? > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Java-API-Serialization-Issue-tp1460p3086.html > Sent from the Apache Spark User List mailing

spark executor/driver log files management

unmanageable. Is there any way to overcome this? Thanks, -- Sourav Chandra Senior Software Engineer · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · sourav.chan...@livestream.com o: +91 80 4121 8723 m: +91 988 699 3746 skype: sourav.chandra Livestream "Ajmera Summit", F

Re: Connect Exception Error in spark interactive shell...

gt; at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:434) > at > org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:560) > at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184) > at org.apache.hadoop.ipc.Client.getConnection(Client.j

Re: Spark shell exits after 1 min

i Prasanna. AN* >> *II M.Tech (CS), SSSIHL* >> >> > > > -- > *Sai Prasanna. AN* > *II M.Tech (CS), SSSIHL* > > > *Entire water in the ocean can never sink a ship, Unless it gets inside. > All the pressures of life can never hurt you, Unless you let them in

Re: is spark.cleaner.ttl safe?

7;s still in use. The effect is I get bizarre >>> exceptions from Spark complaining about missing broadcast data or >>> ArrayIndexOutOfBounds. When is spark.cleaner.ttl safe to use? Is it >>> supposed to delete in-use data or is this a bug/shortcoming? >>> &g

Spark usage patterns and questions

when opened those stage details it said stage did not start. What does this mean? Looking forward for some interesting responses :) Thanks, -- Sourav Chandra Senior Software Engineer · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · sourav.chan...@livestream.com o: +91 80

Re: [External] Re: no stdout output from worker

e some System.out.println in my Java code that is working ok in a > >>local > >> environment. But when I run the same code on a standalone mode in a EC2 > >> cluster I do not see them at the worker stdout (in the worker node under > >> /work ) or at the driver console.