Re: SPARK STREAMING PROBLEM

the logs. Am I doing something wrong? And is there any better way to extract the file names from DStream ? Thanks in advance Animesh -- Sourav Chandra Senior Software Engineer · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · sourav.chan...@livestream.com o: +91 80 4121

Re: SPARK STREAMING PROBLEM

gets printed on the console apart from the logs. Am I doing something wrong? And is there any better way to extract the file names from DStream ? Thanks in advance Animesh -- Sourav Chandra Senior Software Engineer

Re: Best practices on testing Spark jobs

? Should I manually create mocks by e.g. extending all the classes I'd normally mock and changing the implementation of some methods? I don't like this idea but I can't really see any other options now. Kind regards, Michał Michalski, michal.michal...@boxever.com -- Sourav Chandra

Re: Spark Streaming updatyeStateByKey throws OutOfMemory Error

increasing the number of partitions (specify number of partitions in updateStateByKey) ? On Wed, Apr 22, 2015 at 2:34 AM, Sourav Chandra sourav.chan...@livestream.com wrote: Anyone? On Wed, Apr 22, 2015 at 12:29 PM, Sourav Chandra sourav.chan...@livestream.com wrote: Hi Olivier

Re: Spark Streaming updatyeStateByKey throws OutOfMemory Error

*bump* On Thu, Apr 23, 2015 at 3:46 PM, Sourav Chandra sourav.chan...@livestream.com wrote: HI TD, Some observations: 1. If I submit the application using spark-submit tool with *client as deploy mode* it works fine with single master and worker (driver, master and worker are running

Re: Spark Streaming updatyeStateByKey throws OutOfMemory Error

Anyone? On Wed, Apr 22, 2015 at 12:29 PM, Sourav Chandra sourav.chan...@livestream.com wrote: Hi Olivier, *the update function is as below*: *val updateFunc = (values: Seq[IConcurrentUsers], state: Option[(Long, Long)]) = {* * val previousCount = state.getOrElse((0L, 0L))._2

Spark Streaming updatyeStateByKey throws OutOfMemory Error

) - by changing broadcast strategy (switching between http and torrent broadcast) Can anyone give any insight what we are missing here? How can we fix this? Due to akka version mismatch with some other libraries we cannot upgrade the spark version. Thanks, -- Sourav Chandra Senior Software Engineer

spark logging issue

, the log files (i.e. stderr) file is not rolled over. What am I missing here? -- Sourav Chandra Senior Software Engineer · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · sourav.chan...@livestream.com o: +91 80 4121 8723 m: +91 988 699 3746 skype: sourav.chandra Livestream

[Spark/ Spark Streaming] Spark 1.1.0 fails working with akka 2.3.6

$.createSimpleContext(ApplicationContext.scala:63) ~[analytics-engine.jar:1.0.0]* * ... 13 common frames omitted* Thanks, -- Sourav Chandra Senior Software Engineer · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · sourav.chan...@livestream.com o: +91 80 4121 8723 m: +91 988 699 3746

Spark Streaming with Kafka is failing with Error

(ThreadPoolExecutor.java:1110)* * at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)* * at java.lang.Thread.run(Thread.java:722)* Can you guys please help me out here? -- Sourav Chandra Senior Software Engineer

Re: What's wrong with my spark filter? I get org.apache.spark.SparkException: Task not serializable

(ObjectOutputStream.java:1177) ... best, /Shahab -- Sourav Chandra Senior Software Engineer · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · sourav.chan...@livestream.com o: +91 80 4121 8723 m: +91 988 699 3746 skype: sourav.chandra Livestream Ajmera Summit, First Floor, #3

Questions regarding different spark pre-built packages

, -- Sourav Chandra Senior Software Engineer · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · sourav.chan...@livestream.com o: +91 80 4121 8723 m: +91 988 699 3746 skype: sourav.chandra Livestream Ajmera Summit, First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd Block

Spark streaming issue

-127-p.nyc0.ls.local 2014/05/27 07:22:3754 ms Thanks, -- Sourav Chandra Senior Software Engineer · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · sourav.chan...@livestream.com o: +91 80 4121 8723 m: +91 988 699 3746 skype: sourav.chandra Livestream Ajmera Summit, First

Spark Streaming Error: SparkException: Error sending message to BlockManagerMaster

/user/BlockManagerMaster#1305432112]] had already been terminated. at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134) at org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:161) ... 39 more Thanks, -- Sourav Chandra Senior Software Engineer

Re: what does broadcast_0 stand for

List mailing list archive at Nabble.com. -- Sourav Chandra Senior Software Engineer · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · sourav.chan...@livestream.com o: +91 80 4121 8723 m: +91 988 699 3746 skype: sourav.chandra Livestream Ajmera Summit, First Floor, #3/D

Re: Access Last Element of RDD

. !! -- Sourav Chandra Senior Software Engineer · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · sourav.chan...@livestream.com o: +91 80 4121 8723 m: +91 988 699 3746 skype: sourav.chandra Livestream Ajmera Summit, First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd Block

Re: Access Last Element of RDD

Also same thing can be done using rdd.top(1)(reverseOrdering) On Thu, Apr 24, 2014 at 11:28 AM, Sourav Chandra sourav.chan...@livestream.com wrote: You can use rdd.takeOrdered(1)(reverseOrdrering) reverseOrdering is you Ordering[T] instance where you define the ordering logic. This you

Re: about rdd.filter()

in context: http://apache-spark-user-list.1001560.n3.nabble.com/about-rdd-filter-tp4657.html Sent from the Apache Spark User List mailing list archive at Nabble.com. -- Sourav Chandra Senior Software Engineer · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · sourav.chan

Re: Change print() in JavaNetworkWordCount

. I would like changing in the function print() the quantity of words and the frequency number that are sent to driver's screen. The default value is 10. Anyone could help me with this? Best Regards -- Informativa sulla Privacy: http://www.unibs.it/node/8155 -- Sourav Chandra Senior

spark executor/driver log files management

unmanageable. Is there any way to overcome this? Thanks, -- Sourav Chandra Senior Software Engineer · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · sourav.chan...@livestream.com o: +91 80 4121 8723 m: +91 988 699 3746 skype: sourav.chandra Livestream Ajmera Summit, First Floor

Spark usage patterns and questions

but when opened those stage details it said stage did not start. What does this mean? Looking forward for some interesting responses :) Thanks, -- Sourav Chandra Senior Software Engineer · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · sourav.chan...@livestream.com o: +91 80

Re: [External] Re: no stdout output from worker

. But when I run the same code on a standalone mode in a EC2 cluster I do not see them at the worker stdout (in the worker node under spark location/work ) or at the driver console. Could you help me understand how do I troubleshoot? Thanks Ranjan -- Sourav Chandra Senior Software