Side effects of using var inside a class object in a Rdd

2016-02-15 Thread Hemalatha A
ani.animalAge+=1 ani }) Is it advisable to use var in this case? Or can I do ani.copy(animalAge=2) which will reallocate the memory altogether for the animal. Please advice which is the best way to handle such cases. Regards Hemalatha

How Application jar is copied to worker machines?

2016-04-10 Thread Hemalatha A
Hello, I want to know on doing spark-submit, how is the Application jar copied to worker machines? Who does the copying of Jars? Similarly who copies DAG from driver to executors? -- Regards Hemalatha

[no subject]

2016-04-02 Thread Hemalatha A
Regards Hemalatha

Spark streaming time displayed is not current system time but it is processing current messages

2016-04-14 Thread Hemalatha A
Spark streaming display? Also, when there is scheduling delay of say 8 hours, what time does Spark display- current rime or hours behind? -- Regards Hemalatha

How to resolve Scheduling delay in Spark streaming applications?

2016-05-10 Thread Hemalatha A
Hemalatha

Spark streaming batch time displayed is not current system time but it is processing current messages

2016-04-16 Thread Hemalatha A
Can anyone help me in debugging this issue please. On Thu, Apr 14, 2016 at 12:24 PM, Hemalatha A < hemalatha.amru...@googlemail.com> wrote: > Hi, > > I am facing a problem in Spark streaming. > Time

Fail a batch in Spark Streaming forcefully based on business rules

2016-07-26 Thread Hemalatha A
Hello, I have a uescase where in, I have to fail certain batches in my streaming batches, based on my application specific business rules. Ex: If in a batch of 2 seconds, I don't receive 100 message, I should fail the batch and move on. How to achieve this behavior? -- Regards Hemalatha

Re: Fail a batch in Spark Streaming forcefully based on business rules

2016-07-28 Thread Hemalatha A
Another usecase why I need to do this is, If Exception A is caught I should just print it and ignore, but ifException B occurs, I have to end the batch, fail it and stop processing the batch. Is it possible to achieve this?? Any hints on this please. On Wed, Jul 27, 2016 at 10:42 AM, Hemalatha

Any exceptions during an action doesn't fail the Spark streaming batch in yarn-client mode

2016-08-07 Thread Hemalatha A
:346) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:254) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) -- Regards Hemalatha

How to use ManualClock with Spark streaming

2017-02-28 Thread Hemalatha A
Is there a way to easily hack this property to use Manual clock. -- Regards Hemalatha

How does chaining of Windowed Dstreams work?

2016-09-04 Thread Hemalatha A
or base it on StreamB since it has a higher and same sliding interval. Which would be a better choice? Or is it the same as window on Base stream? How does it basically work? -- Regards Hemalatha

Re: How does chaining of Windowed Dstreams work?

2016-09-27 Thread Hemalatha A
Hello, Can anyone please answer the below question and help me understand the windowing operations. On Sun, Sep 4, 2016 at 4:42 PM, Hemalatha A < hemalatha.amru...@googlemail.com> wrote: > Hello, > > I have a set of Dstreams on which I'm performing some computation on each &

Re: How to use ManualClock with Spark streaming

2017-04-05 Thread Hemalatha A
Any updates on how can I use ManualClock other than editing the Spark source code? On Wed, Mar 1, 2017 at 10:19 AM, Hemalatha A < hemalatha.amru...@googlemail.com> wrote: > It is certainly possible through a hack. > I was referring to below post where TD says it is possible th