Re: How to use ManualClock with Spark streaming

2017-04-05 Thread Hemalatha A
Any updates on how can I use ManualClock other than editing the Spark source code? On Wed, Mar 1, 2017 at 10:19 AM, Hemalatha A < hemalatha.amru...@googlemail.com> wrote: > It is certainly possible through a hack. > I was referring to below post where TD says it is possible th

How to use ManualClock with Spark streaming

2017-02-28 Thread Hemalatha A
Is there a way to easily hack this property to use Manual clock. -- Regards Hemalatha

Re: How does chaining of Windowed Dstreams work?

2016-09-27 Thread Hemalatha A
Hello, Can anyone please answer the below question and help me understand the windowing operations. On Sun, Sep 4, 2016 at 4:42 PM, Hemalatha A < hemalatha.amru...@googlemail.com> wrote: > Hello, > > I have a set of Dstreams on which I'm performing some computation on each &

How does chaining of Windowed Dstreams work?

2016-09-04 Thread Hemalatha A
or base it on StreamB since it has a higher and same sliding interval. Which would be a better choice? Or is it the same as window on Base stream? How does it basically work? -- Regards Hemalatha

Any exceptions during an action doesn't fail the Spark streaming batch in yarn-client mode

2016-08-07 Thread Hemalatha A
:346) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:254) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) -- Regards Hemalatha

Re: Fail a batch in Spark Streaming forcefully based on business rules

2016-07-28 Thread Hemalatha A
Another usecase why I need to do this is, If Exception A is caught I should just print it and ignore, but ifException B occurs, I have to end the batch, fail it and stop processing the batch. Is it possible to achieve this?? Any hints on this please. On Wed, Jul 27, 2016 at 10:42 AM, Hemalatha

Fail a batch in Spark Streaming forcefully based on business rules

2016-07-26 Thread Hemalatha A
Hello, I have a uescase where in, I have to fail certain batches in my streaming batches, based on my application specific business rules. Ex: If in a batch of 2 seconds, I don't receive 100 message, I should fail the batch and move on. How to achieve this behavior? -- Regards Hemalatha

How to resolve Scheduling delay in Spark streaming applications?

2016-05-10 Thread Hemalatha A
Hemalatha

Spark streaming batch time displayed is not current system time but it is processing current messages

2016-04-16 Thread Hemalatha A
Can anyone help me in debugging this issue please. On Thu, Apr 14, 2016 at 12:24 PM, Hemalatha A < hemalatha.amru...@googlemail.com> wrote: > Hi, > > I am facing a problem in Spark streaming. > Time

Spark streaming time displayed is not current system time but it is processing current messages

2016-04-14 Thread Hemalatha A
Spark streaming display? Also, when there is scheduling delay of say 8 hours, what time does Spark display- current rime or hours behind? -- Regards Hemalatha

How Application jar is copied to worker machines?

2016-04-10 Thread Hemalatha A
Hello, I want to know on doing spark-submit, how is the Application jar copied to worker machines? Who does the copying of Jars? Similarly who copies DAG from driver to executors? -- Regards Hemalatha

[no subject]

2016-04-02 Thread Hemalatha A
Regards Hemalatha

Side effects of using var inside a class object in a Rdd

2016-02-15 Thread Hemalatha A
ani.animalAge+=1 ani }) Is it advisable to use var in this case? Or can I do ani.copy(animalAge=2) which will reallocate the memory altogether for the animal. Please advice which is the best way to handle such cases. Regards Hemalatha