You could use foreachRDD to do the operations and then inside the
foreach create an accumulator to gather all the errors together
dstream.foreachRDD { rdd =>
val accumulator = new Accumulator[....]
rdd.map { ..... }.count // whatever operation that is error prone
// gather all errors from accumulator and show on webpage, push to
database?
}
On Tue, Jan 27, 2015 at 9:34 PM, Tobias Pfeiffer <[email protected]> wrote:
> Hi,
>
> On Wed, Jan 28, 2015 at 1:45 PM, Soumitra Kumar <[email protected]>
> wrote:
>>
>> It is a Streaming application, so how/when do you plan to access the
>> accumulator on driver?
>
>
> Well... maybe there would be some user command or web interface showing the
> errors that have happened during processing...?
>
> Thanks
> Tobias
>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
