nchez [mailto:langel.gro...@gmail.com]
> *Enviado el:* lunes, 20 de abril de 2015 15:46
> *Para:* Marius Soutier
> *CC:* González Salgado, Miquel; bit1...@163.com; user
> *Asunto:* Re: Streaming problems running 24x7
>
>
>
> You have a window operation; I have seen that beha
[mailto:langel.gro...@gmail.com]
Enviado el: lunes, 20 de abril de 2015 15:46
Para: Marius Soutier
CC: González Salgado, Miquel; bit1...@163.com; user
Asunto: Re: Streaming problems running 24x7
You have a window operation; I have seen that behaviour before with window
operations in spark streaming. My
rocess last longer..
>
> Best regards,
> Miquel
>
>
> *De:* bit1...@163.com [mailto:bit1...@163.com ]
> *Enviado el:* jueves, 16 de abril de 2015 10:58
> *Para:* González Salgado, Miquel; user
> *Asunto:* Re: Streaming problems running 24x7
>
> From your description, loo
st longer..
>
> Best regards,
> Miquel
>
>
> De: bit1...@163.com <mailto:bit1...@163.com> [mailto:bit1...@163.com
> <mailto:bit1...@163.com>]
> Enviado el: jueves, 16 de abril de 2015 10:58
> Para: González Salgado, Miquel; user
> Asunto: Re: Streamin
2015 10:58
Para: González Salgado, Miquel; user
Asunto: Re: Streaming problems running 24x7
>From your description, looks like the data processing speed is far behind the
>data receiving speed
Could you try to increase the core number when you submit the application? such
as l
Asunto: Re: Streaming problems running 24x7
I used to hit this issue when my processing time exceeds the batch duration.
Here's a few workarounds:
- Use storage level MEMORY_AND_DISK
- Enable WAL and check pointing
Above two will slow down things a little bit.
If you want low latency, wha
> ,List("dstport")
> ,List("total")
> )
>
>val l_valcamps = List ("doctets"
> ,"dpkts"
> ,"flows&q
dow((a:Long,b:Long)=>a+b, Seconds(60),
Seconds(10))
.map(_.swap)
.transform(_.sortByKey(false))
.foreachRDD(rdd => {
val data = rdd.take(20)
writeOutput (data, keycamps_s
