Re: Master OOM in "master-rebuild-ui-thread" while running stream app

2016-09-13 Thread Mariano Semelman 
Thanks, I would go with log disabling.
BTW, the master crashed while the application was still running.

--

*Mariano Semelman*
P13N - IT
Av. Corrientes Nº 746 - piso 13 - C.A.B.A. (C1043AAU)
Teléfono (54) 11- *4894-3500*


[image: Seguinos en Twitter!]  [image:
Seguinos en Facebook!]  [image: Seguinos
en YouTube!] 
*Despegar.com*
El mejor precio para tu viaje.

Este mensaje es confidencial y puede contener información amparada por el
secreto profesional. Si usted ha recibido este e-mail por error, por favor
comuníquenoslo inmediatamente respondiendo a este e-mail y luego
eliminándolo de su sistema. El contenido de este mensaje no deberá ser
copiado ni divulgado a ninguna persona.

On 13 September 2016 at 12:52, Bryan Cutler  wrote:

> It looks like you have logging enabled and your application event log is
> too large for the master to build a web UI from it.  In spark 1.6.2 and
> earlier, when an application completes, the master rebuilds a web UI to
> view events after the fact.  This functionality was removed in spark 2.0
> and the history server should be used instead.  If you are unable to
> upgrade could you try disabling logging?
>
> On Sep 13, 2016 7:18 AM, "Mariano Semelman" 
> wrote:
>
>> Hello everybody,
>>
>> I am running a spark streaming app and I am planning to use it as a long
>> running service. However while trying the app in a rc environment I got
>> this exception in the master daemon after 1 hour of running:
>>
>> ​​Exception in thread "master-rebuild-ui-thread"
>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>> at java.util.regex.Pattern.compile(Pattern.java:1667)
>> at java.util.regex.Pattern.(Pattern.java:1351)
>> at java.util.regex.Pattern.compile(Pattern.java:1054)
>> at java.lang.String.replace(String.java:2239)
>> at org.apache.spark.util.Utils$.getFormattedClassName(Utils.sca
>> la:1632)
>> at org.apache.spark.util.JsonProtocol$.sparkEventFromJson(JsonP
>> rotocol.scala:486)
>> at org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayLi
>> stenerBus.scala:58)
>> at org.apache.spark.deploy.master.Master$$anonfun$17.apply(
>> Master.scala:972)
>> at org.apache.spark.deploy.master.Master$$anonfun$17.apply(
>> Master.scala:952)
>> at scala.concurrent.impl.Future$PromiseCompletingRunnable.lifte
>> dTree1$1(Future.scala:24)
>> at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(F
>> uture.scala:24)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1142)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>> As a palliative measure I've increased the master memory to 1.5gb.
>> My job is running with a batch interval of 5 seconds.
>> I'm using spark version 1.6.2.
>>
>> I think it might be related to this issues:
>>
>> https://issues.apache.org/jira/browse/SPARK-6270
>> https://issues.apache.org/jira/browse/SPARK-12062
>> https://issues.apache.org/jira/browse/SPARK-12299
>>
>> But I don't see a clear road to solve this apart from upgrading spark.
>> What would you recommend?
>>
>>
>> Thanks in advance
>> Mariano
>>
>>

Re: Master OOM in "master-rebuild-ui-thread" while running stream app

2016-09-13 Thread Bryan Cutler 
It looks like you have logging enabled and your application event log is
too large for the master to build a web UI from it.  In spark 1.6.2 and
earlier, when an application completes, the master rebuilds a web UI to
view events after the fact.  This functionality was removed in spark 2.0
and the history server should be used instead.  If you are unable to
upgrade could you try disabling logging?

On Sep 13, 2016 7:18 AM, "Mariano Semelman" 
wrote:

> Hello everybody,
>
> I am running a spark streaming app and I am planning to use it as a long
> running service. However while trying the app in a rc environment I got
> this exception in the master daemon after 1 hour of running:
>
> ​​Exception in thread "master-rebuild-ui-thread"
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.regex.Pattern.compile(Pattern.java:1667)
> at java.util.regex.Pattern.(Pattern.java:1351)
> at java.util.regex.Pattern.compile(Pattern.java:1054)
> at java.lang.String.replace(String.java:2239)
> at org.apache.spark.util.Utils$.getFormattedClassName(Utils.
> scala:1632)
> at org.apache.spark.util.JsonProtocol$.sparkEventFromJson(
> JsonProtocol.scala:486)
> at org.apache.spark.scheduler.ReplayListenerBus.replay(
> ReplayListenerBus.scala:58)
> at org.apache.spark.deploy.master.Master$$anonfun$17.
> apply(Master.scala:972)
> at org.apache.spark.deploy.master.Master$$anonfun$17.
> apply(Master.scala:952)
> at scala.concurrent.impl.Future$PromiseCompletingRunnable.
> liftedTree1$1(Future.scala:24)
> at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(
> Future.scala:24)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
>
> As a palliative measure I've increased the master memory to 1.5gb.
> My job is running with a batch interval of 5 seconds.
> I'm using spark version 1.6.2.
>
> I think it might be related to this issues:
>
> https://issues.apache.org/jira/browse/SPARK-6270
> https://issues.apache.org/jira/browse/SPARK-12062
> https://issues.apache.org/jira/browse/SPARK-12299
>
> But I don't see a clear road to solve this apart from upgrading spark.
> What would you recommend?
>
>
> Thanks in advance
> Mariano
>
>