Hi,
I have solved the problem. I used an object which contained a huge
generic MutableList. When kryo serializes the MutableList it does many
recursive calls which lead to a huge stack and to the
StackOverFlowError. I changed the MutableList to a Set and now kryo does
not do so many recursive calls and the stack keeps small :)
Best Regards,
Hilmi
Am 25.01.2016 um 16:34 schrieb Stephan Ewen:
@Hilmi:
A stack size of 1GB is too much, maybe the JVM ignores it? Can you try
"-Xss2m" or so?
Is this to prevent stack overflow errors in some code?
Stephan
On Fri, Jan 22, 2016 at 2:21 PM, Robert Metzger <rmetz...@apache.org> wrote:
Hi,
yes, looks like the argument is passed correctly.
By the way: Why are you sending your answers always twice to the mailing
list?
On Fri, Jan 22, 2016 at 2:17 PM, Hilmi Yildirim <
hilmi.yildirim.d...@gmail.com> wrote:
Hi Robert,
in the logs I found this
13:54:41,291 INFO org.apache.flink.yarn.YarnTaskManagerRunner
- JVM Options:
13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner
- -Xms3072m
13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner
- -Xmx3072m
13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner
- -XX:MaxDirectMemorySize=3072m
13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner
- -Xss1024m
13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner
-
-Dlog.file=/var/log/hadoop-yarn/container/application_1453384265041_0030/container_1453384265041_0030_01_000002/taskmanager.log
13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner
- -Dlogback.configurationFile=file:logback.xml
13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner
- -Dlog4j.configuration=file:log4j.properties
It seems that the argument is passed to the Taskmanager. Or am I wrong?
Best Regards,
Hilmi
Am 22.01.2016 um 12:25 schrieb Robert Metzger:
Hi Hilmi,
it means that each JVM the YARN session is starting (JobManager and
TaskManagers) is initialized with that parameter.
Can you check the logs of the TaskManager to see if the option has been
passed properly?
On Fri, Jan 22, 2016 at 12:22 PM, Hilmi Yildirim <
hilmi.yildi...@dfki.de>
wrote:
Hi,
I added the following line to my flink-conf.yaml:
env.java.opts: "-Xss1024m"
But it does not work. Does this line mean that every single slot of
each
task manager uses a stack size of 1 GB?
Best Regards,
Hilmi
Am 22.01.2016 um 11:46 schrieb Robert Metzger:
Hi Hilmi,
Flink on YARN is respecting the env.java.opts configuration parameter
from
the flink-conf.yaml.
2016-01-22 11:44 GMT+01:00 Hilmi Yildirim <hilmi.yildi...@dfki.de>:
Hi,
does anybody know how I can increase the stack size of a yarn
session?
Best Regards,
Hilmi
--
==================================================================
Hilmi Yildirim, M.Sc.
Researcher
DFKI GmbH
Intelligente Analytik für Massendaten
DFKI Projektbüro Berlin
Alt-Moabit 91c
D-10559 Berlin
Phone: +49 30 23895 1814
E-Mail: hilmi.yildi...@dfki.de
-------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
Geschaeftsfuehrung:
Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats:
Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313
-------------------------------------------------------------
--
==================================================================
Hilmi Yildirim, M.Sc.
Researcher
DFKI GmbH
Intelligente Analytik für Massendaten
DFKI Projektbüro Berlin
Alt-Moabit 91c
D-10559 Berlin
Phone: +49 30 23895 1814
E-Mail: hilmi.yildi...@dfki.de
-------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
Geschaeftsfuehrung:
Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats:
Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313
-------------------------------------------------------------
--
==================================================================
Hilmi Yildirim, M.Sc.
Researcher
DFKI GmbH
Intelligente Analytik für Massendaten
DFKI Projektbüro Berlin
Alt-Moabit 91c
D-10559 Berlin
Phone: +49 30 23895 1814
E-Mail: hilmi.yildi...@dfki.de
-------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
Geschaeftsfuehrung:
Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats:
Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313
-------------------------------------------------------------
