Thanks! They look reasonable to me. Please feel free to open a PR
________________________________
From: Belousov Maksim Eduardovich <[email protected]>
Sent: Saturday, December 30, 2017 5:20:02 AM
To: [email protected]
Subject: [DISCUSS] Increase a few numbers in source code
Hello, team!
There are some numbers in source code. The influence of these numbers is
non-obvious and very important.
1. SchedulerFactory.java [1]:
executor = ExecutorFactory.singleton().createOrGet("SchedulerFactory", 100);
The meaning of "100" is that Zeppelin server can get only 100 started
interpreter processes. When analysts run 100 jvm/interpreter processes then
Zeppelin will be fully stuck: no paragraph can run now even if the paragraph
run a few minutes ago.
I wrote about this case previously [2].
2. ZeppelinConfiguration.java [3]
ZEPPELIN_INTERPRETER_MAX_POOL_SIZE("zeppelin.interpreter.max.poolsize", 10),
"10" - is the number of paragraphs that will be run under cron scheduling.
This behavior arises after apply of paragraph sequential run.
Also there is no description for "interpreter.max.poolsize".
3. spark.port.maxRetries = 16
It's about use of local spark interpreter.
By default server can start only 16 local spark instance and interpreters. The
analysts cannot start the 17-th new spark interpreter and gets error.
It's non-obvious what setting affects on max number of spark processes.
The most valuable server resources are RAM and CPU. Above settings don't
optimize use of RAM/CPU and are more like bugs. It would be good to refactor
code and not to use settings 1 and 2 at all, but this issue is low priority.
Therefore I want to put a very big number, for example 65536, in all cases.
Share please your thoughts about increasing default values in these cases.
1.
https://github.com/apache/zeppelin/blob/master/zeppelin-interpreter/src/main/java/org/apache/zeppelin/scheduler/SchedulerFactory.java#L55
2.
https://lists.apache.org/thread.html/30966875b50b5ac8b4326c23a012d702ea7cac24d75f540ae58f31b3@%3Cusers.zeppelin.apache.org%3E
3.
https://github.com/apache/zeppelin/blob/dd1be03dee9428ade92b8fd47d148c2325179d19/zeppelin-interpreter/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java#L667
Regards,
Maksim Belousov
