[
https://issues.apache.org/jira/browse/TOREE-438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163072#comment-16163072
]
Ribamar Santarosa commented on TOREE-438:
-----------------------------------------
There is a failure on the CI that doesn't look related to that patch:
{code}
failed to register layer: Error processing tar file(exit status 1): write
/opt/conda/envs/python2/lib/python2.7/site-packages/Cython/Compiler/Code.so: no
space left on device
{code}
Unless writing the paths of those 2 env vars are so big that is consuming all
the storage! =)
> CLONE - How to support Spark on Yarn model?
> -------------------------------------------
>
> Key: TOREE-438
> URL: https://issues.apache.org/jira/browse/TOREE-438
> Project: TOREE
> Issue Type: Bug
> Reporter: Ribamar Santarosa
>
> It looks like the TOREE-97 issue -- support for Spark Yarn was closed without
> definitive solution (or something went wrong on the way). Toree does support
> it, but it won't work if a user doesn't add manually in their kernel.json
> definition, the env vars for {{HADOOP_CONF_DIR}}. Without that env var, Spark
> doesn't know what to do with the option {{--master=yarn}} (set in
> {{__TOREE_SPARK_OPTS__}}). It would be desirable to have it by default.
> Probably this is not the nicest way to solve the problem, because it just
> hard codes more vars into the JSON file -- ideally it would be nice to have
> an interface to add or remove env vars from those files, however,
> {{HADOOP_CONF_DIR}} and {{SPARK_CONF_DIR}} look basic to be exported. Even
> for an Spark Standalone deployment, {{HADOOP_CONF_DIR}} won't hurt. So, here
> it goes our 2 cents to improve a bit the situation.
> I cloned the TOREE-97 into TOREE-438 to sign this issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
