[
https://issues.apache.org/jira/browse/LIVY-450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16415723#comment-16415723
]
Ryan Skraba commented on LIVY-450:
----------------------------------
Hello! Thanks for taking a look – I just wanted to add my voice why this is a
*good* feature for us.
In our system, we're launching the livy server in different ways, configured
for different types of user groups. We've had quite a bit of success deploying
these different scenarios with docker (and kubernetes), and it's much easier
for us to configure the *same* image *different* ways via environment
variables. To be clear – it's the admin, not users submitting jobs, that will
be setting these environment variables (via docker-compose or other) and
they're only read when the livy server starts.
If Spark is moving to deprecate environment variables, it's to simplify the
setup for the user submitting spark jobs – this is the right thing to do!
Usually, they'll be able to get all of the default spark configuration directly
from their cluster (or from the cluster admins) and it'll have reasonable
default values. Encapsulating the expertise from the admins in property files
makes this simpler for everyone.
A job server like Livy, serving many users, also shares this goal (simplifying
job submission for the user, encapsulating the cluster config from the
experts). Only the livy admin will be touching the livy.conf – or launching
the livy server process, with or without environment variables. (The blacklist
is kind of irrelevant here, since it applies to the user submitting jobs, not
the admin starting livy).
I hope this helps! There's a simple and clear workaround for us: to mount a
pre-configured livy.conf or modify the existing livy.conf during the image
entrypoint. Both of those feel relatively inelegant in comparison to
[~aromanenko]'s proposed solution, and a long-lived service like Livy, with a
single point of launch, can very likely justify the use of configuration via
environment.
> Override config options by environment variables.
> -------------------------------------------------
>
> Key: LIVY-450
> URL: https://issues.apache.org/jira/browse/LIVY-450
> Project: Livy
> Issue Type: Improvement
> Reporter: Alexey Romanenko
> Priority: Major
>
> Add a possibility to override all config options by environment variables.
> For example:
> {{LIVY_SPARK_MASTER}} automatically overrides {{livy.spark.master}} and
> {{LIVY_SPARK_DEPLOY_MODE}} overrides {{livy.spark.deploy-mode}}.
> It can be quite useful to use the same Livy image in different modes (local
> and cluster).
> Any pros/cons for or against that?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
