[jira] [Updated] (FLINK-6222) YARN: setting environment variables in an easier fashion
[ https://issues.apache.org/jira/browse/FLINK-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Foster updated FLINK-6222: Attachment: patch0-add-yarn-hadoop-conf.diff > YARN: setting environment variables in an easier fashion > > > Key: FLINK-6222 > URL: https://issues.apache.org/jira/browse/FLINK-6222 > Project: Flink > Issue Type: Improvement > Components: Startup Shell Scripts >Affects Versions: 1.2.0 > Environment: YARN, EMR >Reporter: Craig Foster > Attachments: patch0-add-yarn-hadoop-conf.diff > > > Right now we require end-users to set YARN_CONF_DIR or HADOOP_CONF_DIR and > sometimes FLINK_CONF_DIR. > For example, in [1], it is stated: > “Please note that the Client requires the YARN_CONF_DIR or HADOOP_CONF_DIR > environment variable to be set to read the YARN and HDFS configuration.” > In BigTop, we set this with /etc/flink/default and then a wrapper is created > to source that. However, this is slightly cumbersome and we don't have a > central place within the Flink project itself to source environment > variables. config.sh could do this but it doesn't have information about > FLINK_CONF_DIR. For YARN and Hadoop variables, I already have a solution that > would add "env.yarn.confdir" and "env.hadoop.confdir" variables to the > flink-conf.yaml file and then we just symlink /etc/lib/flink/conf/ and > /etc/flink/conf. > But we could also add a flink-env.sh file to set these variables and decouple > them from config.sh entirely. > I'd like to know the opinion/preference of others and what would be more > amenable. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-6222) YARN: setting environment variables in an easier fashion
[ https://issues.apache.org/jira/browse/FLINK-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16257610#comment-16257610 ] Craig Foster commented on FLINK-6222: - I'm submitting a patch here to see if we can move this along. > YARN: setting environment variables in an easier fashion > > > Key: FLINK-6222 > URL: https://issues.apache.org/jira/browse/FLINK-6222 > Project: Flink > Issue Type: Improvement > Components: Startup Shell Scripts >Affects Versions: 1.2.0 > Environment: YARN, EMR >Reporter: Craig Foster > > Right now we require end-users to set YARN_CONF_DIR or HADOOP_CONF_DIR and > sometimes FLINK_CONF_DIR. > For example, in [1], it is stated: > “Please note that the Client requires the YARN_CONF_DIR or HADOOP_CONF_DIR > environment variable to be set to read the YARN and HDFS configuration.” > In BigTop, we set this with /etc/flink/default and then a wrapper is created > to source that. However, this is slightly cumbersome and we don't have a > central place within the Flink project itself to source environment > variables. config.sh could do this but it doesn't have information about > FLINK_CONF_DIR. For YARN and Hadoop variables, I already have a solution that > would add "env.yarn.confdir" and "env.hadoop.confdir" variables to the > flink-conf.yaml file and then we just symlink /etc/lib/flink/conf/ and > /etc/flink/conf. > But we could also add a flink-env.sh file to set these variables and decouple > them from config.sh entirely. > I'd like to know the opinion/preference of others and what would be more > amenable. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-6222) YARN: setting environment variables in an easier fashion
Craig Foster created FLINK-6222: --- Summary: YARN: setting environment variables in an easier fashion Key: FLINK-6222 URL: https://issues.apache.org/jira/browse/FLINK-6222 Project: Flink Issue Type: Improvement Components: Startup Shell Scripts Affects Versions: 1.2.0 Environment: YARN, EMR Reporter: Craig Foster Right now we require end-users to set YARN_CONF_DIR or HADOOP_CONF_DIR and sometimes FLINK_CONF_DIR. For example, in [1], it is stated: “Please note that the Client requires the YARN_CONF_DIR or HADOOP_CONF_DIR environment variable to be set to read the YARN and HDFS configuration.” In BigTop, we set this with /etc/flink/default and then a wrapper is created to source that. However, this is slightly cumbersome and we don't have a central place within the Flink project itself to source environment variables. config.sh could do this but it doesn't have information about FLINK_CONF_DIR. For YARN and Hadoop variables, I already have a solution that would add "env.yarn.confdir" and "env.hadoop.confdir" variables to the flink-conf.yaml file and then we just symlink /etc/lib/flink/conf/ and /etc/flink/conf. But we could also add a flink-env.sh file to set these variables and decouple them from config.sh entirely. I'd like to know the opinion/preference of others and what would be more amenable. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-5013) Flink Kinesis connector doesn't work on old EMR versions
[ https://issues.apache.org/jira/browse/FLINK-5013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15669623#comment-15669623 ] Craig Foster commented on FLINK-5013: - Is there a fix/issue pending for correctly shading in flink-dist? What work is involved in there? The issue potentially affects downstream users such as people using BigTop and not just EMR. > Flink Kinesis connector doesn't work on old EMR versions > > > Key: FLINK-5013 > URL: https://issues.apache.org/jira/browse/FLINK-5013 > Project: Flink > Issue Type: Bug > Components: Kinesis Connector >Reporter: Robert Metzger >Assignee: Tzu-Li (Gordon) Tai >Priority: Critical > > A user reported on the mailing list that our Kinesis connector doesn't work > with EMR 4.4.0: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kinesis-Connector-Dependency-Problems-td9790.html > The problem seems to be that Flink is loading older libraries from the "YARN > container classpath", which on EMR contains the default Amazon libraries. > We should try to shade kinesis and its amazon dependencies into a different > namespace. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5013) Flink Kinesis connector doesn't work on old EMR versions
[ https://issues.apache.org/jira/browse/FLINK-5013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664360#comment-15664360 ] Craig Foster commented on FLINK-5013: - I can fix and verify on the EMR side but shouldn't the Maven enforcer plugin be used to prevent Flink from building using versions that don't properly shade? > Flink Kinesis connector doesn't work on old EMR versions > > > Key: FLINK-5013 > URL: https://issues.apache.org/jira/browse/FLINK-5013 > Project: Flink > Issue Type: Bug > Components: Kinesis Connector >Reporter: Robert Metzger >Assignee: Tzu-Li (Gordon) Tai >Priority: Critical > > A user reported on the mailing list that our Kinesis connector doesn't work > with EMR 4.4.0: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kinesis-Connector-Dependency-Problems-td9790.html > The problem seems to be that Flink is loading older libraries from the "YARN > container classpath", which on EMR contains the default Amazon libraries. > We should try to shade kinesis and its amazon dependencies into a different > namespace. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
