[jira] [Commented] (FLINK-22700) [FLIP-167] Propagate watermarks to Sink API
[ https://issues.apache.org/jira/browse/FLINK-22700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17400495#comment-17400495 ] Eron Wright commented on FLINK-22700: - [~joemoe] thanks for asking, I sincerely don't think additional tests are needed. > [FLIP-167] Propagate watermarks to Sink API > --- > > Key: FLINK-22700 > URL: https://issues.apache.org/jira/browse/FLINK-22700 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Reporter: Eron Wright >Assignee: Eron Wright >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0 > > > Make it possible for sink functions / sink writers to propagate watermarks to > external storage systems, as described in > [FLIP-167|https://cwiki.apache.org/confluence/display/FLINK/FLIP-167%3A+Watermarks+for+Sink+API]. > Note that sink functions already obtain the current watermark upon receiving > a record. This issue is about obtaining the watermark as it is received from > upstream (i.e. not dependent on receipt of a record). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-22700) [FLIP-167] Propagate watermarks to Sink API
[ https://issues.apache.org/jira/browse/FLINK-22700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17361078#comment-17361078 ] Eron Wright commented on FLINK-22700: - FLIP-167 has been accepted by the community, and PR is ready for review. > [FLIP-167] Propagate watermarks to Sink API > --- > > Key: FLINK-22700 > URL: https://issues.apache.org/jira/browse/FLINK-22700 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Reporter: Eron Wright >Assignee: Eron Wright >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0 > > > Make it possible for sink functions / sink writers to propagate watermarks to > external storage systems, as described in > [FLIP-167|https://cwiki.apache.org/confluence/display/FLINK/FLIP-167%3A+Watermarks+for+Sink+API]. > Note that sink functions already obtain the current watermark upon receiving > a record. This issue is about obtaining the watermark as it is received from > upstream (i.e. not dependent on receipt of a record). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22700) [FLIP-167] Propagate watermarks to Sink API
[ https://issues.apache.org/jira/browse/FLINK-22700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eron Wright updated FLINK-22700: Labels: pull-request-available (was: pull-request-available stale-assigned) > [FLIP-167] Propagate watermarks to Sink API > --- > > Key: FLINK-22700 > URL: https://issues.apache.org/jira/browse/FLINK-22700 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Reporter: Eron Wright >Assignee: Eron Wright >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0 > > > Make it possible for sink functions / sink writers to propagate watermarks to > external storage systems, as described in > [FLIP-167|https://cwiki.apache.org/confluence/display/FLINK/FLIP-167%3A+Watermarks+for+Sink+API]. > Note that sink functions already obtain the current watermark upon receiving > a record. This issue is about obtaining the watermark as it is received from > upstream (i.e. not dependent on receipt of a record). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-17859) [flink-s3-fs-presto] Upgrade the dependent presto-hive jar to latest version
[ https://issues.apache.org/jira/browse/FLINK-17859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355230#comment-17355230 ] Eron Wright commented on FLINK-17859: - The outstanding issue is to upgrade to the Trino library. > [flink-s3-fs-presto] Upgrade the dependent presto-hive jar to latest version > > > Key: FLINK-17859 > URL: https://issues.apache.org/jira/browse/FLINK-17859 > Project: Flink > Issue Type: New Feature > Components: Connectors / FileSystem >Affects Versions: 1.7.2 >Reporter: sam lin >Priority: Major > Fix For: 1.8.0 > > > The current version of presto-hive is 0.187.[ > |https://github.com/apache/flink/blob/master/flink-filesystems/flink-s3-fs-presto/pom.xml#L36] > > [https://github.com/apache/flink/blob/master/flink-filesystems/flink-s3-fs-presto/pom.xml#L36] > [ > |https://github.com/apache/flink/blob/master/flink-filesystems/flink-s3-fs-presto/pom.xml#L36] > The latest version is 0.234. [https://github.com/prestodb/presto/releases > ] > There are some nice features we want to use after 0.187. One of them is the > CredentialProviderChain support when using AWS S3 client added in this > [pr]([https://github.com/prestodb/presto/pull/13858]) > Do you have any concerns to upgrade the `presto-hive` to the latest version? > Could you please upgrade it in the latest release? Thanks. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-22828) Allow using a custom AWS credentials provider for the Kinesis Connector
[ https://issues.apache.org/jira/browse/FLINK-22828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355229#comment-17355229 ] Eron Wright commented on FLINK-22828: - Note that the Presto library is quite out-of-date, and doesn't use the default credentials chain as does the newer Trino library. [https://github.com/trinodb/trino/blob/0.184/presto-hive/src/main/java/com/facebook/presto/hive/PrestoS3FileSystem.java#L688-L705] [https://github.com/trinodb/trino/blob/357/plugin/trino-hive/src/main/java/io/trino/plugin/hive/s3/TrinoS3FileSystem.java#L921-L948] > Allow using a custom AWS credentials provider for the Kinesis Connector > --- > > Key: FLINK-22828 > URL: https://issues.apache.org/jira/browse/FLINK-22828 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kinesis >Affects Versions: 1.14.0 >Reporter: Arvid Heise >Assignee: Arvid Heise >Priority: Major > Fix For: 1.14.0 > > > Users currently have to use the credential providers that are pre-configured > in Kinesis connector. > For advanced users, it would be nice to be able to configure it similar to > Presto: > https://prestodb.io/docs/0.187/connector/hive.html#custom-s3-credentials-provider -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-22828) Allow using a custom AWS credentials provider for the Kinesis Connector
[ https://issues.apache.org/jira/browse/FLINK-22828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355224#comment-17355224 ] Eron Wright commented on FLINK-22828: - Linking to a similar issue. > Allow using a custom AWS credentials provider for the Kinesis Connector > --- > > Key: FLINK-22828 > URL: https://issues.apache.org/jira/browse/FLINK-22828 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kinesis >Affects Versions: 1.14.0 >Reporter: Arvid Heise >Assignee: Arvid Heise >Priority: Major > Fix For: 1.14.0 > > > Users currently have to use the credential providers that are pre-configured > in Kinesis connector. > For advanced users, it would be nice to be able to configure it similar to > Presto: > https://prestodb.io/docs/0.187/connector/hive.html#custom-s3-credentials-provider -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22700) [FLIP-167] Propagate watermarks to Sink API
[ https://issues.apache.org/jira/browse/FLINK-22700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eron Wright updated FLINK-22700: Description: Make it possible for sink functions / sink writers to propagate watermarks to external storage systems, as described in [FLIP-167|https://cwiki.apache.org/confluence/display/FLINK/FLIP-167%3A+Watermarks+for+Sink+API]. Note that sink functions already obtain the current watermark upon receiving a record. This issue is about obtaining the watermark as it is received from upstream (i.e. not dependent on receipt of a record). was: Make it possible for sink functions / sink writers to propagate watermarks to external storage systems. Note that sink functions obtain the current watermark upon receiving a record. This issue is about obtaining the watermark as it is received from upstream (i.e. not dependent on receipt of a record). > [FLIP-167] Propagate watermarks to Sink API > --- > > Key: FLINK-22700 > URL: https://issues.apache.org/jira/browse/FLINK-22700 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Reporter: Eron Wright >Assignee: Eron Wright >Priority: Major > Labels: pull-request-available > > Make it possible for sink functions / sink writers to propagate watermarks to > external storage systems, as described in > [FLIP-167|https://cwiki.apache.org/confluence/display/FLINK/FLIP-167%3A+Watermarks+for+Sink+API]. > Note that sink functions already obtain the current watermark upon receiving > a record. This issue is about obtaining the watermark as it is received from > upstream (i.e. not dependent on receipt of a record). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-22700) [FLIP-167] Propagate watermarks to Sink API
[ https://issues.apache.org/jira/browse/FLINK-22700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17347773#comment-17347773 ] Eron Wright edited comment on FLINK-22700 at 5/21/21, 4:15 PM: --- Update: am working on a FLIP for the API change. Update 2: see FLIP-167 was (Author: eronwright): Update: am working on a PIP for the API change. > [FLIP-167] Propagate watermarks to Sink API > --- > > Key: FLINK-22700 > URL: https://issues.apache.org/jira/browse/FLINK-22700 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Reporter: Eron Wright >Assignee: Eron Wright >Priority: Major > Labels: pull-request-available > > Make it possible for sink functions / sink writers to propagate watermarks to > external storage systems, as described in > [FLIP-167|https://cwiki.apache.org/confluence/display/FLINK/FLIP-167%3A+Watermarks+for+Sink+API]. > Note that sink functions already obtain the current watermark upon receiving > a record. This issue is about obtaining the watermark as it is received from > upstream (i.e. not dependent on receipt of a record). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22700) [FLIP-167] Propagate watermarks to Sink API
[ https://issues.apache.org/jira/browse/FLINK-22700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eron Wright updated FLINK-22700: Summary: [FLIP-167] Propagate watermarks to Sink API (was: Propagate watermarks to Sink API) > [FLIP-167] Propagate watermarks to Sink API > --- > > Key: FLINK-22700 > URL: https://issues.apache.org/jira/browse/FLINK-22700 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Reporter: Eron Wright >Assignee: Eron Wright >Priority: Major > Labels: pull-request-available > > Make it possible for sink functions / sink writers to propagate watermarks to > external storage systems. > Note that sink functions obtain the current watermark upon receiving a > record. This issue is about obtaining the watermark as it is received from > upstream (i.e. not dependent on receipt of a record). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22700) Propagate watermarks to Sink API
[ https://issues.apache.org/jira/browse/FLINK-22700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eron Wright updated FLINK-22700: Description: Make it possible for sink functions / sink writers to propagate watermarks to external storage systems. Note that sink functions obtain the current watermark upon receiving a record. This issue is about obtaining the watermark as it is received from upstream (i.e. not dependent on receipt of a record). was: Make it possible for sink functions / sink writers to propagate watermarks to external storage systems. Note that sink functions obtain the current watermark upon receiving a record. This issue is about obtaining the watermark as it is received from upstream (i.e. not dependent on receipt of an event). > Propagate watermarks to Sink API > > > Key: FLINK-22700 > URL: https://issues.apache.org/jira/browse/FLINK-22700 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Reporter: Eron Wright >Assignee: Eron Wright >Priority: Major > Labels: pull-request-available > > Make it possible for sink functions / sink writers to propagate watermarks to > external storage systems. > Note that sink functions obtain the current watermark upon receiving a > record. This issue is about obtaining the watermark as it is received from > upstream (i.e. not dependent on receipt of a record). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-22700) Propagate watermarks to Sink API
[ https://issues.apache.org/jira/browse/FLINK-22700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17347773#comment-17347773 ] Eron Wright commented on FLINK-22700: - Update: am working on a PIP for the API change. > Propagate watermarks to Sink API > > > Key: FLINK-22700 > URL: https://issues.apache.org/jira/browse/FLINK-22700 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Reporter: Eron Wright >Assignee: Eron Wright >Priority: Major > Labels: pull-request-available > > Make it possible for sink functions / sink writers to propagate watermarks to > external storage systems. > Note that sink functions obtain the current watermark upon receiving a > record. This issue is about obtaining the watermark as it is received from > upstream (i.e. not dependent on receipt of an event). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22700) Propagate watermarks to Sink API
Eron Wright created FLINK-22700: --- Summary: Propagate watermarks to Sink API Key: FLINK-22700 URL: https://issues.apache.org/jira/browse/FLINK-22700 Project: Flink Issue Type: Improvement Components: API / DataStream Reporter: Eron Wright Assignee: Eron Wright Make it possible for sink functions / sink writers to propagate watermarks to external storage systems. Note that sink functions obtain the current watermark upon receiving a record. This issue is about obtaining the watermark as it is received from upstream (i.e. not dependent on receipt of an event). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-21307) Revisit activation model of FlinkSecurityManager
[ https://issues.apache.org/jira/browse/FLINK-21307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17279900#comment-17279900 ] Eron Wright edited comment on FLINK-21307 at 2/5/21, 7:10 PM: -- Is it a reasonable expectation that enforcing a security manager for user code would significantly improve the protection afforded to job-level secrets? For example, imagine a connector has configuration property containing a credential; in a session cluster, one job could theoretically access the configuration properties of another job. Imposing a security manager seems like a good step towards preventing that. Also, any special considerations for plugins? Am mentioning this to voice support for applying a security manager in a comprehensive way, and with the hope that the security policy would improve job isolation. was (Author: eronwright): Is it a reasonable expectation that enforcing a security manager for user code would significantly improve the protection afforded to job-level secrets? For example, imagine a connector has configuration property containing a credential; in a session cluster, one job could theoretically access the configuration properties of another job. Imposing a security manager seems like a good step towards preventing that. Also, any special considerations for plugins? > Revisit activation model of FlinkSecurityManager > > > Key: FLINK-21307 > URL: https://issues.apache.org/jira/browse/FLINK-21307 > Project: Flink > Issue Type: Bug > Components: Runtime / Task >Affects Versions: 1.13.0 >Reporter: Robert Metzger >Priority: Critical > Fix For: 1.13.0 > > > In FLINK-15156, we introduced a feature that allows users to log or > completely disable calls to System.exit(). This feature is enabled for > certain threads / code sections intended to execute user-code. > The activation of the security manager (for monitoring user calls to > System.exit() is currently not well-defined, and only implemented on a > best-effort basis. > This ticket is to revisit the activation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-21307) Revisit activation model of FlinkSecurityManager
[ https://issues.apache.org/jira/browse/FLINK-21307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17279900#comment-17279900 ] Eron Wright edited comment on FLINK-21307 at 2/5/21, 6:54 PM: -- Is it a reasonable expectation that enforcing a security manager for user code would significantly improve the protection afforded to job-level secrets? For example, imagine a connector has configuration property containing a credential; in a session cluster, one job could theoretically access the configuration properties of another job. Imposing a security manager seems like a good step towards preventing that. Also, any special considerations for plugins? was (Author: eronwright): Is it a reasonable expectation that enforcing a security manager for user code would significantly improve the protection afforded to job-level secrets? For example, imagine a connector has configuration property containing a credential; in a session cluster, one job could theoretically access the configuration properties of another job. Imposing a security manager seems like a good step towards preventing that. > Revisit activation model of FlinkSecurityManager > > > Key: FLINK-21307 > URL: https://issues.apache.org/jira/browse/FLINK-21307 > Project: Flink > Issue Type: Bug > Components: Runtime / Task >Affects Versions: 1.13.0 >Reporter: Robert Metzger >Priority: Critical > Fix For: 1.13.0 > > > In FLINK-15156, we introduced a feature that allows users to log or > completely disable calls to System.exit(). This feature is enabled for > certain threads / code sections intended to execute user-code. > The activation of the security manager (for monitoring user calls to > System.exit() is currently not well-defined, and only implemented on a > best-effort basis. > This ticket is to revisit the activation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21307) Revisit activation model of FlinkSecurityManager
[ https://issues.apache.org/jira/browse/FLINK-21307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17279900#comment-17279900 ] Eron Wright commented on FLINK-21307: - Is it a reasonable expectation that enforcing a security manager for user code would significantly improve the protection afforded to job-level secrets? For example, imagine a connector has configuration property containing a credential; in a session cluster, one job could theoretically access the configuration properties of another job. Imposing a security manager seems like a good step towards preventing that. > Revisit activation model of FlinkSecurityManager > > > Key: FLINK-21307 > URL: https://issues.apache.org/jira/browse/FLINK-21307 > Project: Flink > Issue Type: Bug > Components: Runtime / Task >Affects Versions: 1.13.0 >Reporter: Robert Metzger >Priority: Critical > Fix For: 1.13.0 > > > In FLINK-15156, we introduced a feature that allows users to log or > completely disable calls to System.exit(). This feature is enabled for > certain threads / code sections intended to execute user-code. > The activation of the security manager (for monitoring user calls to > System.exit() is currently not well-defined, and only implemented on a > best-effort basis. > This ticket is to revisit the activation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-11241) Enhance TableEnvironment to connect to a catalog via a descriptor
[
https://issues.apache.org/jira/browse/FLINK-11241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eron Wright reassigned FLINK-11241:
Assignee: (was: Eron Wright )
> Enhance TableEnvironment to connect to a catalog via a descriptor
> -
>
> Key: FLINK-11241
> URL: https://issues.apache.org/jira/browse/FLINK-11241
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Eron Wright
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Given FLINK-11240, extend {{TableEnvironment}} to connect to an external
> catalog via an {{ExternalCatalogDescriptor}}. Consider extending the
> existing {{connect()}} method.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Assigned] (FLINK-11238) Enhance SQL-Client to recursively list tables
[
https://issues.apache.org/jira/browse/FLINK-11238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eron Wright reassigned FLINK-11238:
Assignee: (was: Eron Wright )
> Enhance SQL-Client to recursively list tables
> -
>
> Key: FLINK-11238
> URL: https://issues.apache.org/jira/browse/FLINK-11238
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Client
>Reporter: Eron Wright
>Priority: Major
>
> The SQL Client provides a {{SHOW TABLES}} command. Tables that are added
> via an external catalog should be listed (presently, only the root schema is
> listed).
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (FLINK-10725) Support for Java 11 (LTS)
[ https://issues.apache.org/jira/browse/FLINK-10725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16841579#comment-16841579 ] Eron Wright commented on FLINK-10725: -- Want to mention an interesting detail, JDK10 introduced the `UseContainerSupport` flag which is said to improve on the experimental containerization flags that were in JDK9. Considering that Flink makes significant use of off-heap memory (e.g. rocksdb) and is typically deployed into a container environment, full support for JDK10/11 may significantly improve usability and stability. > Support for Java 11 (LTS) > - > > Key: FLINK-10725 > URL: https://issues.apache.org/jira/browse/FLINK-10725 > Project: Flink > Issue Type: Improvement > Components: Build System >Affects Versions: 1.8.0, 2.0.0 >Reporter: Sina Madani >Priority: Major > Original Estimate: 2h > Remaining Estimate: 2h > > Java 8 is over 5 years old and will be end of life in 2019/2020. Java 11, the > latest long-term support release, became GA in September 2018. Given that > FLINK-8033 still hasn't been resolved and that Java 9 was end of life > (discontinued / no longer publically available or supported) since March > 2018, it doesn't make sense to continue trying to add Java 9 support when > both Java 9 and Java 10 are end-of-life. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (FLINK-5182) Implement SSL file-shipping
[ https://issues.apache.org/jira/browse/FLINK-5182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eron Wright resolved FLINK-5182. - Resolution: Won't Fix Closing as unimportant. > Implement SSL file-shipping > --- > > Key: FLINK-5182 > URL: https://issues.apache.org/jira/browse/FLINK-5182 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Eron Wright >Priority: Major > > The current handling of keystore and truststore is, the config entry is > treated as a local file path always, and the files aren't shipped > automatically.The behavior is problematic in YARN/Mesos deployments, > where such an assumption doesn't always hold. > Change the behavior to automatically ship the files and update the config > automatically. That behavior is consistent with how keytabs are handled. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (FLINK-3931) Implement Transport Encryption (SSL/TLS)
[ https://issues.apache.org/jira/browse/FLINK-3931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eron Wright resolved FLINK-3931. - Resolution: Fixed > Implement Transport Encryption (SSL/TLS) > > > Key: FLINK-3931 > URL: https://issues.apache.org/jira/browse/FLINK-3931 > Project: Flink > Issue Type: New Feature > Components: Runtime / Coordination >Reporter: Eron Wright >Assignee: Eron Wright >Priority: Major > Labels: security > Original Estimate: 1,008h > Remaining Estimate: 1,008h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > To assure privacy and data integrity between Flink components, enable TLS for > all communication channels. As described in the design doc: > - Accept a configured certificate or generate a certificate. > - Enable Akka SSL > - Implement Data Transfer SSL > - Implement Blob Server SSL > - Implement Web UI HTTPS -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (FLINK-5029) Implement KvState SSL
[ https://issues.apache.org/jira/browse/FLINK-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eron Wright resolved FLINK-5029. - Resolution: Won't Fix > Implement KvState SSL > - > > Key: FLINK-5029 > URL: https://issues.apache.org/jira/browse/FLINK-5029 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Eron Wright >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > The KVState endpoint is new to 1.2 and should support SSL as the others do. > Note that, with FLINK-4898, the SSL support code is decoupled from the > NettyClient/NettyServer, so can be used by the KvState code by simply > installing the `SSLProtocolHandler`. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-5029) Implement KvState SSL
[ https://issues.apache.org/jira/browse/FLINK-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796554#comment-16796554 ] Eron Wright commented on FLINK-5029: - Any objection to resolving this as won't-fix? Seems to me that queryable state is an experimental feature that isn't worth investing in. > Implement KvState SSL > - > > Key: FLINK-5029 > URL: https://issues.apache.org/jira/browse/FLINK-5029 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Eron Wright >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > The KVState endpoint is new to 1.2 and should support SSL as the others do. > Note that, with FLINK-4898, the SSL support code is decoupled from the > NettyClient/NettyServer, so can be used by the KvState code by simply > installing the `SSLProtocolHandler`. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-5029) Implement KvState SSL
[ https://issues.apache.org/jira/browse/FLINK-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eron Wright updated FLINK-5029: Labels: (was: pull-request-available) > Implement KvState SSL > - > > Key: FLINK-5029 > URL: https://issues.apache.org/jira/browse/FLINK-5029 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Eron Wright >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > The KVState endpoint is new to 1.2 and should support SSL as the others do. > Note that, with FLINK-4898, the SSL support code is decoupled from the > NettyClient/NettyServer, so can be used by the KvState code by simply > installing the `SSLProtocolHandler`. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (FLINK-5029) Implement KvState SSL
[ https://issues.apache.org/jira/browse/FLINK-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eron Wright reassigned FLINK-5029: --- Assignee: (was: Eron Wright ) > Implement KvState SSL > - > > Key: FLINK-5029 > URL: https://issues.apache.org/jira/browse/FLINK-5029 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Eron Wright >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The KVState endpoint is new to 1.2 and should support SSL as the others do. > Note that, with FLINK-4898, the SSL support code is decoupled from the > NettyClient/NettyServer, so can be used by the KvState code by simply > installing the `SSLProtocolHandler`. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10744) Integrate Flink with Hive metastore
[ https://issues.apache.org/jira/browse/FLINK-10744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756766#comment-16756766 ] Eron Wright commented on FLINK-10744: -- Update on my end, all issues assigned to me have PRs open; awaiting review. > Integrate Flink with Hive metastore > > > Key: FLINK-10744 > URL: https://issues.apache.org/jira/browse/FLINK-10744 > Project: Flink > Issue Type: Improvement > Components: Table API SQL >Affects Versions: 1.6.2 >Reporter: Xuefu Zhang >Assignee: Xuefu Zhang >Priority: Major > > This JIRA keeps track of the effort of FLINK-10556 on Hive metastore > integration. It mainly covers two aspects: > # Register Hive metastore as an external catalog of Flink, such that Hive > table metadata can be accessed directly. > # Store Flink metadata (tables, views, UDFs, etc) in a catalog that utilizes > Hive as the schema registry. > Discussions and resulting design doc will be shared here, but detailed work > items will be tracked by sub-tasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (FLINK-11245) Update documentation for catalogs in SQL-Client
[ https://issues.apache.org/jira/browse/FLINK-11245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eron Wright reassigned FLINK-11245: Assignee: (was: Eron Wright ) > Update documentation for catalogs in SQL-Client > --- > > Key: FLINK-11245 > URL: https://issues.apache.org/jira/browse/FLINK-11245 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Table API SQL >Reporter: Eron Wright >Priority: Major > > Add to the SQL-Client documentation, information about using catalogs in an > environment file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-11247) Fix DESCRIBE command to support catalog tables
Eron Wright created FLINK-11247:
Summary: Fix DESCRIBE command to support catalog tables
Key: FLINK-11247
URL: https://issues.apache.org/jira/browse/FLINK-11247
Project: Flink
Issue Type: Sub-task
Reporter: Eron Wright
When the {{DESCRIBE}} command is applied to a catalog table, it fails with an
error:
{code}
Flink SQL> DESCRIBE nyc.TaxiRides;
[ERROR] Could not execute SQL statement. Reason:
org.apache.flink.table.api.TableException: Table 'nyc.TaxiRides' was not found.
{code}
The reason appears to be that {{LocalExecutor}} calls
{{TableEnvironment::scan}} with the fully-qualified table name as a parameter
(e.g. {{scan("nyc.TaxiRides")}}) rather than with an array of components (e.g.
{{scan("nyc", "TaxiRides")}}).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-11245) Update documentation for catalogs in SQL-Client
Eron Wright created FLINK-11245: Summary: Update documentation for catalogs in SQL-Client Key: FLINK-11245 URL: https://issues.apache.org/jira/browse/FLINK-11245 Project: Flink Issue Type: Sub-task Components: Documentation, Table API SQL Reporter: Eron Wright Assignee: Eron Wright Add to the SQL-Client documentation, information about using catalogs in an environment file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-11238) Enhance SQL-Client to recursively list tables
[
https://issues.apache.org/jira/browse/FLINK-11238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eron Wright updated FLINK-11238:
-
Description: The SQL Client provides a {{SHOW TABLES}} command. Tables
that are added via an external catalog should be listed (presently, only the
root schema is listed). (was: The SQL Client provides a "SHOW TABLES" command.
Tables that are added via an external catalog should be listed (presently,
only the root schema is listed).)
> Enhance SQL-Client to recursively list tables
> -
>
> Key: FLINK-11238
> URL: https://issues.apache.org/jira/browse/FLINK-11238
> Project: Flink
> Issue Type: Sub-task
> Components: Table API SQL
>Reporter: Eron Wright
>Assignee: Eron Wright
>Priority: Major
> Fix For: 1.8.0
>
>
> The SQL Client provides a {{SHOW TABLES}} command. Tables that are added
> via an external catalog should be listed (presently, only the root schema is
> listed).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (FLINK-10755) Port external catalogs in Table API extension points to flink-table-common
[
https://issues.apache.org/jira/browse/FLINK-10755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16731465#comment-16731465
]
Eron Wright edited comment on FLINK-10755 at 12/31/18 10:49 PM:
-
Please also move {{ExternalCatalogDescriptor}},
{{ExternalCatalogDescriptorValidator}}, and {{ExternalCatalogDescriptorTest}},
assuming FLINK-11240 is merged first.
was (Author: eronwright):
Please also move {{ExternalCatalogDescriptor}},
{{ExternalCatalogDescriptorValidator}}, and {{ExternalCatalogDescriptorTest}}.
> Port external catalogs in Table API extension points to flink-table-common
> --
>
> Key: FLINK-10755
> URL: https://issues.apache.org/jira/browse/FLINK-10755
> Project: Flink
> Issue Type: Sub-task
> Components: Table API SQL
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Major
> Fix For: 1.8.0
>
>
> After FLINK-10687 and FLINK-10688 have been resolved, we should also port the
> remaining extension points of the Table API to flink-table-common. This
> includes interfaces for UDFs and the external catalog interface.
> This ticket is for porting external catalogs. This jira depends on
> FLINK-16088.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-11241) Enhance TableEnvironment to connect to a catalog via a descriptor
Eron Wright created FLINK-11241:
Summary: Enhance TableEnvironment to connect to a catalog via a
descriptor
Key: FLINK-11241
URL: https://issues.apache.org/jira/browse/FLINK-11241
Project: Flink
Issue Type: Sub-task
Components: Table API SQL
Reporter: Eron Wright
Assignee: Eron Wright
Fix For: 1.8.0
Given FLINK-11240, extend {{TableEnvironment}} to connect to an external
catalog via an {{ExternalCatalogDescriptor}}. Consider extending the existing
{{connect()}} method.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9172) Support external catalogs in SQL-Client
[
https://issues.apache.org/jira/browse/FLINK-9172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16731469#comment-16731469
]
Eron Wright commented on FLINK-9172:
-
Update: opened another subtask (FLINK-11240) to implement
{{ExternalCatalogDescriptor}} and {{ExternalCatalogFactory}}. FLINK-9172 (this
issue) will focus on SQL client integration.
> Support external catalogs in SQL-Client
> ---
>
> Key: FLINK-9172
> URL: https://issues.apache.org/jira/browse/FLINK-9172
> Project: Flink
> Issue Type: Sub-task
> Components: SQL Client
>Reporter: Rong Rong
>Assignee: Eron Wright
>Priority: Major
>
> It doesn't seem that the configuration (YAML) file allows specifications of
> external catalogs currently. The request here is to add support for external
> catalog specifications in YAML file. User should also be able to specify one
> catalog is the default.
> It will be great to have SQL-Client to support some external catalogs
> out-of-the-box for SQL users to configure and utilize easily. I am currently
> think of having an external catalog factory that spins up both streaming and
> batch external catalog table sources and sinks. This could greatly unify and
> provide easy access for SQL users.
> The catalog-related configurations then need to be processed and passed to
> TableEnvironment accordingly by calling relevant APIs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-9172) Support external catalogs in SQL-Client
[ https://issues.apache.org/jira/browse/FLINK-9172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eron Wright updated FLINK-9172: Summary: Support external catalogs in SQL-Client (was: Support external catalog factory that comes default with SQL-Client) > Support external catalogs in SQL-Client > --- > > Key: FLINK-9172 > URL: https://issues.apache.org/jira/browse/FLINK-9172 > Project: Flink > Issue Type: Sub-task > Components: SQL Client >Reporter: Rong Rong >Assignee: Eron Wright >Priority: Major > > It doesn't seem that the configuration (YAML) file allows specifications of > external catalogs currently. The request here is to add support for external > catalog specifications in YAML file. User should also be able to specify one > catalog is the default. > It will be great to have SQL-Client to support some external catalogs > out-of-the-box for SQL users to configure and utilize easily. I am currently > think of having an external catalog factory that spins up both streaming and > batch external catalog table sources and sinks. This could greatly unify and > provide easy access for SQL users. > The catalog-related configurations then need to be processed and passed to > TableEnvironment accordingly by calling relevant APIs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-11240) Implement external catalog factory and descriptor
Eron Wright created FLINK-11240: Summary: Implement external catalog factory and descriptor Key: FLINK-11240 URL: https://issues.apache.org/jira/browse/FLINK-11240 Project: Flink Issue Type: Sub-task Components: Table API SQL Reporter: Eron Wright Assignee: Eron Wright Fix For: 1.8.0 Similar to the efforts done in FLINK-8240 and FLINK-8866, implement descriptor-based loading of external catalogs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10755) Port external catalogs in Table API extension points to flink-table-common
[
https://issues.apache.org/jira/browse/FLINK-10755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16731465#comment-16731465
]
Eron Wright commented on FLINK-10755:
--
Please also move {{ExternalCatalogDescriptor}},
{{ExternalCatalogDescriptorValidator}}, and {{ExternalCatalogDescriptorTest}}.
> Port external catalogs in Table API extension points to flink-table-common
> --
>
> Key: FLINK-10755
> URL: https://issues.apache.org/jira/browse/FLINK-10755
> Project: Flink
> Issue Type: Sub-task
> Components: Table API SQL
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Major
> Fix For: 1.8.0
>
>
> After FLINK-10687 and FLINK-10688 have been resolved, we should also port the
> remaining extension points of the Table API to flink-table-common. This
> includes interfaces for UDFs and the external catalog interface.
> This ticket is for porting external catalogs. This jira depends on
> FLINK-16088.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10618) Introduce catalog for Flink tables
[
https://issues.apache.org/jira/browse/FLINK-10618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16731455#comment-16731455
]
Eron Wright commented on FLINK-10618:
--
Sorry, I mistakenly edited this issue.
> Introduce catalog for Flink tables
> --
>
> Key: FLINK-10618
> URL: https://issues.apache.org/jira/browse/FLINK-10618
> Project: Flink
> Issue Type: Sub-task
> Components: SQL Client
>Affects Versions: 1.6.1
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
>Priority: Major
> Fix For: 1.8.0
>
>
> This JIRA covers the 2nd aspect of Flink-Hive metastore integration.
> Besides meta objects such as tables that may come from an
> {{ExternalCatalog}}, Flink also deals with tables/views/functions that are
> created on the fly (in memory), or specified in a configuration file. Those
> objects don't belong to any {{ExternalCatalog}}, yet Flink either stores them
> in memory, which are non-persistent, or recreates them from a file, which is
> a big pain for the user. Those objects are only known to Flink but Flink has
> a poor management for them.
> Since they are typical objects in a database catalog, it's natural to have a
> catalog that manages those objects. The interface will be similar to
> {{ExternalCatalog}}, which contains meta objects that are not managed by
> Flink. There are several possible implementations of the Flink internal
> catalog interface: memory, file, external registry (such as confluent schema
> registry or Hive metastore), and relational database, etc.
> The initial functionality as well as the catalog hierarchy could be very
> simple. The basic functionality of the catalog will be mostly create, alter,
> and drop tables, views, functions, etc. Obviously, this can evolve over the
> time.
> We plan to provide implementations: in-memory and in Hive metastore.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (FLINK-10618) Introduce catalog for Flink tables
[
https://issues.apache.org/jira/browse/FLINK-10618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eron Wright reassigned FLINK-10618:
Assignee: Xuefu Zhang (was: Eron Wright )
> Introduce catalog for Flink tables
> --
>
> Key: FLINK-10618
> URL: https://issues.apache.org/jira/browse/FLINK-10618
> Project: Flink
> Issue Type: Sub-task
> Components: SQL Client
>Affects Versions: 1.6.1
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
>Priority: Major
> Fix For: 1.8.0
>
>
> This JIRA covers the 2nd aspect of Flink-Hive metastore integration.
> Besides meta objects such as tables that may come from an
> {{ExternalCatalog}}, Flink also deals with tables/views/functions that are
> created on the fly (in memory), or specified in a configuration file. Those
> objects don't belong to any {{ExternalCatalog}}, yet Flink either stores them
> in memory, which are non-persistent, or recreates them from a file, which is
> a big pain for the user. Those objects are only known to Flink but Flink has
> a poor management for them.
> Since they are typical objects in a database catalog, it's natural to have a
> catalog that manages those objects. The interface will be similar to
> {{ExternalCatalog}}, which contains meta objects that are not managed by
> Flink. There are several possible implementations of the Flink internal
> catalog interface: memory, file, external registry (such as confluent schema
> registry or Hive metastore), and relational database, etc.
> The initial functionality as well as the catalog hierarchy could be very
> simple. The basic functionality of the catalog will be mostly create, alter,
> and drop tables, views, functions, etc. Obviously, this can evolve over the
> time.
> We plan to provide implementations: in-memory and in Hive metastore.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-11239) Enhance SQL-Client to recursively list UDFs
Eron Wright created FLINK-11239: Summary: Enhance SQL-Client to recursively list UDFs Key: FLINK-11239 URL: https://issues.apache.org/jira/browse/FLINK-11239 Project: Flink Issue Type: Sub-task Components: Table API SQL Reporter: Eron Wright Fix For: 1.8.0 The SQL Client provides a "SHOW FUNCTIONS" to show all registered functions. Enhance it to show functions produced by an external catalog. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-11237) Enhance LocalExecutor to wrap TableEnvironment w/ user classloader
[
https://issues.apache.org/jira/browse/FLINK-11237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eron Wright updated FLINK-11237:
-
Component/s: Table API & SQL
> Enhance LocalExecutor to wrap TableEnvironment w/ user classloader
> --
>
> Key: FLINK-11237
> URL: https://issues.apache.org/jira/browse/FLINK-11237
> Project: Flink
> Issue Type: Sub-task
> Components: Table API SQL
>Reporter: Eron Wright
>Assignee: Eron Wright
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.8.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The SQL Client's {{LocalExecutor}} calls into the table environment to
> execute queries, explain statements, and much more. Any call that involves
> resolving a descriptor to a factory implementation must be wrapped in the
> user classloader. Some of the calls already are wrapped (for resolving
> UDFs). With new functionality coming for resolving external catalogs with a
> descriptor, other call sites must be wrapped.
> Note that the {{TableEnvironment}} resolves the tables defined within an
> external catalog lazily (at query time).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-11238) Enhance SQL-Client to recursively list tables
Eron Wright created FLINK-11238: Summary: Enhance SQL-Client to recursively list tables Key: FLINK-11238 URL: https://issues.apache.org/jira/browse/FLINK-11238 Project: Flink Issue Type: Sub-task Components: Table API SQL Reporter: Eron Wright Assignee: Eron Wright Fix For: 1.8.0 The SQL Client provides a "SHOW TABLES" command. Tables that are added via an external catalog should be listed (presently, only the root schema is listed). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-11237) Enhance LocalExecutor to wrap TableEnvironment w/ user classloader
[
https://issues.apache.org/jira/browse/FLINK-11237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eron Wright updated FLINK-11237:
-
Fix Version/s: 1.8.0
> Enhance LocalExecutor to wrap TableEnvironment w/ user classloader
> --
>
> Key: FLINK-11237
> URL: https://issues.apache.org/jira/browse/FLINK-11237
> Project: Flink
> Issue Type: Sub-task
>Reporter: Eron Wright
>Assignee: Eron Wright
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.8.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The SQL Client's {{LocalExecutor}} calls into the table environment to
> execute queries, explain statements, and much more. Any call that involves
> resolving a descriptor to a factory implementation must be wrapped in the
> user classloader. Some of the calls already are wrapped (for resolving
> UDFs). With new functionality coming for resolving external catalogs with a
> descriptor, other call sites must be wrapped.
> Note that the {{TableEnvironment}} resolves the tables defined within an
> external catalog lazily (at query time).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-11237) Enhance LocalExecutor to wrap TableEnvironment w/ user classloader
Eron Wright created FLINK-11237:
Summary: Enhance LocalExecutor to wrap TableEnvironment w/ user
classloader
Key: FLINK-11237
URL: https://issues.apache.org/jira/browse/FLINK-11237
Project: Flink
Issue Type: Sub-task
Reporter: Eron Wright
Assignee: Eron Wright
The SQL Client's {{LocalExecutor}} calls into the table environment to execute
queries, explain statements, and much more. Any call that involves resolving
a descriptor to a factory implementation must be wrapped in the user
classloader. Some of the calls already are wrapped (for resolving UDFs).
With new functionality coming for resolving external catalogs with a
descriptor, other call sites must be wrapped.
Note that the {{TableEnvironment}} resolves the tables defined within an
external catalog lazily (at query time).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-11234) ExternalTableCatalogBuilder unable to build a batch-only table
[
https://issues.apache.org/jira/browse/FLINK-11234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eron Wright updated FLINK-11234:
-
Fix Version/s: 1.8.0
> ExternalTableCatalogBuilder unable to build a batch-only table
> --
>
> Key: FLINK-11234
> URL: https://issues.apache.org/jira/browse/FLINK-11234
> Project: Flink
> Issue Type: Bug
> Components: Table API SQL, Tests
>Reporter: Eron Wright
>Assignee: Eron Wright
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.8.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The {{ExternalTableCatalogBuilder::supportsBatch}} method should set
> {{isBatch}} to {{true}} and {{isStreaming}} to {{false}}, but the logic is
> presently inverted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-11234) ExternalTableCatalogBuilder unable to build a batch-only table
[
https://issues.apache.org/jira/browse/FLINK-11234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eron Wright updated FLINK-11234:
-
Description: The {{ExternalTableCatalogBuilder::supportsBatch}} method
should set {{isBatch}} to {{true}} and {{isStreaming}} to {{false}}, but the
logic is presently inverted. (was: The
`ExternalTableCatalogBuilder::supportsBatch` method should set `isBatch` to
`true and `isStreaming` to `false`, but the logic is presently inverted.)
> ExternalTableCatalogBuilder unable to build a batch-only table
> --
>
> Key: FLINK-11234
> URL: https://issues.apache.org/jira/browse/FLINK-11234
> Project: Flink
> Issue Type: Bug
> Components: Table API SQL, Tests
>Reporter: Eron Wright
>Assignee: Eron Wright
>Priority: Minor
>
> The {{ExternalTableCatalogBuilder::supportsBatch}} method should set
> {{isBatch}} to {{true}} and {{isStreaming}} to {{false}}, but the logic is
> presently inverted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-11234) ExternalTableCatalogBuilder unable to build a batch-only table
Eron Wright created FLINK-11234: Summary: ExternalTableCatalogBuilder unable to build a batch-only table Key: FLINK-11234 URL: https://issues.apache.org/jira/browse/FLINK-11234 Project: Flink Issue Type: Bug Components: Table API SQL, Tests Reporter: Eron Wright Assignee: Eron Wright The `ExternalTableCatalogBuilder::supportsBatch` method should set `isBatch` to `true and `isStreaming` to `false`, but the logic is presently inverted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (FLINK-10618) Introduce catalog for Flink tables
[
https://issues.apache.org/jira/browse/FLINK-10618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eron Wright reassigned FLINK-10618:
Assignee: Eron Wright (was: Xuefu Zhang)
> Introduce catalog for Flink tables
> --
>
> Key: FLINK-10618
> URL: https://issues.apache.org/jira/browse/FLINK-10618
> Project: Flink
> Issue Type: Sub-task
> Components: SQL Client
>Affects Versions: 1.6.1
>Reporter: Xuefu Zhang
>Assignee: Eron Wright
>Priority: Major
> Fix For: 1.8.0
>
>
> This JIRA covers the 2nd aspect of Flink-Hive metastore integration.
> Besides meta objects such as tables that may come from an
> {{ExternalCatalog}}, Flink also deals with tables/views/functions that are
> created on the fly (in memory), or specified in a configuration file. Those
> objects don't belong to any {{ExternalCatalog}}, yet Flink either stores them
> in memory, which are non-persistent, or recreates them from a file, which is
> a big pain for the user. Those objects are only known to Flink but Flink has
> a poor management for them.
> Since they are typical objects in a database catalog, it's natural to have a
> catalog that manages those objects. The interface will be similar to
> {{ExternalCatalog}}, which contains meta objects that are not managed by
> Flink. There are several possible implementations of the Flink internal
> catalog interface: memory, file, external registry (such as confluent schema
> registry or Hive metastore), and relational database, etc.
> The initial functionality as well as the catalog hierarchy could be very
> simple. The basic functionality of the catalog will be mostly create, alter,
> and drop tables, views, functions, etc. Obviously, this can evolve over the
> time.
> We plan to provide implementations: in-memory and in Hive metastore.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10618) Introduce catalog for Flink tables
[
https://issues.apache.org/jira/browse/FLINK-10618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eron Wright updated FLINK-10618:
-
Fix Version/s: 1.8.0
> Introduce catalog for Flink tables
> --
>
> Key: FLINK-10618
> URL: https://issues.apache.org/jira/browse/FLINK-10618
> Project: Flink
> Issue Type: Sub-task
> Components: SQL Client
>Affects Versions: 1.6.1
>Reporter: Xuefu Zhang
>Assignee: Eron Wright
>Priority: Major
> Fix For: 1.8.0
>
>
> This JIRA covers the 2nd aspect of Flink-Hive metastore integration.
> Besides meta objects such as tables that may come from an
> {{ExternalCatalog}}, Flink also deals with tables/views/functions that are
> created on the fly (in memory), or specified in a configuration file. Those
> objects don't belong to any {{ExternalCatalog}}, yet Flink either stores them
> in memory, which are non-persistent, or recreates them from a file, which is
> a big pain for the user. Those objects are only known to Flink but Flink has
> a poor management for them.
> Since they are typical objects in a database catalog, it's natural to have a
> catalog that manages those objects. The interface will be similar to
> {{ExternalCatalog}}, which contains meta objects that are not managed by
> Flink. There are several possible implementations of the Flink internal
> catalog interface: memory, file, external registry (such as confluent schema
> registry or Hive metastore), and relational database, etc.
> The initial functionality as well as the catalog hierarchy could be very
> simple. The basic functionality of the catalog will be mostly create, alter,
> and drop tables, views, functions, etc. Obviously, this can evolve over the
> time.
> We plan to provide implementations: in-memory and in Hive metastore.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (FLINK-9172) Support external catalog factory that comes default with SQL-Client
[ https://issues.apache.org/jira/browse/FLINK-9172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eron Wright reassigned FLINK-9172: --- Assignee: Eron Wright (was: Rong Rong) > Support external catalog factory that comes default with SQL-Client > --- > > Key: FLINK-9172 > URL: https://issues.apache.org/jira/browse/FLINK-9172 > Project: Flink > Issue Type: Sub-task >Reporter: Rong Rong >Assignee: Eron Wright >Priority: Major > > It will be great to have SQL-Client to support some external catalogs > out-of-the-box for SQL users to configure and utilize easily. I am currently > think of having an external catalog factory that spins up both streaming and > batch external catalog table sources and sinks. This could greatly unify and > provide easy access for SQL users. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9172) Support external catalog factory that comes default with SQL-Client
[ https://issues.apache.org/jira/browse/FLINK-9172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16604258#comment-16604258 ] Eron Wright commented on FLINK-9172: - [~walterddr] I had some time and put together a PR based on the above discussion. Would you be OK with assigning this to me? Thanks! > Support external catalog factory that comes default with SQL-Client > --- > > Key: FLINK-9172 > URL: https://issues.apache.org/jira/browse/FLINK-9172 > Project: Flink > Issue Type: Sub-task >Reporter: Rong Rong >Assignee: Rong Rong >Priority: Major > > It will be great to have SQL-Client to support some external catalogs > out-of-the-box for SQL users to configure and utilize easily. I am currently > think of having an external catalog factory that spins up both streaming and > batch external catalog table sources and sinks. This could greatly unify and > provide easy access for SQL users. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9172) Support external catalog factory that comes default with SQL-Client
[ https://issues.apache.org/jira/browse/FLINK-9172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596562#comment-16596562 ] Eron Wright commented on FLINK-9172: - [~twalthr] do you agree that we need to extend the environment file to register external catalogs? Assumedly the catalog factory would require some connection info and related parameters. Likewise we may need a set of descriptors to provide a typed API. In other words, the pattern that was established for sources could now be applied to catalogs. Maybe a design doc should be written to allow for further discussion on this. BTW I am very much looking forward to this feature, and would like to help. Here's the scenario that I have in mind: to be able to develop a library for a particular domain, that defines a catalog of tables that relate to that domain. Then, use that library in Java/Scala programs and in the SQL Client. In other words, catalogs provide a unified way to define a set of tables. WDYT? > Support external catalog factory that comes default with SQL-Client > --- > > Key: FLINK-9172 > URL: https://issues.apache.org/jira/browse/FLINK-9172 > Project: Flink > Issue Type: Sub-task >Reporter: Rong Rong >Assignee: Rong Rong >Priority: Major > > It will be great to have SQL-Client to support some external catalogs > out-of-the-box for SQL users to configure and utilize easily. I am currently > think of having an external catalog factory that spins up both streaming and > batch external catalog table sources and sinks. This could greatly unify and > provide easy access for SQL users. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10117) REST API for Queryable State
[ https://issues.apache.org/jira/browse/FLINK-10117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595795#comment-16595795 ] Eron Wright commented on FLINK-10117: -- Note that the SSL feature for Queryable State (FLINK-5029) will treat the existing endpoints as 'internal' communication, which means that the client must possess the internal cluster keypair. That's an obvious usability issue that might best be addressed by this enhancement (FLINK-10117). > REST API for Queryable State > > > Key: FLINK-10117 > URL: https://issues.apache.org/jira/browse/FLINK-10117 > Project: Flink > Issue Type: Improvement > Components: Queryable State, REST >Affects Versions: 1.6.0 >Reporter: Elias Levy >Priority: Major > > At the moment, queryable state requires a JVM based client that can make use > of the Java queryable state client API in flink-queryable-state-client > artifact. In addition, the client requires a state descriptor matching the > queried state, which tightly couples the Flink job and query state clients. > I propose that queryable state become accessible via a REST API. FLINK-7040 > mentions this possibility, but does not specify work towards that goal. > I suggest that to enable queryable state over REST, users define JSON > serializers via the state descriptors. > This would allow queryable state clients to be developed in any language, not > require them to use a Flink client library, and permit them to be loosely > coupled with the job, as they could generically parse the returned JSON. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9172) Support external catalog factory that comes default with SQL-Client
[ https://issues.apache.org/jira/browse/FLINK-9172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16592736#comment-16592736 ] Eron Wright commented on FLINK-9172: - Would you mind clarifying the proposed functionality here? FLIP-24 mentions (as-yet unimplemented) support for external catalogs, is this the tracking issue? > Support external catalog factory that comes default with SQL-Client > --- > > Key: FLINK-9172 > URL: https://issues.apache.org/jira/browse/FLINK-9172 > Project: Flink > Issue Type: Sub-task >Reporter: Rong Rong >Assignee: Rong Rong >Priority: Major > > It will be great to have SQL-Client to support some external catalogs > out-of-the-box for SQL users to configure and utilize easily. I am currently > think of having an external catalog factory that spins up both streaming and > batch external catalog table sources and sinks. This could greatly unify and > provide easy access for SQL users. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (FLINK-5029) Implement KvState SSL
[ https://issues.apache.org/jira/browse/FLINK-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eron Wright reassigned FLINK-5029: --- Assignee: Eron Wright > Implement KvState SSL > - > > Key: FLINK-5029 > URL: https://issues.apache.org/jira/browse/FLINK-5029 > Project: Flink > Issue Type: Sub-task > Components: Security >Reporter: Eron Wright >Assignee: Eron Wright >Priority: Major > > The KVState endpoint is new to 1.2 and should support SSL as the others do. > Note that, with FLINK-4898, the SSL support code is decoupled from the > NettyClient/NettyServer, so can be used by the KvState code by simply > installing the `SSLProtocolHandler`. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eron Wright closed FLINK-3930. --- Resolution: Won't Fix _This issue relates to the obsolete plan to use a 'shared secret' for client authentication. Instead, SSL mutual authentication was implemented._ > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan >Priority: Major > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (FLINK-4919) Add secure cookie support for the cluster deployed in Mesos environment
[ https://issues.apache.org/jira/browse/FLINK-4919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eron Wright closed FLINK-4919. --- Resolution: Won't Fix _This issue relates to the obsolete plan to use a 'shared secret' for client authentication. Instead, SSL mutual authentication was implemented._ > Add secure cookie support for the cluster deployed in Mesos environment > --- > > Key: FLINK-4919 > URL: https://issues.apache.org/jira/browse/FLINK-4919 > Project: Flink > Issue Type: Sub-task > Components: Mesos, Security >Reporter: Vijay Srinivasaraghavan >Assignee: Vijay Srinivasaraghavan >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (FLINK-4637) Address Yarn proxy incompatibility with Flink Web UI when service level authorization is enabled
[ https://issues.apache.org/jira/browse/FLINK-4637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eron Wright closed FLINK-4637. --- Resolution: Won't Fix _This issue relates to the obsolete plan to use a 'shared secret' for client authentication. Instead, SSL mutual authentication was implemented._ > Address Yarn proxy incompatibility with Flink Web UI when service level > authorization is enabled > > > Key: FLINK-4637 > URL: https://issues.apache.org/jira/browse/FLINK-4637 > Project: Flink > Issue Type: Task > Components: Security >Reporter: Vijay Srinivasaraghavan >Assignee: Vijay Srinivasaraghavan >Priority: Major > > When service level authorization is enabled (FLINK-3930), the tracking URL > (Yarn RM Proxy) is not forwarding the secure cookie and as a result, the > Flink Web UI cannot be accessed through the proxy layer. Current workaround > is to use the direct Flink Web URL instead of navigating through proxy. This > JIRA should address the Yarn proxy/secure cookie navigation issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-4919) Add secure cookie support for the cluster deployed in Mesos environment
[ https://issues.apache.org/jira/browse/FLINK-4919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eron Wright updated FLINK-4919: Issue Type: Sub-task (was: Task) Parent: FLINK-3930 > Add secure cookie support for the cluster deployed in Mesos environment > --- > > Key: FLINK-4919 > URL: https://issues.apache.org/jira/browse/FLINK-4919 > Project: Flink > Issue Type: Sub-task > Components: Mesos, Security >Reporter: Vijay Srinivasaraghavan >Assignee: Vijay Srinivasaraghavan >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (FLINK-4635) Implement Data Transfer Authentication using shared secret configuration
[ https://issues.apache.org/jira/browse/FLINK-4635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eron Wright closed FLINK-4635. --- Resolution: Won't Fix _This issue relates to the obsolete plan to use a 'shared secret' for client authentication. Instead, SSL mutual authentication was implemented._ > Implement Data Transfer Authentication using shared secret configuration > > > Key: FLINK-4635 > URL: https://issues.apache.org/jira/browse/FLINK-4635 > Project: Flink > Issue Type: Sub-task > Components: Security >Reporter: Vijay Srinivasaraghavan >Priority: Major > > The data transfer authentication (TM/Netty) requirement was not addressed as > part of FLINK-3930 and this JIRA is created to track the issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (FLINK-4635) Implement Data Transfer Authentication using shared secret configuration
[ https://issues.apache.org/jira/browse/FLINK-4635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eron Wright reassigned FLINK-4635: --- Assignee: (was: Vijay Srinivasaraghavan) > Implement Data Transfer Authentication using shared secret configuration > > > Key: FLINK-4635 > URL: https://issues.apache.org/jira/browse/FLINK-4635 > Project: Flink > Issue Type: Sub-task > Components: Security >Reporter: Vijay Srinivasaraghavan >Priority: Major > > The data transfer authentication (TM/Netty) requirement was not addressed as > part of FLINK-3930 and this JIRA is created to track the issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (FLINK-7753) HandlerUtils should close the channel on error responses
[ https://issues.apache.org/jira/browse/FLINK-7753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eron Wright reassigned FLINK-7753: --- Assignee: (was: Eron Wright ) > HandlerUtils should close the channel on error responses > > > Key: FLINK-7753 > URL: https://issues.apache.org/jira/browse/FLINK-7753 > Project: Flink > Issue Type: Sub-task > Components: Cluster Management, Mesos >Reporter: Eron Wright >Priority: Minor > Labels: pull-request-available > > Unexpected errors in the server pipeline correctly cause a 500 error > response. I suggest that such responses also close the channel rather than > allowing keep-alive. This would be a better security posture too since we > don't know if the pipeline is corrupt following an unexpected error. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (FLINK-7738) Create WebSocket handler (server)
[ https://issues.apache.org/jira/browse/FLINK-7738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eron Wright reassigned FLINK-7738: --- Assignee: (was: Eron Wright ) > Create WebSocket handler (server) > - > > Key: FLINK-7738 > URL: https://issues.apache.org/jira/browse/FLINK-7738 > Project: Flink > Issue Type: Sub-task > Components: Cluster Management, Mesos >Reporter: Eron Wright >Priority: Major > Labels: pull-request-available > > An abstract handler is needed to support websocket communication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (FLINK-4849) trustStorePassword should be checked against null in SSLUtils#createSSLClientContext
[
https://issues.apache.org/jira/browse/FLINK-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eron Wright resolved FLINK-4849.
-
Resolution: Invalid
> trustStorePassword should be checked against null in
> SSLUtils#createSSLClientContext
>
>
> Key: FLINK-4849
> URL: https://issues.apache.org/jira/browse/FLINK-4849
> Project: Flink
> Issue Type: Bug
> Components: Security
>Reporter: Ted Yu
>Priority: Minor
>
> {code}
> String trustStorePassword = sslConfig.getString(
> ConfigConstants.SECURITY_SSL_TRUSTSTORE_PASSWORD,
> null);
> ...
> try {
> trustStoreFile = new FileInputStream(new File(trustStoreFilePath));
> trustStore.load(trustStoreFile, trustStorePassword.toCharArray());
> {code}
> If trustStorePassword is null, the load() call would throw NPE.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8291) For security, Job Manager web UI should be accessed with username/password
[ https://issues.apache.org/jira/browse/FLINK-8291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556192#comment-16556192 ] Eron Wright commented on FLINK-8291: - Note that in FLIP-26 we discuss a few options for user authentication in the web UI. [https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=80453255] > For security, Job Manager web UI should be accessed with username/password > --- > > Key: FLINK-8291 > URL: https://issues.apache.org/jira/browse/FLINK-8291 > Project: Flink > Issue Type: Improvement > Components: Security, Webfrontend >Affects Versions: 1.3.2 >Reporter: Lynch Lee >Priority: Major > > Nowaldays, we submit job from jobm webui without any key for login. > For security, Job Manager web UI should be accessed with username/password > Should we ??? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9611) Allow for user-defined artifacts to be specified as part of a mesos overlay
[ https://issues.apache.org/jira/browse/FLINK-9611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16517528#comment-16517528 ] Eron Wright commented on FLINK-9611: - I really like this idea, except for the proposed name because it has a connotation of overlaying the 'user' (which is sort of what happens with the Kerberos overlay). Maybe 'MesosCustomOverlay'. > Allow for user-defined artifacts to be specified as part of a mesos overlay > --- > > Key: FLINK-9611 > URL: https://issues.apache.org/jira/browse/FLINK-9611 > Project: Flink > Issue Type: Improvement > Components: Configuration, Docker, Mesos >Affects Versions: 1.5.0 >Reporter: Addison Higham >Priority: Major > > NOTE: this assumes mesos, but this improvement could also be useful for > future container deployments. > Currently, when deploying to mesos, the "Overlay" functionality is used to > determine which artifacts are to be downloaded into the container. However, > there isn't a way to plug in your own artifacts to be downloaded into the > container. This can cause problems with certain deployment models. > For example, if you are running flink in docker on mesos, you cannot easily > use a private docker image. Typically with mesos and private docker images, > you specify credentials as a URI to be downloaded into the container that > give permissions to download the private image. Typically, this credentials > expire after a few days, so baking them into a docker host isn't a solution. > It would make sense to add a `MesosUserOverlay` that would simplify take some > new configuration parameters and add any custom artifacts (or possibly also > environment variables?) > Another solution (or longer term solution) might be to allow for dynamically > loading an overlay class for even further customization of the container > specification. > > > > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9612) Add option for minimal artifacts being pulled in Mesos
[ https://issues.apache.org/jira/browse/FLINK-9612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16517525#comment-16517525 ] Eron Wright commented on FLINK-9612: - Yes it makes sense that the overlays would be selective and configurable, and especially true that the Flink binaries aren't needed in most scenarios involving a docker image. Specifically on that, I wonder if the Flink conf directory should be treated differently from the bin/libs (perhaps as a different overlay), since the image might be 'stock'. > Add option for minimal artifacts being pulled in Mesos > -- > > Key: FLINK-9612 > URL: https://issues.apache.org/jira/browse/FLINK-9612 > Project: Flink > Issue Type: Improvement > Components: Configuration, Docker, Mesos >Reporter: Addison Higham >Priority: Major > > NOTE: this assumes mesos, but this improvement could also be useful for > future container deployments. > Currently, in mesos, the FlinkDistributionOverlay copies the entire `conf`, > `bin`, and `lib` folders from the running JobManager/ResourceManager. When > using docker with a pre-installed flink distribution, this is relatively > inefficient as it pulls jars that are already baked into the container image. > A new option that disables pulling most (if not all?) of the > FlinkDistributionOverlay could allow for much faster and more scalable > provisions of TaskManagers. As it currently stands, trying to run a few > hundred TaskManagers is likely to result in poor performance in pulling all > the artifacts from the MesosArtifactServer -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-8622) flink-mesos: High memory usage of scheduler + job manager. GC never kicks in.
[ https://issues.apache.org/jira/browse/FLINK-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eron Wright updated FLINK-8622: Fix Version/s: (was: 1.5.0) > flink-mesos: High memory usage of scheduler + job manager. GC never kicks in. > - > > Key: FLINK-8622 > URL: https://issues.apache.org/jira/browse/FLINK-8622 > Project: Flink > Issue Type: Bug > Components: Distributed Coordination, Mesos, ResourceManager >Affects Versions: 1.4.0, 1.3.2 >Reporter: Bhumika Bayani >Priority: Major > Attachments: flink-mem-usage-graph-for-jira.png > > > We are deploying a 1 job manager + 6 taskmanager flink cluster on mesos. > We have observed that the memory usage for 'jobmanager' is high. In spite of > allocating more and more memory resources to it, it hits the limit within > minutes. > We had started with 1.5 GB RAM and 1 GB heap. Currently we have allocated 4 > GB RAM, 3 GB heap to jobmanager cum scheduler. We tried allocating 8GB RAM > and lesser heap (i.e. same, 3GB) too. In that case also, memory graph was > identical. > As per the graph below, the scheduler almost always runs with maximum memory > resources. > !flink-mem-usage-graph-for-jira.png! > > Throughout the run of the scheduler, we do not see memory usage going down > unless it is killed due to OOM. So inferring, garbage collection is never > happening. > We have tried using both flink versions 1.4 and 1.3 but could see same issue > on both versions. > > Is there any way we can find out where and how memory is being used? > Are there any flink config options for jobmanager or jvm parameters which can > help us restrict the memory usage, force garbage collection, and prevent it > from crash? > Please let us know if there any resource recommendations from Flink for > running Flink on mesos at scale. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (FLINK-5030) Support hostname verification
[ https://issues.apache.org/jira/browse/FLINK-5030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eron Wright resolved FLINK-5030. - Resolution: Won't Fix I think this is obsolete and/or covered by FLINK-9103 > Support hostname verification > - > > Key: FLINK-5030 > URL: https://issues.apache.org/jira/browse/FLINK-5030 > Project: Flink > Issue Type: Sub-task > Components: Security >Reporter: Eron Wright >Assignee: Eron Wright >Priority: Major > > _See [Dangerous Code|http://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf] and > [further > commentary|https://tersesystems.com/2014/03/23/fixing-hostname-verification/] > for useful background._ > When hostname verification is performed, it should use the hostname (not IP > address) to match the certificate. The current code is wrongly using the > address. > In technical terms, ensure that calls to `SSLContext::createSSLEngine` supply > the expected hostname, not host address. > Please audit all SSL setup code as to whether hostname verification is > enabled, and file follow-ups where necessary. For example, Akka 2.4 > supports it but 2.3 doesn't > ([ref|http://doc.akka.io/docs/akka/2.4.4/scala/http/client-side/https-support.html#Hostname_verification]). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9312) Perform mutual authentication during SSL handshakes
[ https://issues.apache.org/jira/browse/FLINK-9312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469744#comment-16469744 ] Eron Wright commented on FLINK-9312: - I believe that this enhancement can be considered as part of [FLIP-26|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=80453255], with the goal of hardening Flink's intra-cluster communication. [~StephanEwen] do you agree? > Perform mutual authentication during SSL handshakes > --- > > Key: FLINK-9312 > URL: https://issues.apache.org/jira/browse/FLINK-9312 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Stephan Ewen >Priority: Major > Fix For: 1.6.0 > > > Currently, the Flink processes encrypted connections via SSL: > - Data exchange TM - TM > - RPC JM - TM > - Blob Service JM - TM > However, the server side always accepts any client to build up the > connection, meaning the connections are not strongly authenticated. > Activating SSL mutual authentication solves that - only processes that have > the same certificate can connect. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-9234) Commons Logging is missing from shaded Flink Table library
Eron Wright created FLINK-9234:
---
Summary: Commons Logging is missing from shaded Flink Table library
Key: FLINK-9234
URL: https://issues.apache.org/jira/browse/FLINK-9234
Project: Flink
Issue Type: Bug
Components: Table API SQL
Affects Versions: 1.4.2
Environment: jdk1.8.0_172
flink 1.4.2
Mac High Sierra
Reporter: Eron Wright
Attachments: repro.scala
The flink-table shaded library seems to be missing some classes from
{{org.apache.commons.logging}} that are required by
{{org.apache.commons.configuration}}. Ran into the problem while using the
external catalog support, on Flink 1.4.2.
See attached a repro, which produces:
{code}
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/flink/table/shaded/org/apache/commons/logging/Log
at
org.apache.flink.table.catalog.ExternalTableSourceUtil$.parseScanPackagesFromConfigFile(ExternalTableSourceUtil.scala:153)
at
org.apache.flink.table.catalog.ExternalTableSourceUtil$.(ExternalTableSourceUtil.scala:55)
at
org.apache.flink.table.catalog.ExternalTableSourceUtil$.(ExternalTableSourceUtil.scala)
at
org.apache.flink.table.catalog.ExternalCatalogSchema.getTable(ExternalCatalogSchema.scala:78)
at
org.apache.calcite.jdbc.SimpleCalciteSchema.getImplicitTable(SimpleCalciteSchema.java:82)
at
org.apache.calcite.jdbc.CalciteSchema.getTable(CalciteSchema.java:256)
at
org.apache.calcite.jdbc.CalciteSchema$SchemaPlusImpl.getTable(CalciteSchema.java:561)
at
org.apache.flink.table.api.TableEnvironment.scanInternal(TableEnvironment.scala:497)
at
org.apache.flink.table.api.TableEnvironment.scan(TableEnvironment.scala:485)
at Repro$.main(repro.scala:17)
at Repro.main(repro.scala)
{code}
Dependencies:
{code}
compile 'org.slf4j:slf4j-api:1.7.25'
compile 'org.slf4j:slf4j-log4j12:1.7.25'
runtime 'log4j:log4j:1.2.17'
compile 'org.apache.flink:flink-scala_2.11:1.4.2'
compile 'org.apache.flink:flink-streaming-scala_2.11:1.4.2'
compile 'org.apache.flink:flink-clients_2.11:1.4.2'
compile 'org.apache.flink:flink-table_2.11:1.4.2'
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (FLINK-4898) Refactor HTTP handlers and Netty server/client
[ https://issues.apache.org/jira/browse/FLINK-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eron Wright closed FLINK-4898. --- Resolution: Fixed Obsolete > Refactor HTTP handlers and Netty server/client > -- > > Key: FLINK-4898 > URL: https://issues.apache.org/jira/browse/FLINK-4898 > Project: Flink > Issue Type: Sub-task > Components: Cluster Management >Reporter: Eron Wright >Priority: Minor > > The dispatcher requires an HTTP stack, ideally with a minimum of dependencies > and able to interoperate with Netty 4.0.28 (on which Flink currently > depends). The `runtime-web` module has some home-grown HTTP handlers > already, and the `runtime` module has some low-level Netty code worth reusing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (FLINK-4897) Implement Dispatcher to support Flink sessions
[ https://issues.apache.org/jira/browse/FLINK-4897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eron Wright reassigned FLINK-4897: --- Assignee: Till Rohrmann (was: Eron Wright ) > Implement Dispatcher to support Flink sessions > -- > > Key: FLINK-4897 > URL: https://issues.apache.org/jira/browse/FLINK-4897 > Project: Flink > Issue Type: New Feature > Components: Cluster Management, Mesos > Environment: FLIP-6 feature branch >Reporter: Eron Wright >Assignee: Till Rohrmann >Priority: Major > Labels: flip-6 > > This task is to implement the dispatcher component which reacts to calls from > the cluster's REST endpoint. > The dispatcher is responsible for job submission, job listing, job leader > lookups, restarting jobs in case of a recovery and the cluster's component > lifecycle management. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (FLINK-4898) Refactor HTTP handlers and Netty server/client
[ https://issues.apache.org/jira/browse/FLINK-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eron Wright reassigned FLINK-4898: --- Assignee: (was: Eron Wright ) > Refactor HTTP handlers and Netty server/client > -- > > Key: FLINK-4898 > URL: https://issues.apache.org/jira/browse/FLINK-4898 > Project: Flink > Issue Type: Sub-task > Components: Cluster Management >Reporter: Eron Wright >Priority: Minor > > The dispatcher requires an HTTP stack, ideally with a minimum of dependencies > and able to interoperate with Netty 4.0.28 (on which Flink currently > depends). The `runtime-web` module has some home-grown HTTP handlers > already, and the `runtime` module has some low-level Netty code worth reusing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8533) Support MasterTriggerRestoreHook state reinitialization
[
https://issues.apache.org/jira/browse/FLINK-8533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348017#comment-16348017
]
Eron Wright commented on FLINK-8533:
-
Incidentally, a variation on this problem would (I believe) occur when using
the Kafka consumer and {{setStartFromGroupOffsets}} is enabled. That's
because the Kafka connector uses external storage to initialize its state (i.e.
the starting position) in the non-restore case.
> Support MasterTriggerRestoreHook state reinitialization
> ---
>
> Key: FLINK-8533
> URL: https://issues.apache.org/jira/browse/FLINK-8533
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.3.0
>Reporter: Eron Wright
>Assignee: Eron Wright
>Priority: Major
>
> {{MasterTriggerRestoreHook}} enables coordination with an external system for
> taking or restoring checkpoints. When execution is restarted from a
> checkpoint, {{restoreCheckpoint}} is called to restore or reinitialize the
> external system state. There's an edge case where the external state is not
> adequately reinitialized, that is when execution fails _before the first
> checkpoint_. In that case, the hook is not invoked and has no opportunity to
> restore the external state to initial conditions.
> The impact is a loss of exactly-once semantics in this case. For example, in
> the Pravega source function, the reader group state (e.g. stream position
> data) is stored externally. In the normal restore case, the reader group
> state is forcibly rewound to the checkpointed position. In the edge case
> where no checkpoint has yet been successful, the reader group state is not
> rewound and consequently some amount of stream data is not reprocessed.
> A possible fix would be to introduce an {{initializeState}} method on the
> hook interface. Similar to {{CheckpointedFunction::initializeState}}, this
> method would be invoked unconditionally upon hook initialization. The Pravega
> hook would, for example, initialize or forcibly reinitialize the reader group
> state.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (FLINK-8247) Support Hadoop-free variant of Flink on Mesos
[ https://issues.apache.org/jira/browse/FLINK-8247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eron Wright reassigned FLINK-8247: --- Assignee: Eron Wright > Support Hadoop-free variant of Flink on Mesos > - > > Key: FLINK-8247 > URL: https://issues.apache.org/jira/browse/FLINK-8247 > Project: Flink > Issue Type: Bug > Components: Mesos >Affects Versions: 1.4.0 >Reporter: Eron Wright >Assignee: Eron Wright >Priority: Major > Fix For: 1.5.0, 1.4.1 > > > In Hadoop-free mode, Hadoop isn't on the classpath. The Mesos job manager > normally uses the Hadoop UserGroupInformation class to overlay a user context > (`HADOOP_USER_NAME`) for the task managers. > Detect the absence of Hadoop and skip over the `HadoopUserOverlay`, similar > to the logic in `HadoopModuleFactory`.This may require the introduction > of an overlay factory. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-8541) Mesos RM should recover from failover timeout
Eron Wright created FLINK-8541: --- Summary: Mesos RM should recover from failover timeout Key: FLINK-8541 URL: https://issues.apache.org/jira/browse/FLINK-8541 Project: Flink Issue Type: Bug Components: Cluster Management, Mesos Affects Versions: 1.3.0 Reporter: Eron Wright Assignee: Eron Wright When a framework disconnects unexpectedly from Mesos, the framework's Mesos tasks continue to run for a configurable period of time known as the failover timeout. If the framework reconnects to Mesos after the timeout has expired, Mesos rejects the connection attempt. It is expected that the framework discard the previous framework ID and then connect as a new framework. When Flink is in this situation, the only recourse is to manually delete the ZK state where the framework ID kept. Let's improve the logic of the Mesos RM to automate that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8533) Support MasterTriggerRestoreHook state reinitialization
[
https://issues.apache.org/jira/browse/FLINK-8533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346400#comment-16346400
]
Eron Wright commented on FLINK-8533:
-
This relates to [issue
#89|https://github.com/pravega/flink-connectors/issues/89] in flink-connectors.
cc [~tzulitai]
> Support MasterTriggerRestoreHook state reinitialization
> ---
>
> Key: FLINK-8533
> URL: https://issues.apache.org/jira/browse/FLINK-8533
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.3.0
>Reporter: Eron Wright
>Assignee: Eron Wright
>Priority: Major
>
> {{MasterTriggerRestoreHook}} enables coordination with an external system for
> taking or restoring checkpoints. When execution is restarted from a
> checkpoint, {{restoreCheckpoint}} is called to restore or reinitialize the
> external system state. There's an edge case where the external state is not
> adequately reinitialized, that is when execution fails _before the first
> checkpoint_. In that case, the hook is not invoked and has no opportunity to
> restore the external state to initial conditions.
> The impact is a loss of exactly-once semantics in this case. For example, in
> the Pravega source function, the reader group state (e.g. stream position
> data) is stored externally. In the normal restore case, the reader group
> state is forcibly rewound to the checkpointed position. In the edge case
> where no checkpoint has yet been successful, the reader group state is not
> rewound and consequently some amount of stream data is not reprocessed.
> A possible fix would be to introduce an {{initializeState}} method on the
> hook interface. Similar to {{CheckpointedFunction::initializeState}}, this
> method would be invoked unconditionally upon hook initialization. The Pravega
> hook would, for example, initialize or forcibly reinitialize the reader group
> state.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-8533) Support MasterTriggerRestoreHook state reinitialization
Eron Wright created FLINK-8533:
---
Summary: Support MasterTriggerRestoreHook state reinitialization
Key: FLINK-8533
URL: https://issues.apache.org/jira/browse/FLINK-8533
Project: Flink
Issue Type: Bug
Components: State Backends, Checkpointing
Affects Versions: 1.3.0
Reporter: Eron Wright
Assignee: Eron Wright
{{MasterTriggerRestoreHook}} enables coordination with an external system for
taking or restoring checkpoints. When execution is restarted from a checkpoint,
{{restoreCheckpoint}} is called to restore or reinitialize the external system
state. There's an edge case where the external state is not adequately
reinitialized, that is when execution fails _before the first checkpoint_. In
that case, the hook is not invoked and has no opportunity to restore the
external state to initial conditions.
The impact is a loss of exactly-once semantics in this case. For example, in
the Pravega source function, the reader group state (e.g. stream position data)
is stored externally. In the normal restore case, the reader group state is
forcibly rewound to the checkpointed position. In the edge case where no
checkpoint has yet been successful, the reader group state is not rewound and
consequently some amount of stream data is not reprocessed.
A possible fix would be to introduce an {{initializeState}} method on the hook
interface. Similar to {{CheckpointedFunction::initializeState}}, this method
would be invoked unconditionally upon hook initialization. The Pravega hook
would, for example, initialize or forcibly reinitialize the reader group state.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-5479) Per-partition watermarks in FlinkKafkaConsumer should consider idle partitions
[
https://issues.apache.org/jira/browse/FLINK-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335116#comment-16335116
]
Eron Wright commented on FLINK-5479:
-
To elaborate on my earlier comment about `max.message.time.difference.ms`,
let's consider the ideal watermark for the two types of timestamps supported by
Kafka (as per KIP-32), CreateTime and LogAppendTime.
In LogAppendTime, the timestamp is monotonically increasing with each message,
and corresponds to the wall clock time of the broker at append time. The
per-partition watermark could simply track the message time. The complication
is how to advance the watermark when the partition is idle; an in-band
heartbeat from the broker (informing the client about the progression of its
wall clock) would be ideal.
In CreateTime, the timestamp is supplied by the producer, but the broker may
enforce an upper bound ("max difference") on the delta between the message
timestamp and the broker's current time. The ideal per-partition watermark
would be the broker's current time minus the max difference.
> Per-partition watermarks in FlinkKafkaConsumer should consider idle partitions
> --
>
> Key: FLINK-5479
> URL: https://issues.apache.org/jira/browse/FLINK-5479
> Project: Flink
> Issue Type: Improvement
> Components: Kafka Connector
>Reporter: Tzu-Li (Gordon) Tai
>Priority: Blocker
> Fix For: 1.5.0
>
>
> Reported in ML:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-topic-partition-skewness-causes-watermark-not-being-emitted-td11008.html
> Similar to what's happening to idle sources blocking watermark progression in
> downstream operators (see FLINK-5017), the per-partition watermark mechanism
> in {{FlinkKafkaConsumer}} is also being blocked of progressing watermarks
> when a partition is idle. The watermark of idle partitions is always
> {{Long.MIN_VALUE}}, therefore the overall min watermark across all partitions
> of a consumer subtask will never proceed.
> It's normally not a common case to have Kafka partitions not producing any
> data, but it'll probably be good to handle this as well. I think we should
> have a localized solution similar to FLINK-5017 for the per-partition
> watermarks in {{AbstractFetcher}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-5018) Make source idle timeout user configurable
[
https://issues.apache.org/jira/browse/FLINK-5018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334670#comment-16334670
]
Eron Wright commented on FLINK-5018:
-
Yes, the above does seem like an unsupported combination. The Kafka consumer
is clearly a watermark-aware source, and should use the existing idleness
functionality (possibly with some new support for timeouts provided by the
source context). The app should not also make use of generic idleness logic in
this scenario.
> Make source idle timeout user configurable
> --
>
> Key: FLINK-5018
> URL: https://issues.apache.org/jira/browse/FLINK-5018
> Project: Flink
> Issue Type: Sub-task
> Components: DataStream API
>Reporter: Tzu-Li (Gordon) Tai
>Priority: Major
> Fix For: 1.5.0
>
>
> There are 2 cases where sources are considered idle and should emit an idle
> {{StreamStatus}} downstream, taking Kafka consumer as example:
> - The source instance was not assigned any partitions
> - The source instance was assigned partitions, but they currently don't have
> any data.
> For the second case, we can only consider it idle after a timeout threshold.
> It would be good to make this timeout user configurable besides a default
> value.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-5018) Make source idle timeout user configurable
[
https://issues.apache.org/jira/browse/FLINK-5018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334601#comment-16334601
]
Eron Wright commented on FLINK-5018:
-
To my understanding, watermarks are either generated by a source or generated
by the 'assignTimestampsAndWatermarks' operator. The latter is a generic
solution for dealing with sources that don't support watermarks. In my
opinion, if the source is watermark-aware, it should handle all aspects of
idleness (including the implementation of an idle timeout). Similarly, if the
generic watermark operator is being used, it should provide idleness logic
My fear is that the app developer will try to create unsupported combinations
of watermark-aware sources with generic watermark logic. Maybe this idle
timeout feature should be exposed as a facility for use by the source
implementation, rather than the app directly.
> Make source idle timeout user configurable
> --
>
> Key: FLINK-5018
> URL: https://issues.apache.org/jira/browse/FLINK-5018
> Project: Flink
> Issue Type: Sub-task
> Components: DataStream API
>Reporter: Tzu-Li (Gordon) Tai
>Priority: Major
> Fix For: 1.5.0
>
>
> There are 2 cases where sources are considered idle and should emit an idle
> {{StreamStatus}} downstream, taking Kafka consumer as example:
> - The source instance was not assigned any partitions
> - The source instance was assigned partitions, but they currently don't have
> any data.
> For the second case, we can only consider it idle after a timeout threshold.
> It would be good to make this timeout user configurable besides a default
> value.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8431) Allow to specify # GPUs for TaskManager in Mesos
[ https://issues.apache.org/jira/browse/FLINK-8431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329407#comment-16329407 ] Eron Wright commented on FLINK-8431: - I think that you can skip testing the FLIP-6 scripts, if you agree that the enhancement is orthogonal. Thanks for the description of your test procedure, it sounds good. > Allow to specify # GPUs for TaskManager in Mesos > > > Key: FLINK-8431 > URL: https://issues.apache.org/jira/browse/FLINK-8431 > Project: Flink > Issue Type: Improvement > Components: Cluster Management, Mesos >Reporter: Dongwon Kim >Assignee: Dongwon Kim >Priority: Minor > > Mesos provides first-class support for Nvidia GPUs [1], but Flink does not > exploit it when scheduling TaskManagers. If Mesos agents are configured to > isolate GPUs as shown in [2], TaskManagers that do not specify to use GPUs > cannot see GPUs at all. > We, therefore, need to introduce a new configuration property named > "mesos.resourcemanager.tasks.gpus" to allow users to specify # of GPUs for > each TaskManager process in Mesos. > [1] http://mesos.apache.org/documentation/latest/gpu-support/ > [2] http://mesos.apache.org/documentation/latest/gpu-support/#agent-flags -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8431) Allow to specify # GPUs for TaskManager in Mesos
[ https://issues.apache.org/jira/browse/FLINK-8431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325704#comment-16325704 ] Eron Wright commented on FLINK-8431: - [~eastcirclek] thanks for the detailed explanation. > Allow to specify # GPUs for TaskManager in Mesos > > > Key: FLINK-8431 > URL: https://issues.apache.org/jira/browse/FLINK-8431 > Project: Flink > Issue Type: Improvement > Components: Cluster Management, Mesos >Reporter: Dongwon Kim >Assignee: Dongwon Kim >Priority: Minor > > Mesos provides first-class support for Nvidia GPUs [1], but Flink does not > exploit it when scheduling TaskManagers. If Mesos agents are configured to > isolate GPUs as shown in [2], TaskManagers that do not specify to use GPUs > cannot see GPUs at all. > We, therefore, need to introduce a new configuration property named > "mesos.resourcemanager.tasks.gpus" to allow users to specify # of GPUs for > each TaskManager process in Mesos. > [1] http://mesos.apache.org/documentation/latest/gpu-support/ > [2] http://mesos.apache.org/documentation/latest/gpu-support/#agent-flags -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (FLINK-8431) Allow to specify # GPUs for TaskManager in Mesos
[
https://issues.apache.org/jira/browse/FLINK-8431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325339#comment-16325339
]
Eron Wright edited comment on FLINK-8431 at 1/13/18 9:19 PM:
--
Note that, as per the Mesos docs, Flink will need to advertise the
{{GPU_RESOURCES}} capability. I would suggest that it advertise that
capability only when Flink is configured to request GPU resources.
was (Author: eronwright):
Note that, as per the Mesos docs, Flink will need to advertise the
GPU_RESOURCES capability. I would suggest that it advertise that capability
only when Flink is configured to request GPU resources.
> Allow to specify # GPUs for TaskManager in Mesos
>
>
> Key: FLINK-8431
> URL: https://issues.apache.org/jira/browse/FLINK-8431
> Project: Flink
> Issue Type: Improvement
> Components: Cluster Management, Mesos
>Reporter: Dongwon Kim
>Assignee: Dongwon Kim
>Priority: Minor
>
> Mesos provides first-class support for Nvidia GPUs [1], but Flink does not
> exploit it when scheduling TaskManagers. If Mesos agents are configured to
> isolate GPUs as shown in [2], TaskManagers that do not specify to use GPUs
> cannot see GPUs at all.
> We, therefore, need to introduce a new configuration property named
> "mesos.resourcemanager.tasks.gpus" to allow users to specify # of GPUs for
> each TaskManager process in Mesos.
> [1] http://mesos.apache.org/documentation/latest/gpu-support/
> [2] http://mesos.apache.org/documentation/latest/gpu-support/#agent-flags
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-8431) Allow to specify # GPUs for TaskManager in Mesos
[ https://issues.apache.org/jira/browse/FLINK-8431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325339#comment-16325339 ] Eron Wright commented on FLINK-8431: - Note that, as per the Mesos docs, Flink will need to advertise the GPU_RESOURCES capability. I would suggest that it advertise that capability only when Flink is configured to request GPU resources. > Allow to specify # GPUs for TaskManager in Mesos > > > Key: FLINK-8431 > URL: https://issues.apache.org/jira/browse/FLINK-8431 > Project: Flink > Issue Type: Improvement > Components: Cluster Management, Mesos >Reporter: Dongwon Kim >Assignee: Dongwon Kim >Priority: Minor > > Mesos provides first-class support for Nvidia GPUs [1], but Flink does not > exploit it when scheduling TaskManagers. If Mesos agents are configured to > isolate GPUs as shown in [2], TaskManagers that do not specify to use GPUs > cannot see GPUs at all. > We, therefore, need to introduce a new configuration property named > "mesos.resourcemanager.tasks.gpus" to allow users to specify # of GPUs for > each TaskManager process in Mesos. > [1] http://mesos.apache.org/documentation/latest/gpu-support/ > [2] http://mesos.apache.org/documentation/latest/gpu-support/#agent-flags -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-8431) Allow to specify # GPUs for TaskManager in Mesos
[
https://issues.apache.org/jira/browse/FLINK-8431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325337#comment-16325337
]
Eron Wright commented on FLINK-8431:
-
Good news, it appears that Fenzo supports generic scalar resources as of 0.10.0
([related
commit|https://github.com/Netflix/Fenzo/commit/c689f5a133ff4e34a7ea3f1ec805a225bf454f9e#diff-b847cf74328e261119031aed8254f4a7]).
See {{com.netflix.fenzo.TaskRequest}} and
{{com.netflix.fenzo.VirtualMachineLease}}.
It should be possible to extend
{{org.apache.flink.mesos.runtime.clusterframework.LaunchableMesosWorker}} to
convey a GPU requirement as a generic scalar resource. We just need to update
the Fenzo dependency to a newer version. I'm not aware of any impediment to
using the latest version, but be sure to add me to the PR review.
> Allow to specify # GPUs for TaskManager in Mesos
>
>
> Key: FLINK-8431
> URL: https://issues.apache.org/jira/browse/FLINK-8431
> Project: Flink
> Issue Type: Improvement
> Components: Cluster Management, Mesos
>Reporter: Dongwon Kim
>Assignee: Dongwon Kim
>Priority: Minor
>
> Mesos provides first-class support for Nvidia GPUs [1], but Flink does not
> exploit it when scheduling TaskManagers. If Mesos agents are configured to
> isolate GPUs as shown in [2], TaskManagers that do not specify to use GPUs
> cannot see GPUs at all.
> We, therefore, need to introduce a new configuration property named
> "mesos.resourcemanager.tasks.gpus" to allow users to specify # of GPUs for
> each TaskManager process in Mesos.
> [1] http://mesos.apache.org/documentation/latest/gpu-support/
> [2] http://mesos.apache.org/documentation/latest/gpu-support/#agent-flags
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (FLINK-7883) Stop fetching source before a cancel with savepoint
[
https://issues.apache.org/jira/browse/FLINK-7883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16321463#comment-16321463
]
Eron Wright edited comment on FLINK-7883 at 1/11/18 12:45 AM:
---
Can we restate the problem that we're trying to solve? To my understanding,
problem one is that cancel-with-savepoint is not atomic; cancellation happens
some time after the checkpoint state is collected, causing undesirable
at-least-once behavior or rollback behavior. Maybe this could be solved by
enhancing the checkpoint barrier with a termination flag, which would cause the
{{cancel}} function to be invoked while the checkpoint synchronization lock is
still held. I don't know whether this would play nice with
{{CheckpointListener}}.
Problem two is the existence of two similar operations, cancel and stop. Stop
seems to be aimed at turning an unbounded source into a bounded source. I
think it would be awesome to be able to parameterize the stop call w/ connector
specifics, e.g. "stop at such-and-such offset". I would hesitate to co-opt the
'stop' functionality / {{StoppableFunction}} to solve problem one.
was (Author: eronwright):
Can we restate the problem that we're trying to solve? To my understanding,
problem one is that cancel-with-savepoint is not atomic; cancellation happens
some time after the checkpoint state is collected, causing undesirable
at-least-once behavior or rollback behavior. Maybe this could be solved by
enhancing the checkpoint barrier with a termination flag, which would cause the
cancel function to be invoked while the checkpoint synchronization lock is
still held. I don't know whether this would play nice with CheckpointListener.
Problem two is the existence of two similar operations, cancel and stop. Stop
seems to be aimed at turning an unbounded source into a bounded source. I
think it would be awesome to be able to parameterize the stop call w/ connector
specifics, e.g. "stop at such-and-such offset". I would hesitate to co-opt the
'stop' functionality / StoppableFunction to solve problem one.
> Stop fetching source before a cancel with savepoint
> ---
>
> Key: FLINK-7883
> URL: https://issues.apache.org/jira/browse/FLINK-7883
> Project: Flink
> Issue Type: Improvement
> Components: DataStream API, Kafka Connector, State Backends,
> Checkpointing
>Affects Versions: 1.4.0, 1.3.2
>Reporter: Antoine Philippot
>
> For a cancel with savepoint command, the JobManager trigger the cancel call
> once the savepoint is finished, but during the savepoint execution, kafka
> source continue to poll new messages which will not be part of the savepoint
> and will be replayed on the next application start.
> A solution could be to stop fetching the source stream task before triggering
> the savepoint.
> I suggest to add an interface {{StoppableFetchingSourceFunction}} with a
> method {{stopFetching}} that existant SourceFunction implementations could
> implement.
> We can add a {{stopFetchingSource}} property in
> {{CheckpointOptions}} class to pass the desired behaviour from
> {{JobManager.handleMessage(CancelJobWithSavepoint)}} to
> {{SourceStreamTask.triggerCheckpoint}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-7883) Stop fetching source before a cancel with savepoint
[
https://issues.apache.org/jira/browse/FLINK-7883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16321463#comment-16321463
]
Eron Wright commented on FLINK-7883:
-
Can we restate the problem that we're trying to solve? To my understanding,
problem one is that cancel-with-savepoint is not atomic; cancellation happens
some time after the checkpoint state is collected, causing undesirable
at-least-once behavior or rollback behavior. Maybe this could be solved by
enhancing the checkpoint barrier with a termination flag, which would cause the
cancel function to be invoked while the checkpoint synchronization lock is
still held. I don't know whether this would play nice with CheckpointListener.
Problem two is the existence of two similar operations, cancel and stop. Stop
seems to be aimed at turning an unbounded source into a bounded source. I
think it would be awesome to be able to parameterize the stop call w/ connector
specifics, e.g. "stop at such-and-such offset". I would hesitate to co-opt the
'stop' functionality / StoppableFunction to solve problem one.
> Stop fetching source before a cancel with savepoint
> ---
>
> Key: FLINK-7883
> URL: https://issues.apache.org/jira/browse/FLINK-7883
> Project: Flink
> Issue Type: Improvement
> Components: DataStream API, Kafka Connector, State Backends,
> Checkpointing
>Affects Versions: 1.4.0, 1.3.2
>Reporter: Antoine Philippot
>
> For a cancel with savepoint command, the JobManager trigger the cancel call
> once the savepoint is finished, but during the savepoint execution, kafka
> source continue to poll new messages which will not be part of the savepoint
> and will be replayed on the next application start.
> A solution could be to stop fetching the source stream task before triggering
> the savepoint.
> I suggest to add an interface {{StoppableFetchingSourceFunction}} with a
> method {{stopFetching}} that existant SourceFunction implementations could
> implement.
> We can add a {{stopFetchingSource}} property in
> {{CheckpointOptions}} class to pass the desired behaviour from
> {{JobManager.handleMessage(CancelJobWithSavepoint)}} to
> {{SourceStreamTask.triggerCheckpoint}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-8383) flink-mesos build failing: duplicate Jackson relocation in shaded jar
[
https://issues.apache.org/jira/browse/FLINK-8383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314892#comment-16314892
]
Eron Wright commented on FLINK-8383:
-
Odd since the travis build passed on the PR.
https://github.com/apache/flink/pull/5208
I'll try to spot the difference between those builds...
> flink-mesos build failing: duplicate Jackson relocation in shaded jar
> --
>
> Key: FLINK-8383
> URL: https://issues.apache.org/jira/browse/FLINK-8383
> Project: Flink
> Issue Type: Bug
> Components: Build System, Mesos
>Reporter: Tzu-Li (Gordon) Tai
>Priority: Critical
>
> Example: https://travis-ci.org/apache/flink/jobs/325604587
> The build for {{flink-mesos}} is failing with:
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.0.0:shade (shade-flink) on
> project flink-mesos_2.11: Error creating shaded jar: duplicate entry:
> META-INF/services/org.apache.flink.mesos.shaded.com.fasterxml.jackson.core.JsonFactory
> -> [Help 1]
> {code}
> Seems to be caused by
> https://github.com/apache/flink/commit/9ae4c5447a2f5aae2b65d5860f822d452a9d5af1.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-7860) Support YARN proxy user in Flink (impersonation)
[ https://issues.apache.org/jira/browse/FLINK-7860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312208#comment-16312208 ] Eron Wright commented on FLINK-7860: - That is correct, and my comments were made with the assumption that you intend to use a keytab for the long-running job. > Support YARN proxy user in Flink (impersonation) > > > Key: FLINK-7860 > URL: https://issues.apache.org/jira/browse/FLINK-7860 > Project: Flink > Issue Type: New Feature > Components: YARN >Reporter: Shuyi Chen >Assignee: Shuyi Chen > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-7860) Support YARN proxy user in Flink (impersonation)
[
https://issues.apache.org/jira/browse/FLINK-7860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310669#comment-16310669
]
Eron Wright commented on FLINK-7860:
-
I think you're overstating what Hadoop does. The proxy user functionality in
particular is intended for high-privilege processes like the Yarn RM, but
doesn't support running low-privilege code in that same process. It is just
meant to establish an effective user for RPC purposes.
The super service will need to access the service account ('joe') keytab to be
able to deploy a Flink cluster that uses it. I see no other way. Maybe I'm
missing something about your proposal.
> Support YARN proxy user in Flink (impersonation)
>
>
> Key: FLINK-7860
> URL: https://issues.apache.org/jira/browse/FLINK-7860
> Project: Flink
> Issue Type: New Feature
> Components: YARN
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-7860) Support YARN proxy user in Flink (impersonation)
[
https://issues.apache.org/jira/browse/FLINK-7860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310598#comment-16310598
]
Eron Wright commented on FLINK-7860:
-
If I understand correctly, your scenario is as follows. You have a job
submission service that launches Flink jobs into YARN. You'd like the Flink
CLI to authenticate to YARN using a super account ('flink') but impersonating a
user ('joe'). You'd like the AM to run as 'joe' and to use 'joe.keytab' to
authenticate to HDFS during job execution.
Why not run the Flink CLI using joe.keytab? I don't see the need to use a
super account here. Keep in mind that the CLI also executes Joe's code (to
construct a job graph) and thus might grab your super account keytab.
> Support YARN proxy user in Flink (impersonation)
>
>
> Key: FLINK-7860
> URL: https://issues.apache.org/jira/browse/FLINK-7860
> Project: Flink
> Issue Type: New Feature
> Components: YARN
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLINK-8275) Flink YARN deployment with Kerberos enabled not working
[ https://issues.apache.org/jira/browse/FLINK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eron Wright updated FLINK-8275: Fix Version/s: 1.4.1 > Flink YARN deployment with Kerberos enabled not working > > > Key: FLINK-8275 > URL: https://issues.apache.org/jira/browse/FLINK-8275 > Project: Flink > Issue Type: Bug > Components: Security >Affects Versions: 1.4.0 >Reporter: Shuyi Chen >Assignee: Shuyi Chen >Priority: Blocker > Fix For: 1.5.0, 1.4.1 > > > The local keytab path in YarnTaskManagerRunner is incorrectly set to the > ApplicationMaster's local keytab path. This causes jobs to fail because the > TaskManager can't read the keytab. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLINK-8265) Missing jackson dependency for flink-mesos
[
https://issues.apache.org/jira/browse/FLINK-8265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eron Wright updated FLINK-8265:
Fix Version/s: 1.5.0
> Missing jackson dependency for flink-mesos
> --
>
> Key: FLINK-8265
> URL: https://issues.apache.org/jira/browse/FLINK-8265
> Project: Flink
> Issue Type: Bug
> Components: Mesos
>Affects Versions: 1.4.0
>Reporter: Eron Wright
>Assignee: Eron Wright
>Priority: Critical
> Fix For: 1.5.0, 1.4.1
>
>
> The Jackson library that is required by Fenzo is missing from the Flink
> distribution jar-file.
> This manifests as an exception in certain circumstances when a hard
> constraint is configured ("mesos.constraints.hard.hostattribute").
> {code}
> NoClassDefFoundError:
> org/apache/flink/mesos/shaded/com/fasterxml/jackson/databind/ObjectMapper
> at com.netflix.fenzo.ConstraintFailure.(ConstraintFailure.java:35)
> at
> com.netflix.fenzo.AssignableVirtualMachine.findFailedHardConstraints(AssignableVirtualMachine.java:784)
> at
> com.netflix.fenzo.AssignableVirtualMachine.tryRequest(AssignableVirtualMachine.java:581)
> at com.netflix.fenzo.TaskScheduler.evalAssignments(TaskScheduler.java:796)
> at com.netflix.fenzo.TaskScheduler.access$1500(TaskScheduler.java:70)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLINK-8311) Flink needs documentation for network access control
[
https://issues.apache.org/jira/browse/FLINK-8311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eron Wright updated FLINK-8311:
Description:
There is a need for better documentation on what connects to what over which
ports in a Flink cluster to allow users to configure network access control
rules.
E.g. I was under the impression that in a ZK HA configuration the Job Managers
were essentially independent and only coordinated via ZK. But starting
multiple JMs in HA with the JM RPC port blocked between JMs shows that the
second JM's Akka subsystem is trying to connect to the leading JM:
{code}
INFO akka.remote.transport.ProtocolStateActor - No
response from remote for outbound association. Associate timed out after [2
ms].
WARN akka.remote.ReliableDeliverySupervisor-
Association with remote system [akka.tcp://flink@10.210.210.127:6123] has
failed, address is now gated for [5000] ms. Reason: [Association failed with
[akka.tcp://flink@10.210.210.127:6123]] Caused by: [No response from remote for
outbound association. Associate timed out after [2 ms].]
WARN akka.remote.transport.netty.NettyTransport- Remote
connection to [null] failed with
org.apache.flink.shaded.akka.org.jboss.netty.channel.ConnectTimeoutException:
connection timed out: /10.210.210.127:6123
{code}
was:
There is a need for better documentation on what connects to what over which
ports in a Flink cluster to allow users to configure network access control
rules.
E.g. I was under the impression that in a ZK HA configuration the Job Managers
were essentially independent and only coordinated via ZK. But starting
multiple JMs in HA with the JM RPC port blocked between JMs shows that the
second JM's Akka subsystem is trying to connect to the leading JM:
INFO akka.remote.transport.ProtocolStateActor - No
response from remote for outbound association. Associate timed out after [2
ms].
WARN akka.remote.ReliableDeliverySupervisor-
Association with remote system [akka.tcp://flink@10.210.210.127:6123] has
failed, address is now gated for [5000] ms. Reason: [Association failed with
[akka.tcp://flink@10.210.210.127:6123]] Caused by: [No response from remote for
outbound association. Associate timed out after [2 ms].]
WARN akka.remote.transport.netty.NettyTransport- Remote
connection to [null] failed with
org.apache.flink.shaded.akka.org.jboss.netty.channel.ConnectTimeoutException:
connection timed out: /10.210.210.127:6123
> Flink needs documentation for network access control
>
>
> Key: FLINK-8311
> URL: https://issues.apache.org/jira/browse/FLINK-8311
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Affects Versions: 1.4.0
>Reporter: Elias Levy
>
> There is a need for better documentation on what connects to what over which
> ports in a Flink cluster to allow users to configure network access control
> rules.
> E.g. I was under the impression that in a ZK HA configuration the Job
> Managers were essentially independent and only coordinated via ZK. But
> starting multiple JMs in HA with the JM RPC port blocked between JMs shows
> that the second JM's Akka subsystem is trying to connect to the leading JM:
> {code}
> INFO akka.remote.transport.ProtocolStateActor - No
> response from remote for outbound association. Associate timed out after
> [2 ms].
> WARN akka.remote.ReliableDeliverySupervisor-
> Association with remote system [akka.tcp://flink@10.210.210.127:6123] has
> failed, address is now gated for [5000] ms. Reason: [Association failed with
> [akka.tcp://flink@10.210.210.127:6123]] Caused by: [No response from remote
> for outbound association. Associate timed out after [2 ms].]
> WARN akka.remote.transport.netty.NettyTransport- Remote
> connection to [null] failed with
> org.apache.flink.shaded.akka.org.jboss.netty.channel.ConnectTimeoutException:
> connection timed out: /10.210.210.127:6123
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-8289) The RestServerEndpoint should return the address with real ip when getRestAdddress
[ https://issues.apache.org/jira/browse/FLINK-8289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301817#comment-16301817 ] Eron Wright commented on FLINK-8289: - I guess we should be clear about who is being advertised to. For example, if the value is intended for use by the client, we'd want to give the proxy address. If the value is intended for use by the proxy (as the upstream address), we'd want to give the server address. > The RestServerEndpoint should return the address with real ip when > getRestAdddress > -- > > Key: FLINK-8289 > URL: https://issues.apache.org/jira/browse/FLINK-8289 > Project: Flink > Issue Type: Bug >Affects Versions: 1.5.0 >Reporter: shuai.xu > Labels: flip-6 > > Now when RestServerEndpoint.getRestAddress, it will return an address same > with the value of config rest.address, the default it 127.0.0.1:9067, but > this address can not be accessed from another machine. And the ip for > Dispatcher and JobMaster are usually dynamically, so user will configure it > to 0.0.0.0, and the getRestAddress will return 0.0.0.0:9067, this address > will be registered to YARN or Mesos, but this address can not be accessed > from another machine also. So it need to return the real ip:port for user to > access the web monitor anywhere. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (FLINK-7860) Support YARN proxy user in Flink (impersonation)
[ https://issues.apache.org/jira/browse/FLINK-7860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301806#comment-16301806 ] Eron Wright edited comment on FLINK-7860 at 12/22/17 6:54 PM: --- Regarding how a proxy user would be configured, the goal is to set the login user to a proxy user UGI that wraps the kerberos (real) UGI. The real UGI must continue to be initialized using a keytab as normal. Rather than introduce new config settings, Flink could simply make use of Hadoop's built-in `HADOOP_PROXY_USER` environment variable. I suggest that Flink simply propagate the `HADOOP_PROXY_USER` variable to the AM/TM. Then, in `org.apache.flink.runtime.security.modules.HadoopModule`, wrap the `loginUser` with a proxy-user UGI when `HADOOP_PROXY_USER` is set and then call `UGI.setLoginUser`. This need only be done in the `loginUserFromKeytab` scenario, not in the `loginUserFromSubject` scenario since `loginUserFromSubject` already does exactly that. See HADOOP-8561. was (Author: eronwright): Regarding how a proxy user would be configured, the goal is to set the login user to a proxy user UGI that wraps the kerberos (real) UGI. The real UGI must continue to be initialized using a keytab as normal. Rather than introduce new config settings, Flink could simply make use of Hadoop's built-in `HADOOP_PROXY_USER` environment variable. I suggest that Flink simply propagate the `HADOOP_PROXY_USER` variable to the AM/TM. Then, in `org.apache.flink.runtime.security.modules.HadoopModule`, wrap the `loginUser` with a proxy-user UGI when `HADOOP_PROXY_USER` is set and then call `UGI.setLoginUser`. This need only be done in the `loginUserFromKeytab` scenario, not in the `loginUserFromSubject` scenario since `loginUserFromSubject` already does exactly that. > Support YARN proxy user in Flink (impersonation) > > > Key: FLINK-7860 > URL: https://issues.apache.org/jira/browse/FLINK-7860 > Project: Flink > Issue Type: New Feature > Components: YARN >Reporter: Shuyi Chen >Assignee: Shuyi Chen > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-7860) Support YARN proxy user in Flink (impersonation)
[ https://issues.apache.org/jira/browse/FLINK-7860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301806#comment-16301806 ] Eron Wright commented on FLINK-7860: - Regarding how a proxy user would be configured, the goal is to set the login user to a proxy user UGI that wraps the kerberos (real) UGI. The real UGI must continue to be initialized using a keytab as normal. Rather than introduce new config settings, Flink could simply make use of Hadoop's built-in `HADOOP_PROXY_USER` environment variable. I suggest that Flink simply propagate the `HADOOP_PROXY_USER` variable to the AM/TM. Then, in `org.apache.flink.runtime.security.modules.HadoopModule`, wrap the `loginUser` with a proxy-user UGI when `HADOOP_PROXY_USER` is set and then call `UGI.setLoginUser`. This need only be done in the `loginUserFromKeytab` scenario, not in the `loginUserFromSubject` scenario since `loginUserFromSubject` already does exactly that. > Support YARN proxy user in Flink (impersonation) > > > Key: FLINK-7860 > URL: https://issues.apache.org/jira/browse/FLINK-7860 > Project: Flink > Issue Type: New Feature > Components: YARN >Reporter: Shuyi Chen >Assignee: Shuyi Chen > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-7860) Support YARN proxy user in Flink (impersonation)
[ https://issues.apache.org/jira/browse/FLINK-7860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16300504#comment-16300504 ] Eron Wright commented on FLINK-7860: - Please elaborate on the scenario you hope to support. From the description, seems you want to create a powerful service account (e.g. 'Flink') with an associated keytab, then launch jobs with that keytab that impersonate other users (e.g. 'Joe'). Reads and writes to HDFS would authenticate with the 'Flink' keytab but would behave as though made by 'Joe'. In addition, the HDFS service would be configured to allow 'Flink' to impersonate 'Joe'. If that's the scenario, I think it is a reasonable one. But please consider two complications: 1. The job code has access to the keytab, since the code runs in-process. The job code must be considered 'trusted', since it could steal the keytab and impersonate other users. 2. The Flink cluster doesn't protect the keytab very well; other processes could also steal it. It seems unwise to give such a powerful keytab to Flink. > Support YARN proxy user in Flink (impersonation) > > > Key: FLINK-7860 > URL: https://issues.apache.org/jira/browse/FLINK-7860 > Project: Flink > Issue Type: New Feature > Components: YARN >Reporter: Shuyi Chen >Assignee: Shuyi Chen > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (FLINK-8289) The RestServerEndpoint should return the address with real ip when getRestAdddress
[ https://issues.apache.org/jira/browse/FLINK-8289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297187#comment-16297187 ] Eron Wright edited comment on FLINK-8289 at 12/19/17 6:21 PM: --- I'll use the terms 'advertised' versus 'bind' address to discuss this issue. Do you agree that the goal here is to return the advertised address? The Flink docs are unclear on which configuration setting is applicable. Two complications: 1. *Yarn Proxy / Mesos Admin Router.* In both environments, web traffic is expected to be proxied, so the advertised address should be the proxy address. 2. *SSL*. To enable SSL on the web endpoints, two things are needed: a. Advertise a name-based (not IP-based) address. b. Construct the advertised address with 'https' scheme. See the proposed SSL spec for more information on point (2). [FLIP - Service Authorization (SSL)|https://docs.google.com/document/d/13IRPb2GdL842rIzMgEn0ibOQHNku6W8aMf1p7gCPJjg/edit?usp=sharing] was (Author: eronwright): I'll use the terms 'advertised' versus 'bind' address to discuss this issue. Do you agree that the goal here is to return the advertised address? The Flink docs are unclear on which configuration setting is applicable. Two complications: 1. *Yarn Proxy / Mesos Admin Router.* In both environments, web traffic is expected to be proxied, so the advertised address should be the proxy address. 2. *SSL*. To enable SSL on the web endpoints, two things are needed: a. Advertise a name-based (not IP-based) address. b. Construct the advertised address with 'https' scheme. > The RestServerEndpoint should return the address with real ip when > getRestAdddress > -- > > Key: FLINK-8289 > URL: https://issues.apache.org/jira/browse/FLINK-8289 > Project: Flink > Issue Type: Bug >Affects Versions: 1.5.0 >Reporter: shuai.xu > Labels: flip-6 > > Now when RestServerEndpoint.getRestAddress, it will return an address same > with the value of config rest.address, the default it 127.0.0.1:9067, but > this address can not be accessed from another machine. And the ip for > Dispatcher and JobMaster are usually dynamically, so user will configure it > to 0.0.0.0, and the getRestAddress will return 0.0.0.0:9067, this address > will be registered to YARN or Mesos, but this address can not be accessed > from another machine also. So it need to return the real ip:port for user to > access the web monitor anywhere. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-8289) The RestServerEndpoint should return the address with real ip when getRestAdddress
[ https://issues.apache.org/jira/browse/FLINK-8289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297187#comment-16297187 ] Eron Wright commented on FLINK-8289: - I'll use the terms 'advertised' versus 'bind' address to discuss this issue. Do you agree that the goal here is to return the advertised address? The Flink docs are unclear on which configuration setting is applicable. Two complications: 1. *Yarn Proxy / Mesos Admin Router.* In both environments, web traffic is expected to be proxied, so the advertised address should be the proxy address. 2. *SSL*. To enable SSL on the web endpoints, two things are needed: a. Advertise a name-based (not IP-based) address. b. Construct the advertised address with 'https' scheme. > The RestServerEndpoint should return the address with real ip when > getRestAdddress > -- > > Key: FLINK-8289 > URL: https://issues.apache.org/jira/browse/FLINK-8289 > Project: Flink > Issue Type: Bug >Affects Versions: 1.5.0 >Reporter: shuai.xu > Labels: flip-6 > > Now when RestServerEndpoint.getRestAddress, it will return an address same > with the value of config rest.address, the default it 127.0.0.1:9067, but > this address can not be accessed from another machine. And the ip for > Dispatcher and JobMaster are usually dynamically, so user will configure it > to 0.0.0.0, and the getRestAddress will return 0.0.0.0:9067, this address > will be registered to YARN or Mesos, but this address can not be accessed > from another machine also. So it need to return the real ip:port for user to > access the web monitor anywhere. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
