[jira] [Commented] (FLINK-35740) [mysql] Allow column as chunk key even if not in Primary Keys
[ https://issues.apache.org/jira/browse/FLINK-35740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17867938#comment-17867938 ] ouyangwulin commented on FLINK-35740: - [~seung-min] As this ticket describe，I have some Questtion. # why we only supported mysql? # Can we do the support configuration to discover primarykeys first, before discovering unique indexes, if neither of these indexes should be supported? > [mysql] Allow column as chunk key even if not in Primary Keys > - > > Key: FLINK-35740 > URL: https://issues.apache.org/jira/browse/FLINK-35740 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: Lee SeungMin >Priority: Major > Labels: pull-request-available > > PR: [[FLINK-35740] Allow column as chunk key even if not in Primary Keys by > SML0127 · Pull Request #3448 · apache/flink-cdc > (github.com)|https://github.com/apache/flink-cdc/pull/3448] > Allow column as chunk key even if it is not in the primary keys. > > There are cases where the primary key is not a numeric type, such as varchar > or varbinary. > In this case, the `distributed factor` and `chunk range` may be calculated > incorrectly, resulting in one chunk containing too many records. > But there was no conditions for restrict type for primary keys or chunk key > columns, so this may cause out of memory in the task manager. > Actually, in our company, there was a mysql tables that PK is a varbinary > column and the above situation occurred. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-35354) Support host mapping in Flink tikv cdc
[ https://issues.apache.org/jira/browse/FLINK-35354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ouyangwulin closed FLINK-35354. --- Resolution: Fixed master:302a69122538fdb76b98e73ebb3c83ee733a0c02 > Support host mapping in Flink tikv cdc > -- > > Key: FLINK-35354 > URL: https://issues.apache.org/jira/browse/FLINK-35354 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Affects Versions: cdc-3.1.0, cdc-3.2.0 >Reporter: ouyangwulin >Priority: Major > Labels: pull-request-available > Fix For: cdc-3.2.0 > > > In tidb production environment deployment, there are usually two kinds of > network: internal network and public network. When we use pd mode in tikv, we > need to do network mapping, such as `spark.tispark.host_mapping` in > [https://github.com/pingcap/tispark/blob/master/docs/userguide_3.0.md]. So I > think we need support `host_mapping` in our Flink tikv cdc connector. > > Add param: > tikv.host_mapping:192.168.0.2:8.8.8.8;192.168.0.3:9.9.9.9 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35459) Use Incremental Source Framework in Flink CDC TiKV Source Connector
ouyangwulin created FLINK-35459: --- Summary: Use Incremental Source Framework in Flink CDC TiKV Source Connector Key: FLINK-35459 URL: https://issues.apache.org/jira/browse/FLINK-35459 Project: Flink Issue Type: New Feature Components: Flink CDC Affects Versions: cdc-3.2.0 Reporter: ouyangwulin Fix For: cdc-3.2.0 Use Incremental Source Framework in Flink CDC TiKV Source Connector -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35450) Introduce Tikv pipeline DataSource
ouyangwulin created FLINK-35450: --- Summary: Introduce Tikv pipeline DataSource Key: FLINK-35450 URL: https://issues.apache.org/jira/browse/FLINK-35450 Project: Flink Issue Type: New Feature Components: Flink CDC Affects Versions: cdc-3.2.0 Reporter: ouyangwulin Fix For: cdc-3.2.0 After we add hostmapping(https://issues.apache.org/jira/browse/FLINK-35354) to tikv，we can use flink cdc sync data from tikv. We need more convenient and high-performance data synchronization capabilities from tikv -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34841) [3.1][pipeline-connectors] Add jdbc pipeline sink
[ https://issues.apache.org/jira/browse/FLINK-34841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17849443#comment-17849443 ] ouyangwulin commented on FLINK-34841: - [~gongzhongqiang] Looks like a very good job, cdc data entry warehouse is needed in many scenarios. > [3.1][pipeline-connectors] Add jdbc pipeline sink > -- > > Key: FLINK-34841 > URL: https://issues.apache.org/jira/browse/FLINK-34841 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: Flink CDC Issue Import >Assignee: Zhongqiang Gong >Priority: Major > Labels: github-import > > ### Search before asking > - [X] I searched in the > [issues|https://github.com/ververica/flink-cdc-connectors/issues] and found > nothing similar. > ### Motivation > From my side and I saw in dingding group, Some user want to sync data to > relation database by flink cdc. > ### Solution > _No response_ > ### Alternatives > _No response_ > ### Anything else? > _No response_ > ### Are you willing to submit a PR? > - [X] I'm willing to submit a PR! > Imported from GitHub > Url: https://github.com/apache/flink-cdc/issues/2866 > Created by: [GOODBOY008|https://github.com/GOODBOY008] > Labels: enhancement, > Created at: Wed Dec 13 15:34:21 CST 2023 > State: open -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34688) CDC framework split snapshot chunks asynchronously
[ https://issues.apache.org/jira/browse/FLINK-34688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17849441#comment-17849441 ] ouyangwulin commented on FLINK-34688: - When the size of the data is relatively large, this feature is really needed. > CDC framework split snapshot chunks asynchronously > -- > > Key: FLINK-34688 > URL: https://issues.apache.org/jira/browse/FLINK-34688 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: Hongshun Wang >Priority: Major > Fix For: cdc-3.2.0 > > > In Mysql CDC, MysqlSnapshotSplitAssigner splits snapshot chunks > asynchronously([https://github.com/apache/flink-cdc/pull/931).] But CDC > framework lacks it. > If table is too big to split, the enumerator will be stuck, and checkpoint > will be influenced( sometime will checkpoint timeout occurs). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34901) Update clause must EXCLUDED unique index
[
https://issues.apache.org/jira/browse/FLINK-34901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ouyangwulin updated FLINK-34901:
Summary: Update clause must EXCLUDED unique index (was: Improve the
performance of the update Postgres database)
> Update clause must EXCLUDED unique index
>
>
> Key: FLINK-34901
> URL: https://issues.apache.org/jira/browse/FLINK-34901
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: jdbc-3.1.2
>Reporter: ouyangwulin
>Priority: Critical
> Labels: pull-request-available
>
> When writing Postgres data using flinker-connector-jdbc, using upsert mode,
> we need only update no uniqueColumns to improve performance.
>
> Whether you can modify the code is as follows：
> String updateClause =
>
> {code:java}
> Arrays.stream(fieldNames)
> .filter(fieldName -> !Arrays.asList(uniqueKeyFields).contains(fieldName))
> .map(f -> quoteIdentifier(f) + "=EXCLUDED." + quoteIdentifier(f))
> .collect(Collectors.joining(", "));{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-34901) Update clause must EXCLUDED unique index
[
https://issues.apache.org/jira/browse/FLINK-34901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ouyangwulin updated FLINK-34901:
Description:
When writing Postgres data using flinke-connector-jdbc, using upsert mode, As
docs '[https://www.postgresql.org/docs/current/sql-insert.html]', when we use
ON CONFLICT, Update clause must EXCLUDED unique index.
Or it will throw exceptions.like " modification of distribution columns in
OnConflictUpdate is not supported Call getNextException to see other errors in
the batch."
Whether you can modify the code is as follows：
String updateClause =
{code:java}
Arrays.stream(fieldNames)
.filter(fieldName -> !Arrays.asList(uniqueKeyFields).contains(fieldName))
.map(f -> quoteIdentifier(f) + "=EXCLUDED." + quoteIdentifier(f))
.collect(Collectors.joining(", "));{code}
was:
When writing Postgres data using flinker-connector-jdbc, using upsert mode, we
need only update no uniqueColumns to improve performance.
Whether you can modify the code is as follows：
String updateClause =
{code:java}
Arrays.stream(fieldNames)
.filter(fieldName -> !Arrays.asList(uniqueKeyFields).contains(fieldName))
.map(f -> quoteIdentifier(f) + "=EXCLUDED." + quoteIdentifier(f))
.collect(Collectors.joining(", "));{code}
> Update clause must EXCLUDED unique index
>
>
> Key: FLINK-34901
> URL: https://issues.apache.org/jira/browse/FLINK-34901
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: jdbc-3.1.2
>Reporter: ouyangwulin
>Priority: Critical
> Labels: pull-request-available
>
> When writing Postgres data using flinke-connector-jdbc, using upsert mode, As
> docs '[https://www.postgresql.org/docs/current/sql-insert.html]', when we use
> ON CONFLICT, Update clause must EXCLUDED unique index.
> Or it will throw exceptions.like " modification of distribution columns in
> OnConflictUpdate is not supported Call getNextException to see other errors
> in the batch."
> Whether you can modify the code is as follows：
> String updateClause =
>
> {code:java}
> Arrays.stream(fieldNames)
> .filter(fieldName -> !Arrays.asList(uniqueKeyFields).contains(fieldName))
> .map(f -> quoteIdentifier(f) + "=EXCLUDED." + quoteIdentifier(f))
> .collect(Collectors.joining(", "));{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-35360) [Feature] Submit Flink CDC pipeline job yarn Application mode
[ https://issues.apache.org/jira/browse/FLINK-35360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17848423#comment-17848423 ] ouyangwulin commented on FLINK-35360: - With many enterprises still running on yarn, this functionality is desperately needed. [~kwafor] Let's push it together. > [Feature] Submit Flink CDC pipeline job yarn Application mode > - > > Key: FLINK-35360 > URL: https://issues.apache.org/jira/browse/FLINK-35360 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Affects Versions: cdc-3.2.0 >Reporter: wangjunbo >Priority: Minor > > For now flink-cdc pipeline support cli yarn session mode submit.I'm willing > to support yarn application mode submit. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-35337) Keep up with the latest version of tikv client
[ https://issues.apache.org/jira/browse/FLINK-35337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ouyangwulin closed FLINK-35337. --- Release Note: tikv version not supported Resolution: Won't Fix > Keep up with the latest version of tikv client > -- > > Key: FLINK-35337 > URL: https://issues.apache.org/jira/browse/FLINK-35337 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: ouyangwulin >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35354) Support host mapping in Flink tikv cdc
[ https://issues.apache.org/jira/browse/FLINK-35354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ouyangwulin updated FLINK-35354: Description: In tidb production environment deployment, there are usually two kinds of network: internal network and public network. When we use pd mode in tikv, we need to do network mapping, such as `spark.tispark.host_mapping` in [https://github.com/pingcap/tispark/blob/master/docs/userguide_3.0.md]. So I think we need support `host_mapping` in our Flink tikv cdc connector. Add param: tikv.host_mapping:192.168.0.2:8.8.8.8;192.168.0.3:9.9.9.9 was: In tidb production environment deployment, there are usually two kinds of network: internal network and public network. When we use pd mode in tikv, we need to do network mapping, such as `spark.tispark.host_mapping` in [https://github.com/pingcap/tispark/blob/master/docs/userguide_3.0.md]. So I think we need support `host_mapping` in our Flink tikv cdc connector. add param: tikv.host_mapping:192.168.0.2:8.8.8.8;192.168.0.3:9.9.9.9 > Support host mapping in Flink tikv cdc > -- > > Key: FLINK-35354 > URL: https://issues.apache.org/jira/browse/FLINK-35354 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Affects Versions: cdc-3.1.0, cdc-3.2.0 >Reporter: ouyangwulin >Priority: Major > Fix For: cdc-3.1.0, cdc-3.2.0 > > > In tidb production environment deployment, there are usually two kinds of > network: internal network and public network. When we use pd mode in tikv, we > need to do network mapping, such as `spark.tispark.host_mapping` in > [https://github.com/pingcap/tispark/blob/master/docs/userguide_3.0.md]. So I > think we need support `host_mapping` in our Flink tikv cdc connector. > > Add param: > tikv.host_mapping:192.168.0.2:8.8.8.8;192.168.0.3:9.9.9.9 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35354) Support host mapping in Flink tikv cdc
[ https://issues.apache.org/jira/browse/FLINK-35354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ouyangwulin updated FLINK-35354: Description: In tidb production environment deployment, there are usually two kinds of network: internal network and public network. When we use pd mode in tikv, we need to do network mapping, such as `spark.tispark.host_mapping` in [https://github.com/pingcap/tispark/blob/master/docs/userguide_3.0.md]. So I think we need support `host_mapping` in our Flink tikv cdc connector. add param: tikv.host_mapping:192.168.0.2:8.8.8.8;192.168.0.3:9.9.9.9 was: In tidb production environment deployment, there are usually two kinds of network: internal network and public network. When we use pd mode in tikv, we need to do network mapping, such as `spark.tispark.host_mapping` in [https://github.com/pingcap/tispark/blob/master/docs/userguide_3.0.md]. So I think we need support `host_mapping` in our Flink tikv cdc connector. add param: tikv.host_mapping > Support host mapping in Flink tikv cdc > -- > > Key: FLINK-35354 > URL: https://issues.apache.org/jira/browse/FLINK-35354 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Affects Versions: cdc-3.1.0, cdc-3.2.0 >Reporter: ouyangwulin >Priority: Major > Fix For: cdc-3.1.0, cdc-3.2.0 > > > In tidb production environment deployment, there are usually two kinds of > network: internal network and public network. When we use pd mode in tikv, we > need to do network mapping, such as `spark.tispark.host_mapping` in > [https://github.com/pingcap/tispark/blob/master/docs/userguide_3.0.md]. So I > think we need support `host_mapping` in our Flink tikv cdc connector. > > add param: > tikv.host_mapping:192.168.0.2:8.8.8.8;192.168.0.3:9.9.9.9 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35354) Support host mapping in Flink tikv cdc
[ https://issues.apache.org/jira/browse/FLINK-35354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ouyangwulin updated FLINK-35354: Description: In tidb production environment deployment, there are usually two kinds of network: internal network and public network. When we use pd mode in tikv, we need to do network mapping, such as `spark.tispark.host_mapping` in [https://github.com/pingcap/tispark/blob/master/docs/userguide_3.0.md]. So I think we need support `host_mapping` in our Flink tikv cdc connector. add param: tikv.host_mapping was:In tidb production environment deployment, there are usually two kinds of network: internal network and public network. When we use pd mode in tikv, we need to do network mapping, such as `spark.tispark.host_mapping` in [https://github.com/pingcap/tispark/blob/master/docs/userguide_3.0.md]. So I think we need support `host_mapping` in our Flink tikv cdc connector. > Support host mapping in Flink tikv cdc > -- > > Key: FLINK-35354 > URL: https://issues.apache.org/jira/browse/FLINK-35354 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Affects Versions: cdc-3.1.0, cdc-3.2.0 >Reporter: ouyangwulin >Priority: Major > Fix For: cdc-3.1.0, cdc-3.2.0 > > > In tidb production environment deployment, there are usually two kinds of > network: internal network and public network. When we use pd mode in tikv, we > need to do network mapping, such as `spark.tispark.host_mapping` in > [https://github.com/pingcap/tispark/blob/master/docs/userguide_3.0.md]. So I > think we need support `host_mapping` in our Flink tikv cdc connector. > > add param: > tikv.host_mapping -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35354) Support host mapping in Flink tikv cdc
[ https://issues.apache.org/jira/browse/FLINK-35354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ouyangwulin updated FLINK-35354: Description: In tidb production environment deployment, there are usually two kinds of network: internal network and public network. When we use pd mode in tikv, we need to do network mapping, such as `spark.tispark.host_mapping` in [https://github.com/pingcap/tispark/blob/master/docs/userguide_3.0.md]. So I think we need support `host_mapping` in our Flink tikv cdc connector. (was: In tidb production environment deployment, there are usually two kinds of network: internal network and public network. When we use pd mode kv, we need to do network mapping, such as `spark.tispark.host_mapping` in https://github.com/pingcap/tispark/blob/master/docs/userguide_3.0.md. So I think we need support `host_mapping` in our Flink tikv cdc connector.) > Support host mapping in Flink tikv cdc > -- > > Key: FLINK-35354 > URL: https://issues.apache.org/jira/browse/FLINK-35354 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Affects Versions: cdc-3.1.0, cdc-3.2.0 >Reporter: ouyangwulin >Priority: Major > Fix For: cdc-3.1.0, cdc-3.2.0 > > > In tidb production environment deployment, there are usually two kinds of > network: internal network and public network. When we use pd mode in tikv, we > need to do network mapping, such as `spark.tispark.host_mapping` in > [https://github.com/pingcap/tispark/blob/master/docs/userguide_3.0.md]. So I > think we need support `host_mapping` in our Flink tikv cdc connector. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35354) Support host mapping in Flink tikv cdc
[ https://issues.apache.org/jira/browse/FLINK-35354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ouyangwulin updated FLINK-35354: Summary: Support host mapping in Flink tikv cdc (was: [discuss] Support host mapping in Flink tikv cdc) > Support host mapping in Flink tikv cdc > -- > > Key: FLINK-35354 > URL: https://issues.apache.org/jira/browse/FLINK-35354 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Affects Versions: cdc-3.1.0, cdc-3.2.0 >Reporter: ouyangwulin >Priority: Major > Fix For: cdc-3.1.0, cdc-3.2.0 > > > In tidb production environment deployment, there are usually two kinds of > network: internal network and public network. When we use pd mode kv, we need > to do network mapping, such as `spark.tispark.host_mapping` in > https://github.com/pingcap/tispark/blob/master/docs/userguide_3.0.md. So I > think we need support `host_mapping` in our Flink tikv cdc connector. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35354) [discuss] Support host mapping in Flink tikv cdc
ouyangwulin created FLINK-35354: --- Summary: [discuss] Support host mapping in Flink tikv cdc Key: FLINK-35354 URL: https://issues.apache.org/jira/browse/FLINK-35354 Project: Flink Issue Type: Improvement Components: Flink CDC Affects Versions: cdc-3.1.0, cdc-3.2.0 Reporter: ouyangwulin Fix For: cdc-3.1.0, cdc-3.2.0 In tidb production environment deployment, there are usually two kinds of network: internal network and public network. When we use pd mode kv, we need to do network mapping, such as `spark.tispark.host_mapping` in https://github.com/pingcap/tispark/blob/master/docs/userguide_3.0.md. So I think we need support `host_mapping` in our Flink tikv cdc connector. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34901) Improve the performance of the update Postgres database
[
https://issues.apache.org/jira/browse/FLINK-34901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ouyangwulin updated FLINK-34901:
Affects Version/s: jdbc-3.1.2
> Improve the performance of the update Postgres database
> ---
>
> Key: FLINK-34901
> URL: https://issues.apache.org/jira/browse/FLINK-34901
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / JDBC
>Affects Versions: jdbc-3.1.2
>Reporter: ouyangwulin
>Priority: Critical
> Labels: pull-request-available
>
> When writing Postgres data using flinker-connector-jdbc, using upsert mode,
> we need only update no uniqueColumns to improve performance.
>
> Whether you can modify the code is as follows：
> String updateClause =
>
> {code:java}
> Arrays.stream(fieldNames)
> .filter(fieldName -> !Arrays.asList(uniqueKeyFields).contains(fieldName))
> .map(f -> quoteIdentifier(f) + "=EXCLUDED." + quoteIdentifier(f))
> .collect(Collectors.joining(", "));{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-34901) Improve the performance of the update Postgres database
[
https://issues.apache.org/jira/browse/FLINK-34901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ouyangwulin updated FLINK-34901:
Priority: Critical (was: Minor)
> Improve the performance of the update Postgres database
> ---
>
> Key: FLINK-34901
> URL: https://issues.apache.org/jira/browse/FLINK-34901
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / JDBC
>Reporter: ouyangwulin
>Priority: Critical
> Labels: pull-request-available
>
> When writing Postgres data using flinker-connector-jdbc, using upsert mode,
> we need only update no uniqueColumns to improve performance.
>
> Whether you can modify the code is as follows：
> String updateClause =
>
> {code:java}
> Arrays.stream(fieldNames)
> .filter(fieldName -> !Arrays.asList(uniqueKeyFields).contains(fieldName))
> .map(f -> quoteIdentifier(f) + "=EXCLUDED." + quoteIdentifier(f))
> .collect(Collectors.joining(", "));{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34901) Improve the performance of the update Postgres database
[
https://issues.apache.org/jira/browse/FLINK-34901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17846235#comment-17846235
]
ouyangwulin commented on FLINK-34901:
-
Some versions of pgsql also give this error：
{code:java}
//ERROR: modification of distribution columns in OnConflictUpdate is not
supported Call getNextException to see other errors in the batch.
at
org.postgresql.jdbc.BatchResultHandler.handleCompletion(BatchResultHandler.java:186)
~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.jdbc.PgStatement.internalExecuteBatch(PgStatement.java:878)
~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:916)
~[postgresql-42.5.4.jar:42.5.4]
at
org.postgresql.jdbc.PgPreparedStatement.executeBatch(PgPreparedStatement.java:1684)
~[postgresql-42.5.4.jar:42.5.4]
at
org.apache.flink.connector.jdbc.statement.FieldNamedPreparedStatementImpl.executeBatch(FieldNamedPreparedStatementImpl.java:65)
~[flink-connector-jdbc-3.1.2-1.18.jar:3.1.2-1.18]
at
org.apache.flink.connector.jdbc.internal.executor.TableSimpleStatementExecutor.executeBatch(TableSimpleStatementExecutor.java:64)
~[flink-connector-jdbc-3.1.2-1.18.jar:3.1.2-1.18]
at
org.apache.flink.connector.jdbc.internal.executor.TableBufferReducedStatementExecutor.executeBatch(TableBufferReducedStatementExecutor.java:101)
~[flink-connector-jdbc-3.1.2-1.18.jar:3.1.2-1.18]
at
org.apache.flink.connector.jdbc.internal.JdbcOutputFormat.attemptFlush(JdbcOutputFormat.java:246)
~[flink-connector-jdbc-3.1.2-1.18.jar:3.1.2-1.18]
at
org.apache.flink.connector.jdbc.internal.JdbcOutputFormat.flush(JdbcOutputFormat.java:216)
~[flink-connector-jdbc-3.1.2-1.18.jar:3.1.2-1.18]
at
org.apache.flink.connector.jdbc.internal.JdbcOutputFormat.writeRecord(JdbcOutputFormat.java:195)
~[flink-connector-jdbc-3.1.2-1.18.jar:3.1.2-1.18]
... 15 more
Caused by: org.postgresql.util.PSQLException: ERROR: modification of
distribution columns in OnConflictUpdate is not supported
at
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2676)
~[postgresql-42.5.4.jar:42.5.4]
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2366)
~[postgresql-42.5.4.jar:42.5.4]
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:356)
~[postgresql-42.5.4.jar:42.5.4]
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:316)
~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.jdbc.PgStatement.internalExecuteBatch(PgStatement.java:874)
~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:916)
~[postgresql-42.5.4.jar:42.5.4]
at org. {code}
> Improve the performance of the update Postgres database
> ---
>
> Key: FLINK-34901
> URL: https://issues.apache.org/jira/browse/FLINK-34901
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / JDBC
>Reporter: ouyangwulin
>Priority: Minor
> Labels: pull-request-available
>
> When writing Postgres data using flinker-connector-jdbc, using upsert mode,
> we need only update no uniqueColumns to improve performance.
>
> Whether you can modify the code is as follows：
> String updateClause =
>
> {code:java}
> Arrays.stream(fieldNames)
> .filter(fieldName -> !Arrays.asList(uniqueKeyFields).contains(fieldName))
> .map(f -> quoteIdentifier(f) + "=EXCLUDED." + quoteIdentifier(f))
> .collect(Collectors.joining(", "));{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-35337) Keep up with the latest version of tikv client
[ https://issues.apache.org/jira/browse/FLINK-35337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845887#comment-17845887 ] ouyangwulin commented on FLINK-35337: - [~Leonard]， Our production environment has been updated tikv version 6.5.4,Can we update it to new version. > Keep up with the latest version of tikv client > -- > > Key: FLINK-35337 > URL: https://issues.apache.org/jira/browse/FLINK-35337 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: ouyangwulin >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35337) Keep up with the latest version of tikv client
[ https://issues.apache.org/jira/browse/FLINK-35337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ouyangwulin updated FLINK-35337: Component/s: Flink CDC > Keep up with the latest version of tikv client > -- > > Key: FLINK-35337 > URL: https://issues.apache.org/jira/browse/FLINK-35337 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: ouyangwulin >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35337) Keep up with the latest version of tikv client
ouyangwulin created FLINK-35337: --- Summary: Keep up with the latest version of tikv client Key: FLINK-35337 URL: https://issues.apache.org/jira/browse/FLINK-35337 Project: Flink Issue Type: Improvement Reporter: ouyangwulin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34722) Support conditional upserts with Postgres JDBC sink
[
https://issues.apache.org/jira/browse/FLINK-34722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829775#comment-17829775
]
ouyangwulin commented on FLINK-34722:
-
[~pietro97] I think it is a very useful provement for jdbc connector in our
production env. we can impl provide the statement through options of the JDBC
sink connector replace 'rewrite the whole JDBC'.
> Support conditional upserts with Postgres JDBC sink
> ---
>
> Key: FLINK-34722
> URL: https://issues.apache.org/jira/browse/FLINK-34722
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / JDBC
>Affects Versions: jdbc-3.1.2
>Reporter: Pietro
>Priority: Major
>
> The default Postgres dialect used by the JDBC sink for PostgreSQL DBs does
> not support custom _WHERE_ conditions inside upsert statements at the moment.
> Indeed, upsert statements returned by the
> {{[getUpsertStatement()|https://github.com/apache/flink-connector-jdbc/blob/95294ffbc57c93c2af34cda94c27fc5781e06177/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/dialect/AbstractPostgresCompatibleDialect.java#L61]}}
> method are limited to:
> {code:sql}
> ON CONFLICT (col1, ..., colN)
> DO UPDATE SET (col1=EXCLUDED.col1, ..., colN=EXCLUDED.colN)
> {code}
> PostgreSQL allows a finer-grained control of upsert statements by specifying
> a _WHERE_ statement (see [ON CONFLICT
> Clause|https://www.postgresql.org/docs/current/sql-insert.html#SQL-ON-CONFLICT]),
> for instance:
> {code:sql}
> ON CONFLICT (col1, ..., colN)
> DO UPDATE SET (col1=EXCLUDED.col1, ..., colN=EXCLUDED.colN)
> WHERE colN < EXCLUDED.colN
> {code}
>
> This could be useful in many use cases, for instance, in a CDC scenario where
> a batch reconciliation process has written records in the destination, which
> now face the risk of being overwritten by late arriving, stale, records in
> the streaming pipeline (adding a condition on the operation timestamp could
> protect from these events).
> My proposal is to extend the
> {{[AbstractPostgresCompatibleDialect|https://github.com/apache/flink-connector-jdbc/blob/main/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/dialect/AbstractPostgresCompatibleDialect.java]}}
> functionalities by making the upsert query support _WHERE_ statements
> provided by users.
> I'm thinking of two possible approaches, but I'd love to hear your opinion on
> this:
> # provide the statement through options of the JDBC sink connector.
> # allow users to plug custom dialects without them having to rewrite the
> whole JDBC sink (about this I'll open a separate issue soon)
> Thanks for your consideration
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34901) Improve the performance of the update Postgres database
ouyangwulin created FLINK-34901:
---
Summary: Improve the performance of the update Postgres database
Key: FLINK-34901
URL: https://issues.apache.org/jira/browse/FLINK-34901
Project: Flink
Issue Type: Improvement
Components: Connectors / JDBC
Reporter: ouyangwulin
When writing Postgres data using flinker-connector-jdbc, using upsert mode, we
need only update no uniqueColumns to improve performance.
Whether you can modify the code is as follows：
String updateClause =
{code:java}
Arrays.stream(fieldNames)
.filter(fieldName -> !Arrays.asList(uniqueKeyFields).contains(fieldName))
.map(f -> quoteIdentifier(f) + "=EXCLUDED." + quoteIdentifier(f))
.collect(Collectors.joining(", "));{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33319) Add AverageTime metric to measure delta change in GC time
[ https://issues.apache.org/jira/browse/FLINK-33319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17803379#comment-17803379 ] ouyangwulin commented on FLINK-33319: - [~gyfora] Could you assign this ticket to me？ > Add AverageTime metric to measure delta change in GC time > - > > Key: FLINK-33319 > URL: https://issues.apache.org/jira/browse/FLINK-33319 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Metrics >Reporter: Gyula Fora >Priority: Minor > Fix For: 1.19.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-33491) Support json column validated
[
https://issues.apache.org/jira/browse/FLINK-33491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ouyangwulin closed FLINK-33491.
---
Resolution: Not A Problem
> Support json column validated
> -
>
> Key: FLINK-33491
> URL: https://issues.apache.org/jira/browse/FLINK-33491
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / Runtime
>Affects Versions: 1.19.0
>Reporter: ouyangwulin
>Assignee: ouyangwulin
>Priority: Minor
>
> Just like the {{is_valid_json}} function in PostgreSQL, it would be useful to
> have an inbuilt function to check whether a string conforms to the JSON
> specification.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33491) Support json column validated
[
https://issues.apache.org/jira/browse/FLINK-33491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17785468#comment-17785468
]
ouyangwulin commented on FLINK-33491:
-
This is my error, I didn't find the is json function.
> Support json column validated
> -
>
> Key: FLINK-33491
> URL: https://issues.apache.org/jira/browse/FLINK-33491
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / Runtime
>Affects Versions: 1.19.0
>Reporter: ouyangwulin
>Assignee: ouyangwulin
>Priority: Minor
>
> Just like the {{is_valid_json}} function in PostgreSQL, it would be useful to
> have an inbuilt function to check whether a string conforms to the JSON
> specification.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33491) Support json column validated
[
https://issues.apache.org/jira/browse/FLINK-33491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ouyangwulin updated FLINK-33491:
Description: Just like the {{is_valid_json}} function in PostgreSQL, it
would be useful to have an inbuilt function to check whether a string conforms
to the JSON specification.
> Support json column validated
> -
>
> Key: FLINK-33491
> URL: https://issues.apache.org/jira/browse/FLINK-33491
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / Runtime
>Affects Versions: 1.8.4, 1.9.4
>Reporter: ouyangwulin
>Priority: Minor
> Fix For: 1.8.4, 1.9.4
>
>
> Just like the {{is_valid_json}} function in PostgreSQL, it would be useful to
> have an inbuilt function to check whether a string conforms to the JSON
> specification.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-33491) Support json column validated
ouyangwulin created FLINK-33491: --- Summary: Support json column validated Key: FLINK-33491 URL: https://issues.apache.org/jira/browse/FLINK-33491 Project: Flink Issue Type: New Feature Components: Table SQL / Runtime Affects Versions: 1.8.4, 1.9.4 Reporter: ouyangwulin Fix For: 1.8.4, 1.9.4 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-27925) Avoid to create watcher without the resourceVersion
[ https://issues.apache.org/jira/browse/FLINK-27925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721745#comment-17721745 ] ouyangwulin commented on FLINK-27925: - [~huwh] tks, I will update. > Avoid to create watcher without the resourceVersion > --- > > Key: FLINK-27925 > URL: https://issues.apache.org/jira/browse/FLINK-27925 > Project: Flink > Issue Type: Improvement > Components: Deployment / Kubernetes >Reporter: Aitozi >Assignee: ouyangwulin >Priority: Major > Labels: pull-request-available > Attachments: image-2022-12-19-20-19-41-303.png > > > Currently, we create the watcher in KubernetesResourceManager. But it do not > pass the resourceVersion parameter, it will trigger a request to etcd. It > will bring the burden to the etcd in large scale cluster (which have been > seen in our internal k8s cluster). More detail can be found > [here|https://kubernetes.io/docs/reference/using-api/api-concepts/#the-resourceversion-parameter] > > I think we could use the informer to improve it (which will spawn a > list-watch and maintain the resourceVersion internally) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-29705) Document the least access with RBAC setting for native K8s integration
[ https://issues.apache.org/jira/browse/FLINK-29705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17701118#comment-17701118 ] ouyangwulin edited comment on FLINK-29705 at 3/16/23 11:49 AM: --- apiVersion: v1 kind: ServiceAccount metadata: labels: system: taskmanager-serviceaccount name: taskmanager-serviceaccount — apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: taskmanager-serviceaccount rules: - apiGroups: [""] resources: ["configmaps"] verbs: ["get", "list", "watch"] — apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: taskmanager-serviceaccount subjects: - kind: ServiceAccount name: taskmanager-serviceaccount roleRef: kind: Role name: taskmanager-serviceaccount apiGroup: rbac.authorization.k8s.io apiVersion: v1 kind: ServiceAccount metadata: labels: system: jobmanager-serviceaccount name: jobmanager-serviceaccount — apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: jobmanager-serviceaccount rules: - apiGroups: [""] resources: ["pods","configmaps"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: ["apps"] resources: ["deployments"] verbs: ["get", "list", "create", "update", "patch", "delete"] — apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: jobmanager-serviceaccount subjects: - kind: ServiceAccount name: jobmanager-serviceaccount roleRef: kind: Role name: jobmanager-serviceaccount apiGroup: rbac.authorization.k8s.io [~Wencong Liu] can you help review this rbac config ? was (Author: ouyangwuli): apiVersion: v1 kind: ServiceAccount metadata: labels: system: taskmanager-serviceaccount name: taskmanager-serviceaccount --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: taskmanager-serviceaccount rules: - apiGroups: [""] resources: ["configmaps"] verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: taskmanager-serviceaccount subjects: - kind: ServiceAccount name: taskmanager-serviceaccount roleRef: kind: Role name: taskmanager-serviceaccount apiGroup: rbac.authorization.k8s.io apiVersion: v1 kind: ServiceAccount metadata: labels: system: jobmanager-serviceaccount name: jobmanager-serviceaccount --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: jobmanager-serviceaccount rules: - apiGroups: [""] resources: ["pods","configmaps"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: ["apps"] resources: ["deployments"] verbs: ["get", "list", "create", "update", "patch", "delete"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: jobmanager-serviceaccount subjects: - kind: ServiceAccount name: jobmanager-serviceaccount roleRef: kind: Role name: jobmanager-serviceaccount apiGroup: rbac.authorization.k8s.io > Document the least access with RBAC setting for native K8s integration > -- > > Key: FLINK-29705 > URL: https://issues.apache.org/jira/browse/FLINK-29705 > Project: Flink > Issue Type: Improvement > Components: Deployment / Kubernetes, Documentation >Reporter: Yang Wang >Assignee: ouyangwulin >Priority: Major > > We should document the least access with RBAC settings[1]. And the operator > docs could be taken as a reference[2]. > > [1]. > [https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/resource-providers/native_kubernetes/#rbac] > [2]. > [https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/rbac/] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29705) Document the least access with RBAC setting for native K8s integration
[ https://issues.apache.org/jira/browse/FLINK-29705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17701118#comment-17701118 ] ouyangwulin commented on FLINK-29705: - apiVersion: v1 kind: ServiceAccount metadata: labels: system: taskmanager-serviceaccount name: taskmanager-serviceaccount --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: taskmanager-serviceaccount rules: - apiGroups: [""] resources: ["configmaps"] verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: taskmanager-serviceaccount subjects: - kind: ServiceAccount name: taskmanager-serviceaccount roleRef: kind: Role name: taskmanager-serviceaccount apiGroup: rbac.authorization.k8s.io apiVersion: v1 kind: ServiceAccount metadata: labels: system: jobmanager-serviceaccount name: jobmanager-serviceaccount --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: jobmanager-serviceaccount rules: - apiGroups: [""] resources: ["pods","configmaps"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: ["apps"] resources: ["deployments"] verbs: ["get", "list", "create", "update", "patch", "delete"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: jobmanager-serviceaccount subjects: - kind: ServiceAccount name: jobmanager-serviceaccount roleRef: kind: Role name: jobmanager-serviceaccount apiGroup: rbac.authorization.k8s.io > Document the least access with RBAC setting for native K8s integration > -- > > Key: FLINK-29705 > URL: https://issues.apache.org/jira/browse/FLINK-29705 > Project: Flink > Issue Type: Improvement > Components: Deployment / Kubernetes, Documentation >Reporter: Yang Wang >Assignee: ouyangwulin >Priority: Major > > We should document the least access with RBAC settings[1]. And the operator > docs could be taken as a reference[2]. > > [1]. > [https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/resource-providers/native_kubernetes/#rbac] > [2]. > [https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/rbac/] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30231) Update to Fabric8 Kubernetes Client to a version that has automatic renewal of service account tokens
[ https://issues.apache.org/jira/browse/FLINK-30231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17654420#comment-17654420 ] ouyangwulin commented on FLINK-30231: - [~martijnvisser] Can you help me check it out？ > Update to Fabric8 Kubernetes Client to a version that has automatic renewal > of service account tokens > - > > Key: FLINK-30231 > URL: https://issues.apache.org/jira/browse/FLINK-30231 > Project: Flink > Issue Type: Improvement > Components: Deployment / Kubernetes >Affects Versions: 1.15.2 > Environment: Kubernetes 1.23 environment provided by Amazon Web > Services managed Kubernetes service (EKS), using Flink 1.15.2. >Reporter: Chris Thomson >Priority: Minor > Labels: pull-request-available > > The Fabric8 Kubernetes Client library was updated to account for Kubernetes > configuration changes that result in service account tokens becoming bounded > in duration, needing to be renewed after an hour. The AWS managed Kubernetes > service (AWS EKS) currently has a configuration change that extends the one > hour bounded duration for the account to 90 days but this will eventually be > removed by AWS and produces warnings. > It appears that Fabric8 Kubernetes Client library version 5.12.4 is the > closest version to 5.12.3 that is currently in use by the Apache Flink > project to contain https://github.com/fabric8io/kubernetes-client/issues/2271. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29705) Document the least access with RBAC setting for native K8s integration
[ https://issues.apache.org/jira/browse/FLINK-29705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17654320#comment-17654320 ] ouyangwulin commented on FLINK-29705: - [~wangyang0918] Can assign this ticket for me ? > Document the least access with RBAC setting for native K8s integration > -- > > Key: FLINK-29705 > URL: https://issues.apache.org/jira/browse/FLINK-29705 > Project: Flink > Issue Type: Improvement > Components: Deployment / Kubernetes, Documentation >Reporter: Yang Wang >Priority: Major > > We should document the least access with RBAC settings[1]. And the operator > docs could be taken as a reference[2]. > > [1]. > [https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/resource-providers/native_kubernetes/#rbac] > [2]. > [https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/rbac/] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-27925) Avoid to create watcher without the resourceVersion
[ https://issues.apache.org/jira/browse/FLINK-27925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17649232#comment-17649232 ] ouyangwulin edited comment on FLINK-27925 at 12/19/22 12:31 PM: In the case of large-scale start and stop jobs, constantly reading data from etcd can cause a bottleneck in etcd performance. I agree with [~wangyang0918] using informer will increase memory pressure. We can increase resourceversion=0 in watcher to reduce data read from etcd. As [https://kubernetes.io/docs/reference/using-api/api-concepts/#the-resourceversion-parameter] describe and the screenshots of code， 1 ."resourceVersion unset" is means "Most Recent" ，The returned data must be consistent (in detail: served from etcd via a quorum read). 2. "resourceVersion="0" is means "Any". Return data at any resource version. The newest available resource version is preferred, but strong consistency is not required; data at any resource version may be served. It is possible for the request to return data at a much older resource version that the client has previously observed, particularly in high availability configurations, due to partitions or stale caches. !image-2022-12-19-20-19-41-303.png! was (Author: ouyangwuli): In the case of large-scale start and stop jobs, constantly reading data from etcd can cause a bottleneck in etcd performance. I agree with [~wangyang0918] using informer will increase memory pressure. We can increase resourceversion=0 in watch to reduce data read from etcd. > Avoid to create watcher without the resourceVersion > --- > > Key: FLINK-27925 > URL: https://issues.apache.org/jira/browse/FLINK-27925 > Project: Flink > Issue Type: Improvement > Components: Deployment / Kubernetes >Reporter: Aitozi >Priority: Major > Labels: pull-request-available > Attachments: image-2022-12-19-20-19-41-303.png > > > Currently, we create the watcher in KubernetesResourceManager. But it do not > pass the resourceVersion parameter, it will trigger a request to etcd. It > will bring the burden to the etcd in large scale cluster (which have been > seen in our internal k8s cluster). More detail can be found > [here|https://kubernetes.io/docs/reference/using-api/api-concepts/#the-resourceversion-parameter] > > I think we could use the informer to improve it (which will spawn a > list-watch and maintain the resourceVersion internally) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-27925) Avoid to create watcher without the resourceVersion
[ https://issues.apache.org/jira/browse/FLINK-27925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17649232#comment-17649232 ] ouyangwulin commented on FLINK-27925: - In the case of large-scale start and stop jobs, constantly reading data from etcd can cause a bottleneck in etcd performance. I agree with [~wangyang0918] using informer will increase memory pressure. We can increase resourceversion=0 in watch to reduce data read from etcd. > Avoid to create watcher without the resourceVersion > --- > > Key: FLINK-27925 > URL: https://issues.apache.org/jira/browse/FLINK-27925 > Project: Flink > Issue Type: Improvement > Components: Deployment / Kubernetes >Reporter: Aitozi >Priority: Major > Labels: pull-request-available > > Currently, we create the watcher in KubernetesResourceManager. But it do not > pass the resourceVersion parameter, it will trigger a request to etcd. It > will bring the burden to the etcd in large scale cluster (which have been > seen in our internal k8s cluster). More detail can be found > [here|https://kubernetes.io/docs/reference/using-api/api-concepts/#the-resourceversion-parameter] > > I think we could use the informer to improve it (which will spawn a > list-watch and maintain the resourceVersion internally) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] (FLINK-26541) SQL Client should support submitting SQL jobs in application mode
[ https://issues.apache.org/jira/browse/FLINK-26541 ] ouyangwulin deleted comment on FLINK-26541: - was (Author: ouyangwuli): Please assign the ticket to me [~wangyang0918] > SQL Client should support submitting SQL jobs in application mode > - > > Key: FLINK-26541 > URL: https://issues.apache.org/jira/browse/FLINK-26541 > Project: Flink > Issue Type: Improvement > Components: Deployment / Kubernetes, Deployment / YARN, Table SQL / > Client >Reporter: Jark Wu >Priority: Major > > Currently, the SQL Client only supports submitting jobs in session mode and > per-job mode. As the community going to drop the per-job mode (FLINK-26000), > SQL Client should support application mode as well. Otherwise, SQL Client can > only submit SQL in session mode then, but streaming jobs should be submitted > in per-job or application mode to have bettter resource isolation. > Disucssions: https://lists.apache.org/thread/2yq351nb721x23rz1q8qlyf2tqrk147r -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-26541) SQL Client should support submitting SQL jobs in application mode
[ https://issues.apache.org/jira/browse/FLINK-26541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17649023#comment-17649023 ] ouyangwulin commented on FLINK-26541: - Please assign the ticket to me [~wangyang0918] > SQL Client should support submitting SQL jobs in application mode > - > > Key: FLINK-26541 > URL: https://issues.apache.org/jira/browse/FLINK-26541 > Project: Flink > Issue Type: Improvement > Components: Deployment / Kubernetes, Deployment / YARN, Table SQL / > Client >Reporter: Jark Wu >Priority: Major > > Currently, the SQL Client only supports submitting jobs in session mode and > per-job mode. As the community going to drop the per-job mode (FLINK-26000), > SQL Client should support application mode as well. Otherwise, SQL Client can > only submit SQL in session mode then, but streaming jobs should be submitted > in per-job or application mode to have bettter resource isolation. > Disucssions: https://lists.apache.org/thread/2yq351nb721x23rz1q8qlyf2tqrk147r -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-24334) Configuration kubernetes.flink.log.dir not working
[
https://issues.apache.org/jira/browse/FLINK-24334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425441#comment-17425441
]
ouyangwulin edited comment on FLINK-24334 at 10/7/21, 9:44 AM:
---
[~wangyang0918] I think, Solving this problem can be divided into two steps:
1. Check whether'kubernetes.flink.log.dir' is configured. If configured, it
will be converted to '{{env.log.dir}}', but '{{env.log.dir}}' has priority.
2. The document states that the'kubernetes.flink.log. dir' parameter is a
deprecated parameter, and next version will cancelled.
was (Author: ouyangwuli):
Solving this problem can be divided into two steps:
1. Check whether'kubernetes.flink.log.dir' is configured. If configured, it
will be converted to '{{env.log.dir}}', but '{{env.log.dir}}' has priority.
2. The document states that the'kubernetes.flink.log. dir' parameter is a
deprecated parameter, and next version will cancelled.
> Configuration kubernetes.flink.log.dir not working
> --
>
> Key: FLINK-24334
> URL: https://issues.apache.org/jira/browse/FLINK-24334
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes
>Affects Versions: 1.14.0, 1.13.2
>Reporter: Yang Wang
>Priority: Major
>
> After FLINK-21128, {{kubernetes.flink.log.dir}} is useless and could be
> replaced with {{env.log.dir}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-24334) Configuration kubernetes.flink.log.dir not working
[
https://issues.apache.org/jira/browse/FLINK-24334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425441#comment-17425441
]
ouyangwulin commented on FLINK-24334:
-
Solving this problem can be divided into two steps:
1. Check whether'kubernetes.flink.log.dir' is configured. If configured, it
will be converted to '{{env.log.dir}}', but '{{env.log.dir}}' has priority.
2. The document states that the'kubernetes.flink.log. dir' parameter is a
deprecated parameter, and next version will cancelled.
> Configuration kubernetes.flink.log.dir not working
> --
>
> Key: FLINK-24334
> URL: https://issues.apache.org/jira/browse/FLINK-24334
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes
>Affects Versions: 1.14.0, 1.13.2
>Reporter: Yang Wang
>Priority: Major
>
> After FLINK-21128, {{kubernetes.flink.log.dir}} is useless and could be
> replaced with {{env.log.dir}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-24178) Flink on Kubernetes TaskManager
[ https://issues.apache.org/jira/browse/FLINK-24178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425392#comment-17425392 ] ouyangwulin commented on FLINK-24178: - [~frey] I guess, if the tm resource address error? Is the logs like "akka.tcp://flink@10.244.3.115:6123/user/rpc/resourcemanager_0(8c997dd4ad13ae3a4299c6e3025e49df)." appeared in TM(1-1)? > Flink on Kubernetes TaskManager > > > Key: FLINK-24178 > URL: https://issues.apache.org/jira/browse/FLINK-24178 > Project: Flink > Issue Type: Bug > Components: Client / Job Submission >Affects Versions: 1.13.2 > Environment: flink version :1.13.2 > kubernetes version : 1.19.3 >Reporter: frey >Priority: Blocker > Attachments: image-2021-09-07-13-31-10-077.png, > image-2021-09-07-13-31-40-796.png, image-2021-09-07-13-31-51-206.png, > image-2021-09-13-14-05-54-681.png, image-2021-09-13-14-06-05-433.png, > image-2021-09-13-14-14-11-384.png, jobmanager.log, > k8s-flink-session-message-01-taskmanager-1-2.log > > > > when submit a job on kubernetes in native session mode, > sometimes the TaskManager is created,but we can't find the TaskManager at all. > eg: > kubernetes is already created the TaskManager pod,and it's running > but flink can't find it > !image-2021-09-13-14-05-54-681.png! > > !image-2021-09-13-14-06-05-433.png! > !image-2021-09-13-14-14-11-384.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-24315) Cannot rebuild watcher thread while the K8S API server is unavailable
[ https://issues.apache.org/jira/browse/FLINK-24315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17416493#comment-17416493 ] ouyangwulin commented on FLINK-24315: - [~karmagyz] Can we configure parameters ‘kubernetes.watch.reconnectInterval’ and ‘kubernetes.watch.reconnectLimit’ to fix this problem ？ > Cannot rebuild watcher thread while the K8S API server is unavailable > - > > Key: FLINK-24315 > URL: https://issues.apache.org/jira/browse/FLINK-24315 > Project: Flink > Issue Type: Bug > Components: Deployment / Kubernetes >Affects Versions: 1.14.0, 1.13.2 >Reporter: ouyangwulin >Priority: Major > Fix For: 1.13.3, 1.14.1 > > > In native k8s integration, Flink will try to rebuild the watcher thread if > the API server is temporarily unavailable. However, if the jitter is longer > than the web socket timeout, the rebuilding of the watcher will timeout and > Flink cannot handle the pod event correctly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24315) Flink native on k8s wacther thread will down，when k8s api server not work or network timeout
ouyangwulin created FLINK-24315: --- Summary: Flink native on k8s wacther thread will down，when k8s api server not work or network timeout Key: FLINK-24315 URL: https://issues.apache.org/jira/browse/FLINK-24315 Project: Flink Issue Type: Bug Components: Deployment / Kubernetes Affects Versions: 1.13.2, 1.14.0, 1.14.1 Reporter: ouyangwulin Fix For: 1.14.0, 1.14.1, 1.13.2 Jobmanager use fabric-client to watch api-server.When k8s api-server or network problems. The watcher thread will closed , can use "jstack 1 && grep -i 'websocket'" to check the watcher thread is exists. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14986) Support to get detailed Kubernetes cluster description
[ https://issues.apache.org/jira/browse/FLINK-14986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187663#comment-17187663 ] ouyangwulin commented on FLINK-14986: - [~trohrmann] I have implement it, but Not have enough test. I will push it this week > Support to get detailed Kubernetes cluster description > -- > > Key: FLINK-14986 > URL: https://issues.apache.org/jira/browse/FLINK-14986 > Project: Flink > Issue Type: Sub-task > Components: Deployment / Kubernetes >Reporter: Yang Wang >Assignee: ouyangwulin >Priority: Major > > Currently Flink supports get yarn cluster description by > `YarnClusterDescriptor#getClusterDescription`. We should support the same > behavior in Kubernetes cluster. > Usually the cluster description contains the "total resources, available > resources, etc." -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-15641) Support to start init container
[ https://issues.apache.org/jira/browse/FLINK-15641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086760#comment-17086760 ] ouyangwulin edited comment on FLINK-15641 at 4/19/20, 5:38 AM: --- [~fly_in_gis]，in progressing. I will implement something in the doc before share it. was (Author: ouyangwuli): [~fly_in_gis]，in progressing. > Support to start init container > --- > > Key: FLINK-15641 > URL: https://issues.apache.org/jira/browse/FLINK-15641 > Project: Flink > Issue Type: New Feature > Components: Deployment / Kubernetes >Reporter: Yang Wang >Priority: Major > > >> Why do we need init container? > The init container could be used to prepare the use jars and dependencies. > Then we could always set the user image to Flink official image both for > standalone per-job on K8s or native K8s per-job. When the JobManager and > TaskManager container launched, the user jars will already exist there. I > think many users are running standalone per-job cluster in production by > using this way. > The init container only works for K8s cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15641) Support to start init container
[ https://issues.apache.org/jira/browse/FLINK-15641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086760#comment-17086760 ] ouyangwulin commented on FLINK-15641: - [~fly_in_gis]，in progressing. > Support to start init container > --- > > Key: FLINK-15641 > URL: https://issues.apache.org/jira/browse/FLINK-15641 > Project: Flink > Issue Type: New Feature > Components: Deployment / Kubernetes >Reporter: Yang Wang >Priority: Major > > >> Why do we need init container? > The init container could be used to prepare the use jars and dependencies. > Then we could always set the user image to Flink official image both for > standalone per-job on K8s or native K8s per-job. When the JobManager and > TaskManager container launched, the user jars will already exist there. I > think many users are running standalone per-job cluster in production by > using this way. > The init container only works for K8s cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-15656) Support user-specified pod templates
[ https://issues.apache.org/jira/browse/FLINK-15656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17078050#comment-17078050 ] ouyangwulin edited comment on FLINK-15656 at 4/8/20, 10:51 AM: --- I have implement init-container for download jar files in our project base on flink operator. But I have some problem to fixed. When i download from hdfs ,It exuast 40s, It waste too many times. was (Author: ouyangwuli): I have implement init-container for download jar files. But I have some problem to fixed. When i download from hdfs ,It exuast 40s, It waste too many times. > Support user-specified pod templates > > > Key: FLINK-15656 > URL: https://issues.apache.org/jira/browse/FLINK-15656 > Project: Flink > Issue Type: New Feature > Components: Deployment / Kubernetes >Reporter: Canbin Zheng >Priority: Major > > The current approach of introducing new configuration options for each aspect > of pod specification a user might wish is becoming unwieldy, we have to > maintain more and more Flink side Kubernetes configuration options and users > have to learn the gap between the declarative model used by Kubernetes and > the configuration model used by Flink. It's a great improvement to allow > users to specify pod templates as central places for all customization needs > for the jobmanager and taskmanager pods. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-15656) Support user-specified pod templates
[ https://issues.apache.org/jira/browse/FLINK-15656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17078050#comment-17078050 ] ouyangwulin edited comment on FLINK-15656 at 4/8/20, 10:51 AM: --- I have implement init-container for download jar files. But I have some problem to fixed. When i download from hdfs ,It exuast 40s, It waste too many times. was (Author: ouyangwuli): I have implement init-container for download jar files. But I have some problem to fixed. When i download from hdfs ,It exuast too many times. > Support user-specified pod templates > > > Key: FLINK-15656 > URL: https://issues.apache.org/jira/browse/FLINK-15656 > Project: Flink > Issue Type: New Feature > Components: Deployment / Kubernetes >Reporter: Canbin Zheng >Priority: Major > > The current approach of introducing new configuration options for each aspect > of pod specification a user might wish is becoming unwieldy, we have to > maintain more and more Flink side Kubernetes configuration options and users > have to learn the gap between the declarative model used by Kubernetes and > the configuration model used by Flink. It's a great improvement to allow > users to specify pod templates as central places for all customization needs > for the jobmanager and taskmanager pods. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15656) Support user-specified pod templates
[ https://issues.apache.org/jira/browse/FLINK-15656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17078050#comment-17078050 ] ouyangwulin commented on FLINK-15656: - I have implement init-container for download jar files. But I have some problem to fixed. When i download from hdfs ,It exuast too many times. > Support user-specified pod templates > > > Key: FLINK-15656 > URL: https://issues.apache.org/jira/browse/FLINK-15656 > Project: Flink > Issue Type: New Feature > Components: Deployment / Kubernetes >Reporter: Canbin Zheng >Priority: Major > > The current approach of introducing new configuration options for each aspect > of pod specification a user might wish is becoming unwieldy, we have to > maintain more and more Flink side Kubernetes configuration options and users > have to learn the gap between the declarative model used by Kubernetes and > the configuration model used by Flink. It's a great improvement to allow > users to specify pod templates as central places for all customization needs > for the jobmanager and taskmanager pods. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15656) Support user-specified pod templates
[ https://issues.apache.org/jira/browse/FLINK-15656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060856#comment-17060856 ] ouyangwulin commented on FLINK-15656: - Ok, If the pod templement support sidecar container and init container, I think it is a very good idea, and the feature of the sidecar container and init container is not need to do? > Support user-specified pod templates > > > Key: FLINK-15656 > URL: https://issues.apache.org/jira/browse/FLINK-15656 > Project: Flink > Issue Type: New Feature > Components: Deployment / Kubernetes >Reporter: Canbin Zheng >Priority: Major > > The current approach of introducing new configuration options for each aspect > of pod specification a user might wish is becoming unwieldy, we have to > maintain more and more Flink side Kubernetes configuration options and users > have to learn the gap between the declarative model used by Kubernetes and > the configuration model used by Flink. It's a great improvement to allow > users to specify pod templates as central places for all customization needs > for the jobmanager and taskmanager pods. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15871) Support to start sidecar container
[ https://issues.apache.org/jira/browse/FLINK-15871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060854#comment-17060854 ] ouyangwulin commented on FLINK-15871: - [~felixzheng] [~fly_in_gis] Sidecar for log is worked in our product env, It is a very good method. And [~felixzheng] Is the pod templemt supported init container? > Support to start sidecar container > -- > > Key: FLINK-15871 > URL: https://issues.apache.org/jira/browse/FLINK-15871 > Project: Flink > Issue Type: Sub-task > Components: Deployment / Kubernetes >Affects Versions: 1.10.0 >Reporter: Yang Wang >Priority: Major > Fix For: 1.11.0 > > > >> How does sidecar container work? > A sidecar container is running beside the Jobmanager and TaskManager > container. It could be used to collect log or debug some problems. For > example, when we configure the sidecar container to fluentd and share the > TaskManager log with volume, then it could be used to upload the logs to > HDFS, elastic search, etc. Also we could start a sidecar container with > debugging image which contains lots of tools and help to debug the network > problems. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-15641) Support to start init container
[ https://issues.apache.org/jira/browse/FLINK-15641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060654#comment-17060654 ] ouyangwulin edited comment on FLINK-15641 at 3/17/20, 6:00 AM: --- [~fly_in_gis] As flink on k8s ,not native. I think init container has another Scenario, that when before the tm start ,It use to probe the jm pod have started。 and flink native on k8s, the tm pod start after the jm pod started. So it don't need the probe. was (Author: ouyangwuli): [~fly_in_gis] As flink on k8s ,not native. I think init container has another Scenario, that when before the tm start ,It use to probe the jm pod have started。 > Support to start init container > --- > > Key: FLINK-15641 > URL: https://issues.apache.org/jira/browse/FLINK-15641 > Project: Flink > Issue Type: Sub-task > Components: Deployment / Kubernetes >Reporter: Yang Wang >Priority: Major > > >> Why do we need init container? > The init container could be used to prepare the use jars and dependencies. > Then we could always set the user image to Flink official image both for > standalone per-job on K8s or native K8s per-job. When the JobManager and > TaskManager container launched, the user jars will already exist there. I > think many users are running standalone per-job cluster in production by > using this way. > The init container only works for K8s cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15641) Support to start init container
[ https://issues.apache.org/jira/browse/FLINK-15641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060654#comment-17060654 ] ouyangwulin commented on FLINK-15641: - [~fly_in_gis] As flink on k8s ,not native. I think init container has another Scenario, that when before the tm start ,It use to probe the jm pod have started。 > Support to start init container > --- > > Key: FLINK-15641 > URL: https://issues.apache.org/jira/browse/FLINK-15641 > Project: Flink > Issue Type: Sub-task > Components: Deployment / Kubernetes >Reporter: Yang Wang >Priority: Major > > >> Why do we need init container? > The init container could be used to prepare the use jars and dependencies. > Then we could always set the user image to Flink official image both for > standalone per-job on K8s or native K8s per-job. When the JobManager and > TaskManager container launched, the user jars will already exist there. I > think many users are running standalone per-job cluster in production by > using this way. > The init container only works for K8s cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15871) Support to start sidecar container
[ https://issues.apache.org/jira/browse/FLINK-15871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060649#comment-17060649 ] ouyangwulin commented on FLINK-15871: - [~felixzheng] > As collect logs, I think have two method, one is use sidecar, the other is >use deamonset. But I don't think deamonset is a good idea for flink project, >Beacause It need user logs to the pods's console. So in our env, we still use >sidecar for collect log. we don't think sidecar is consume too many resources, >but it is more flexible. > Support to start sidecar container > -- > > Key: FLINK-15871 > URL: https://issues.apache.org/jira/browse/FLINK-15871 > Project: Flink > Issue Type: Sub-task > Components: Deployment / Kubernetes >Affects Versions: 1.10.0 >Reporter: Yang Wang >Priority: Major > Fix For: 1.11.0 > > > >> How does sidecar container work? > A sidecar container is running beside the Jobmanager and TaskManager > container. It could be used to collect log or debug some problems. For > example, when we configure the sidecar container to fluentd and share the > TaskManager log with volume, then it could be used to upload the logs to > HDFS, elastic search, etc. Also we could start a sidecar container with > debugging image which contains lots of tools and help to debug the network > problems. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15641) Support to start init container
[ https://issues.apache.org/jira/browse/FLINK-15641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060144#comment-17060144 ] ouyangwulin commented on FLINK-15641: - Ok I will complement it this week. > Support to start init container > --- > > Key: FLINK-15641 > URL: https://issues.apache.org/jira/browse/FLINK-15641 > Project: Flink > Issue Type: Sub-task > Components: Deployment / Kubernetes >Reporter: Yang Wang >Priority: Major > > >> Why do we need init container? > The init container could be used to prepare the use jars and dependencies. > Then we could always set the user image to Flink official image both for > standalone per-job on K8s or native K8s per-job. When the JobManager and > TaskManager container launched, the user jars will already exist there. I > think many users are running standalone per-job cluster in production by > using this way. > The init container only works for K8s cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14986) Support to get detailed Kubernetes cluster description
[ https://issues.apache.org/jira/browse/FLINK-14986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060136#comment-17060136 ] ouyangwulin commented on FLINK-14986: - Last time, I am busy in push flink on k8s online env, this time I will use more time in community. I will still do it. > Support to get detailed Kubernetes cluster description > -- > > Key: FLINK-14986 > URL: https://issues.apache.org/jira/browse/FLINK-14986 > Project: Flink > Issue Type: Sub-task > Components: Deployment / Kubernetes >Reporter: Yang Wang >Assignee: ouyangwulin >Priority: Major > > Currently Flink supports get yarn cluster description by > `YarnClusterDescriptor#getClusterDescription`. We should support the same > behavior in Kubernetes cluster. > Usually the cluster description contains the "total resources, available > resources, etc." -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15400) elasticsearch sink support dynamic index.
[ https://issues.apache.org/jira/browse/FLINK-15400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041547#comment-17041547 ] ouyangwulin commented on FLINK-15400: - [~Leonard Xu] If configed 'connector.dynamicIndex', ‘connector.indes’ is unused? > elasticsearch sink support dynamic index. > - > > Key: FLINK-15400 > URL: https://issues.apache.org/jira/browse/FLINK-15400 > Project: Flink > Issue Type: New Feature > Components: Connectors / ElasticSearch, Table SQL / Ecosystem >Affects Versions: 1.11.0 >Reporter: ouyangwulin >Priority: Major > Labels: usability > Fix For: 1.11.0 > > > From > user...@flink.apache.org([https://lists.apache.org/thread.html/ac4e0c068baeb3b070f0213a2e1314e6552b226b8132a4c49d667ecd%40%3Cuser-zh.flink.apache.org%3E]), > Becuase the es 6/7 not support ttl. so User need clean the index by > timestamp. Add dynamic index is a useful function. Add with properties > 'dynamicIndex' as a switch for open dynamicIndex. Add with properties > 'indexField' for the extract time field, Add properties 'indexInterval' for > change cycle mode. > > ||With property||discribe||default||Required|| > |dynamicIndex|Dynamic or not|false(true/false)|false| > |indexField|extract index field| none|dynamicIndex is true , then indexField > is required,only supported type "timestamp","date","long" | > |indexInterval|mode for cycle|d|ddynamicIndex is true , this field is > required ，the value optional is： > d：day > m：mouth > w：week| > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15400) elasticsearch sink support dynamic index.
[ https://issues.apache.org/jira/browse/FLINK-15400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041545#comment-17041545 ] ouyangwulin commented on FLINK-15400: - [~Leonard Xu]. Yes I have a design from blink. And I have a implement in our own branch in netease. And I want get more info before start implement . > elasticsearch sink support dynamic index. > - > > Key: FLINK-15400 > URL: https://issues.apache.org/jira/browse/FLINK-15400 > Project: Flink > Issue Type: New Feature > Components: Connectors / ElasticSearch, Table SQL / Ecosystem >Affects Versions: 1.11.0 >Reporter: ouyangwulin >Priority: Major > Labels: usability > Fix For: 1.11.0 > > > From > user...@flink.apache.org([https://lists.apache.org/thread.html/ac4e0c068baeb3b070f0213a2e1314e6552b226b8132a4c49d667ecd%40%3Cuser-zh.flink.apache.org%3E]), > Becuase the es 6/7 not support ttl. so User need clean the index by > timestamp. Add dynamic index is a useful function. Add with properties > 'dynamicIndex' as a switch for open dynamicIndex. Add with properties > 'indexField' for the extract time field, Add properties 'indexInterval' for > change cycle mode. > > ||With property||discribe||default||Required|| > |dynamicIndex|Dynamic or not|false(true/false)|false| > |indexField|extract index field| none|dynamicIndex is true , then indexField > is required,only supported type "timestamp","date","long" | > |indexInterval|mode for cycle|d|ddynamicIndex is true , this field is > required ，the value optional is： > d：day > m：mouth > w：week| > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15641) Support to start init container
[ https://issues.apache.org/jira/browse/FLINK-15641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17031314#comment-17031314 ] ouyangwulin commented on FLINK-15641: - [~fly_in_gis] ok， I will do this job > Support to start init container > --- > > Key: FLINK-15641 > URL: https://issues.apache.org/jira/browse/FLINK-15641 > Project: Flink > Issue Type: Sub-task > Components: Deployment / Kubernetes >Reporter: Yang Wang >Priority: Major > > >> Why do we need init container? > The init container could be used to prepare the use jars and dependencies. > Then we could always set the user image to Flink official image both for > standalone per-job on K8s or native K8s per-job. When the JobManager and > TaskManager container launched, the user jars will already exist there. I > think many users are running standalone per-job cluster in production by > using this way. > The init container only works for K8s cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15641) Support to start init container
[ https://issues.apache.org/jira/browse/FLINK-15641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029524#comment-17029524 ] ouyangwulin commented on FLINK-15641: - [~gjy] , I think I can do this work。 and I will test the feature in our k8s cluster at netease. > Support to start init container > --- > > Key: FLINK-15641 > URL: https://issues.apache.org/jira/browse/FLINK-15641 > Project: Flink > Issue Type: Sub-task > Components: Deployment / Kubernetes >Reporter: Yang Wang >Priority: Major > > >> Why do we need init container? > The init container could be used to prepare the use jars and dependencies. > Then we could always set the user image to Flink official image both for > standalone per-job on K8s or native K8s per-job. When the JobManager and > TaskManager container launched, the user jars will already exist there. I > think many users are running standalone per-job cluster in production by > using this way. > The init container only works for K8s cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15871) Support to start sidecar container
[ https://issues.apache.org/jira/browse/FLINK-15871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029522#comment-17029522 ] ouyangwulin commented on FLINK-15871: - Could assign this ticket for me? > Support to start sidecar container > -- > > Key: FLINK-15871 > URL: https://issues.apache.org/jira/browse/FLINK-15871 > Project: Flink > Issue Type: Sub-task > Components: Deployment / Kubernetes >Affects Versions: 1.10.0 >Reporter: Yang Wang >Priority: Major > Fix For: 1.11.0 > > > >> How does sidecar container work? > A sidecar container is running beside the Jobmanager and TaskManager > container. It could be used to collect log or debug some problems. For > example, when we configure the sidecar container to fluentd and share the > TaskManager log with volume, then it could be used to upload the logs to > HDFS, elastic search, etc. Also we could start a sidecar container with > debugging image which contains lots of tools and help to debug the network > problems. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15641) Support to start sidecar container and init container
[ https://issues.apache.org/jira/browse/FLINK-15641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029503#comment-17029503 ] ouyangwulin commented on FLINK-15641: - [~fly_in_gis] A sidecar container is running beside the TaskManager/Jobmanager container。Logs of Jobmanager should collect in the same way with logs of TaskManager. > Support to start sidecar container and init container > - > > Key: FLINK-15641 > URL: https://issues.apache.org/jira/browse/FLINK-15641 > Project: Flink > Issue Type: Sub-task > Components: Deployment / Kubernetes >Reporter: Yang Wang >Priority: Major > > * Sider Container > * Add a sidecar container of FlinkMaster and TaskManager to collector log to > shared storage(hdfs, elastic search, etc.). > * It could also be used for debugging purpose > * Init Container > * Use init container to download users jars dynamically or do something else > before start jobmanager and taskmanager. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14460) Active Kubernetes integration phase2 - Advanced Features
[ https://issues.apache.org/jira/browse/FLINK-14460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17018774#comment-17018774 ] ouyangwulin commented on FLINK-14460: - I have try to implement sidecars and initcontainer in flink operator, So can assign https://issues.apache.org/jira/browse/FLINK-15641 and https://issues.apache.org/jira/browse/FLINK-15642 for me? > Active Kubernetes integration phase2 - Advanced Features > > > Key: FLINK-14460 > URL: https://issues.apache.org/jira/browse/FLINK-14460 > Project: Flink > Issue Type: New Feature > Components: Deployment / Kubernetes >Reporter: Yang Wang >Priority: Major > > This is phase2 of active kubernetes integration. It is a umbrella jira to > track all the advanced features and make Flink on Kubernetes production ready. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15642) Support to set JobManager liveness check
[
https://issues.apache.org/jira/browse/FLINK-15642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17018771#comment-17018771
]
ouyangwulin commented on FLINK-15642:
-
Could assign this ticket for me?
> Support to set JobManager liveness check
>
>
> Key: FLINK-15642
> URL: https://issues.apache.org/jira/browse/FLINK-15642
> Project: Flink
> Issue Type: Sub-task
> Components: Deployment / Kubernetes
>Reporter: Yang Wang
>Priority: Major
>
> The liveness of TaskManager will be controlled by Flink Master. When it
> failed, timeout, a new pod will be started to replace. We need to add a
> liveness check for JobManager.
>
> It just like what we could do in the yaml.
> {code:java}
> ...
> livenessProbe:
> tcpSocket:
> port: 6123
> initialDelaySeconds: 30
> periodSeconds: 60
> ...{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15641) Support to start sidecar container and init container
[ https://issues.apache.org/jira/browse/FLINK-15641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17018770#comment-17018770 ] ouyangwulin commented on FLINK-15641: - Could assign this ticket for me? > Support to start sidecar container and init container > - > > Key: FLINK-15641 > URL: https://issues.apache.org/jira/browse/FLINK-15641 > Project: Flink > Issue Type: Sub-task > Components: Deployment / Kubernetes >Reporter: Yang Wang >Priority: Major > > * Sider Container > * Add a sidecar container of FlinkMaster and TaskManager to collector log to > shared storage(hdfs, elastic search, etc.). > * It could also be used for debugging purpose > * Init Container > * Use init container to download users jars dynamically or do something else > before start jobmanager and taskmanager. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-15378) StreamFileSystemSink supported mutil hdfs plugins.
[
https://issues.apache.org/jira/browse/FLINK-15378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006769#comment-17006769
]
ouyangwulin edited comment on FLINK-15378 at 1/3/20 2:49 AM:
-
[~pnowojski] thanks for your reply.
{noformat}
For one thing, the current approach in the proposed PR is not generic enough.
It limits the support for different configurations to just
StreamingFileSink.{noformat}
I think it can use for tm/jm , change tm/jm to get filesytem use
FileSystem.get(uri, identify).
{noformat}
If we allow to identify plugins by parts from the URI (for example host or port
as suggested by Yang Wang ), that would be better.{noformat}
I think use identify from the URI, It must change tm/jm to get filesystem
method. and another identify it seems more ‘flexible’, It can use URI as
specical identify
{noformat}
one plugin could be used by different file system instances.{noformat}
It's will good idea. I will read the code again, and try to find how to
implement it.
was (Author: ouyangwuli):
[~pnowojski] thanks for your reply.
{noformat}
For one thing, the current approach in the proposed PR is not generic enough.
It limits the support for different configurations to just
StreamingFileSink.{noformat}
I think it can use for tm/jm , change tm/jm to get filesytem use
FileSystem.get(uri, identify).
{noformat}
If we allow to identify plugins by parts from the URI (for example host or port
as suggested by Yang Wang ), that would be better.{noformat}
I think use identify from the URI, It must change tm/jm to get filesystem
method. and another identify it seems more ‘flexible’
{noformat}
one plugin could be used by different file system instances.{noformat}
It's will good idea. I will read the code again, and try to find how to
implement it.
> StreamFileSystemSink supported mutil hdfs plugins.
> --
>
> Key: FLINK-15378
> URL: https://issues.apache.org/jira/browse/FLINK-15378
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem, FileSystems
>Affects Versions: 1.9.2, 1.10.0
>Reporter: ouyangwulin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Attachments: jobmananger.log
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> [As report from
> maillist|[https://lists.apache.org/thread.html/7a6b1e341bde0ef632a82f8d46c9c93da358244b6bac0d8d544d11cb%40%3Cuser.flink.apache.org%3E]]
> Request 1: FileSystem plugins not effect the default yarn dependecies.
> Request 2: StreamFileSystemSink supported mutil hdfs plugins under the same
> schema
> As Problem describe :
> when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
> clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
> implements '*FileSystemFactory*', when jm start, It will call
> FileSystem.initialize(configuration,
> PluginUtils.createPluginManagerFromRootFolder(configuration)) to load
> factories to map FileSystem#**{color}FS_FACTORIES, and the key is only
> schema. When tm/jm use local hadoop conf A , the user code use hadoop conf
> Bin 'filesystem plugin', Conf A and Conf B is used to different hadoop
> cluster. and The Jm will start failed, beacuse of the blodserver in JM will
> load Conf B to get filesystem. the full log add appendix.
>
> AS reslove method:
> use schema and spec identify as key for ' FileSystem#**FS_FACTORIES '
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-15378) StreamFileSystemSink supported mutil hdfs plugins.
[
https://issues.apache.org/jira/browse/FLINK-15378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006769#comment-17006769
]
ouyangwulin edited comment on FLINK-15378 at 1/3/20 2:49 AM:
-
[~pnowojski] thanks for your reply.
{noformat}
For one thing, the current approach in the proposed PR is not generic enough.
It limits the support for different configurations to just
StreamingFileSink.{noformat}
I think it can use for tm/jm , change tm/jm to get filesytem use
FileSystem.get(uri, identify).
{noformat}
If we allow to identify plugins by parts from the URI (for example host or port
as suggested by Yang Wang ), that would be better.{noformat}
I think use identify from the URI, It must change tm/jm to get filesystem
method. and another identify it seems more ‘flexible’, It can use URI as
specical identify.as FileSystem.get(uri, uri.getAuthority()).
{noformat}
one plugin could be used by different file system instances.{noformat}
It's will good idea. I will read the code again, and try to find how to
implement it.
was (Author: ouyangwuli):
[~pnowojski] thanks for your reply.
{noformat}
For one thing, the current approach in the proposed PR is not generic enough.
It limits the support for different configurations to just
StreamingFileSink.{noformat}
I think it can use for tm/jm , change tm/jm to get filesytem use
FileSystem.get(uri, identify).
{noformat}
If we allow to identify plugins by parts from the URI (for example host or port
as suggested by Yang Wang ), that would be better.{noformat}
I think use identify from the URI, It must change tm/jm to get filesystem
method. and another identify it seems more ‘flexible’, It can use URI as
specical identify
{noformat}
one plugin could be used by different file system instances.{noformat}
It's will good idea. I will read the code again, and try to find how to
implement it.
> StreamFileSystemSink supported mutil hdfs plugins.
> --
>
> Key: FLINK-15378
> URL: https://issues.apache.org/jira/browse/FLINK-15378
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem, FileSystems
>Affects Versions: 1.9.2, 1.10.0
>Reporter: ouyangwulin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Attachments: jobmananger.log
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> [As report from
> maillist|[https://lists.apache.org/thread.html/7a6b1e341bde0ef632a82f8d46c9c93da358244b6bac0d8d544d11cb%40%3Cuser.flink.apache.org%3E]]
> Request 1: FileSystem plugins not effect the default yarn dependecies.
> Request 2: StreamFileSystemSink supported mutil hdfs plugins under the same
> schema
> As Problem describe :
> when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
> clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
> implements '*FileSystemFactory*', when jm start, It will call
> FileSystem.initialize(configuration,
> PluginUtils.createPluginManagerFromRootFolder(configuration)) to load
> factories to map FileSystem#**{color}FS_FACTORIES, and the key is only
> schema. When tm/jm use local hadoop conf A , the user code use hadoop conf
> Bin 'filesystem plugin', Conf A and Conf B is used to different hadoop
> cluster. and The Jm will start failed, beacuse of the blodserver in JM will
> load Conf B to get filesystem. the full log add appendix.
>
> AS reslove method:
> use schema and spec identify as key for ' FileSystem#**FS_FACTORIES '
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-12382) HA + ResourceManager exception: Fencing token not set
[
https://issues.apache.org/jira/browse/FLINK-12382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17007184#comment-17007184
]
ouyangwulin commented on FLINK-12382:
-
The program finished with the following exception:
org.apache.flink.client.program.ProgramInvocationException: Could not retrieve
the execution result. (JobID: c71c27d2fdfdc7d75686e820acae742c)
at
org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:261)
at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:483)
at
org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:66)
at
org.apache.flink.streaming.examples.windowing.TopSpeedWindowing.main(TopSpeedWindowing.java:96)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:423)
at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:813)
at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:287)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
at
org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)
at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
at
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)
Caused by: org.apache.flink.runtime.rest.util.RestClientException: [Internal
server error., ]
at org.apache.flink.runtime.rest.RestClient.parseResponse(RestClient.java:389)
at
org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$3(RestClient.java:373)
at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:966)
at
java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:940)
at
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
> HA + ResourceManager exception: Fencing token not set
> -
>
> Key: FLINK-12382
> URL: https://issues.apache.org/jira/browse/FLINK-12382
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.8.0
> Environment: Same all all previous bugs filed by myself, today, but
> this time with HA with zetcd.
>Reporter: Henrik
>Priority: Major
>
> I'm testing zetcd + session jobs in k8s, and testing what happens when I kill
> both the job-cluster and task-manager at the same time, but maintain ZK/zetcd
> up and running.
> Then I get this stacktrace, that's completely non-actionable for me, and also
> resolves itself. I expect a number of retries, and if this exception is part
> of the protocol signalling to retry, then it should not be printed as a log
> entry.
> This might be related to an older bug:
> [https://jira.apache.org/jira/browse/FLINK-7734]
> {code:java}
> [tm] 2019-05-01 11:32:01,641 ERROR
> org.apache.flink.runtime.taskexecutor.TaskExecutor - Registration
> at ResourceManager failed due to an error
> [tm] java.util.concurrent.CompletionException:
> org.apache.flink.runtime.rpc.exceptions.FencingTokenException: Fencing token
> not set: Ignoring message
> RemoteFencedMessage(b8fb92460bdd5c250de1415e6cf04d4e,
> RemoteRpcInvocation(registerTaskExecutor(String, ResourceID, int,
> HardwareDescription, Time))) sent to
> akka.tcp://flink@analytics-job:6123/user/resourcemanager because the fencing
> token is null.
> [tm] at
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
> [tm] at
> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
> [tm] at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
> [tm] at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> [tm] at
> java.util.concurrent.CompletableFuture.
[jira] [Commented] (FLINK-12382) HA + ResourceManager exception: Fencing token not set
[
https://issues.apache.org/jira/browse/FLINK-12382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17007183#comment-17007183
]
ouyangwulin commented on FLINK-12382:
-
I have try zookeeper， But the error log seems same
> HA + ResourceManager exception: Fencing token not set
> -
>
> Key: FLINK-12382
> URL: https://issues.apache.org/jira/browse/FLINK-12382
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.8.0
> Environment: Same all all previous bugs filed by myself, today, but
> this time with HA with zetcd.
>Reporter: Henrik
>Priority: Major
>
> I'm testing zetcd + session jobs in k8s, and testing what happens when I kill
> both the job-cluster and task-manager at the same time, but maintain ZK/zetcd
> up and running.
> Then I get this stacktrace, that's completely non-actionable for me, and also
> resolves itself. I expect a number of retries, and if this exception is part
> of the protocol signalling to retry, then it should not be printed as a log
> entry.
> This might be related to an older bug:
> [https://jira.apache.org/jira/browse/FLINK-7734]
> {code:java}
> [tm] 2019-05-01 11:32:01,641 ERROR
> org.apache.flink.runtime.taskexecutor.TaskExecutor - Registration
> at ResourceManager failed due to an error
> [tm] java.util.concurrent.CompletionException:
> org.apache.flink.runtime.rpc.exceptions.FencingTokenException: Fencing token
> not set: Ignoring message
> RemoteFencedMessage(b8fb92460bdd5c250de1415e6cf04d4e,
> RemoteRpcInvocation(registerTaskExecutor(String, ResourceID, int,
> HardwareDescription, Time))) sent to
> akka.tcp://flink@analytics-job:6123/user/resourcemanager because the fencing
> token is null.
> [tm] at
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
> [tm] at
> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
> [tm] at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
> [tm] at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> [tm] at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> [tm] at
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
> [tm] at
> org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:815)
> [tm] at akka.dispatch.OnComplete.internal(Future.scala:258)
> [tm] at akka.dispatch.OnComplete.internal(Future.scala:256)
> [tm] at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)
> [tm] at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)
> [tm] at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
> [tm] at
> org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:74)
> [tm] at
> scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
> [tm] at
> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
> [tm] at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:534)
> [tm] at akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:97)
> [tm] at
> akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:982)
> [tm] at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
> [tm] at akka.remote.EndpointActor.aroundReceive(Endpoint.scala:446)
> [tm] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
> [tm] at akka.actor.ActorCell.invoke(ActorCell.scala:495)
> [tm] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
> [tm] at akka.dispatch.Mailbox.run(Mailbox.scala:224)
> [tm] at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
> [tm] at
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> [tm] at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> [tm] at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> [tm] at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> [tm] Caused by:
> org.apache.flink.runtime.rpc.exceptions.FencingTokenException: Fencing token
> not set: Ignoring message
> RemoteFencedMessage(b8fb92460bdd5c250de1415e6cf04d4e,
> RemoteRpcInvocation(registerTaskExecutor(String, ResourceID, int,
> HardwareDescription, Time))) sent to
> akka.tcp://flink@analytics-job:6123/user/resourcemanager because the fencing
> token is null.
> [tm] at
> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:63)
> [tm] at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:147)
> [tm] at
[jira] [Comment Edited] (FLINK-15378) StreamFileSystemSink supported mutil hdfs plugins.
[
https://issues.apache.org/jira/browse/FLINK-15378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006769#comment-17006769
]
ouyangwulin edited comment on FLINK-15378 at 1/2/20 12:36 PM:
--
[~pnowojski] thanks for your reply.
{noformat}
For one thing, the current approach in the proposed PR is not generic enough.
It limits the support for different configurations to just
StreamingFileSink.{noformat}
I think it can use for tm/jm , change tm/jm to get filesytem use
FileSystem.get(uri, identify).
{noformat}
If we allow to identify plugins by parts from the URI (for example host or port
as suggested by Yang Wang ), that would be better.{noformat}
I think use identify from the URI, It must change tm/jm to get filesystem
method. and another identify it seems more ‘flexible’
{noformat}
one plugin could be used by different file system instances.{noformat}
It's will good idea. I will read the code again, and try to find how to
implement it.
was (Author: ouyangwuli):
[~pnowojski] thanks for your reply.
{noformat}
For one thing, the current approach in the proposed PR is not generic enough.
It limits the support for different configurations to just
StreamingFileSink.{noformat}
{color:#172b4d}{color}I think it can use for tm/jm , change tm/jm to get
filesytem use FileSystem.get(uri, identify).
{noformat}
If we allow to identify plugins by parts from the URI (for example host or port
as suggested by Yang Wang ), that would be better.{noformat}
I think use identify from the URI, It must change tm/jm to filesystem method.
and another identify it seems more ‘flexible’
{noformat}
one plugin could be used by different file system instances.{noformat}
It's will good idea. I will read the code again, and try to find how to
implement it.
> StreamFileSystemSink supported mutil hdfs plugins.
> --
>
> Key: FLINK-15378
> URL: https://issues.apache.org/jira/browse/FLINK-15378
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem, FileSystems
>Affects Versions: 1.9.2, 1.10.0
>Reporter: ouyangwulin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Attachments: jobmananger.log
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> [As report from
> maillist|[https://lists.apache.org/thread.html/7a6b1e341bde0ef632a82f8d46c9c93da358244b6bac0d8d544d11cb%40%3Cuser.flink.apache.org%3E]]
> Request 1: FileSystem plugins not effect the default yarn dependecies.
> Request 2: StreamFileSystemSink supported mutil hdfs plugins under the same
> schema
> As Problem describe :
> when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
> clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
> implements '*FileSystemFactory*', when jm start, It will call
> FileSystem.initialize(configuration,
> PluginUtils.createPluginManagerFromRootFolder(configuration)) to load
> factories to map FileSystem#**{color}FS_FACTORIES, and the key is only
> schema. When tm/jm use local hadoop conf A , the user code use hadoop conf
> Bin 'filesystem plugin', Conf A and Conf B is used to different hadoop
> cluster. and The Jm will start failed, beacuse of the blodserver in JM will
> load Conf B to get filesystem. the full log add appendix.
>
> AS reslove method:
> use schema and spec identify as key for ' FileSystem#**FS_FACTORIES '
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15378) StreamFileSystemSink supported mutil hdfs plugins.
[
https://issues.apache.org/jira/browse/FLINK-15378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006769#comment-17006769
]
ouyangwulin commented on FLINK-15378:
-
[~pnowojski] thanks for your reply.
{noformat}
For one thing, the current approach in the proposed PR is not generic enough.
It limits the support for different configurations to just
StreamingFileSink.{noformat}
{color:#172b4d}{color}I think it can use for tm/jm , change tm/jm to get
filesytem use FileSystem.get(uri, identify).
{noformat}
If we allow to identify plugins by parts from the URI (for example host or port
as suggested by Yang Wang ), that would be better.{noformat}
I think use identify from the URI, It must change tm/jm to filesystem method.
and another identify it seems more ‘flexible’
{noformat}
one plugin could be used by different file system instances.{noformat}
It's will good idea. I will read the code again, and try to find how to
implement it.
> StreamFileSystemSink supported mutil hdfs plugins.
> --
>
> Key: FLINK-15378
> URL: https://issues.apache.org/jira/browse/FLINK-15378
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem, FileSystems
>Affects Versions: 1.9.2, 1.10.0
>Reporter: ouyangwulin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Attachments: jobmananger.log
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> [As report from
> maillist|[https://lists.apache.org/thread.html/7a6b1e341bde0ef632a82f8d46c9c93da358244b6bac0d8d544d11cb%40%3Cuser.flink.apache.org%3E]]
> Request 1: FileSystem plugins not effect the default yarn dependecies.
> Request 2: StreamFileSystemSink supported mutil hdfs plugins under the same
> schema
> As Problem describe :
> when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
> clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
> implements '*FileSystemFactory*', when jm start, It will call
> FileSystem.initialize(configuration,
> PluginUtils.createPluginManagerFromRootFolder(configuration)) to load
> factories to map FileSystem#**{color}FS_FACTORIES, and the key is only
> schema. When tm/jm use local hadoop conf A , the user code use hadoop conf
> Bin 'filesystem plugin', Conf A and Conf B is used to different hadoop
> cluster. and The Jm will start failed, beacuse of the blodserver in JM will
> load Conf B to get filesystem. the full log add appendix.
>
> AS reslove method:
> use schema and spec identify as key for ' FileSystem#**FS_FACTORIES '
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15378) StreamFileSystemSink supported mutil hdfs plugins.
[
https://issues.apache.org/jira/browse/FLINK-15378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006568#comment-17006568
]
ouyangwulin commented on FLINK-15378:
-
[~fly_in_gis] Also， I need different kerberos auth between the a hdfs and b
hdfs.
> StreamFileSystemSink supported mutil hdfs plugins.
> --
>
> Key: FLINK-15378
> URL: https://issues.apache.org/jira/browse/FLINK-15378
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem, FileSystems
>Affects Versions: 1.9.2, 1.10.0
>Reporter: ouyangwulin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Attachments: jobmananger.log
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> [As report from
> maillist|[https://lists.apache.org/thread.html/7a6b1e341bde0ef632a82f8d46c9c93da358244b6bac0d8d544d11cb%40%3Cuser.flink.apache.org%3E]]
> Request 1: FileSystem plugins not effect the default yarn dependecies.
> Request 2: StreamFileSystemSink supported mutil hdfs plugins under the same
> schema
> As Problem describe :
> when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
> clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
> implements '*FileSystemFactory*', when jm start, It will call
> FileSystem.initialize(configuration,
> PluginUtils.createPluginManagerFromRootFolder(configuration)) to load
> factories to map FileSystem#**{color}FS_FACTORIES, and the key is only
> schema. When tm/jm use local hadoop conf A , the user code use hadoop conf
> Bin 'filesystem plugin', Conf A and Conf B is used to different hadoop
> cluster. and The Jm will start failed, beacuse of the blodserver in JM will
> load Conf B to get filesystem. the full log add appendix.
>
> AS reslove method:
> use schema and spec identify as key for ' FileSystem#**FS_FACTORIES '
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-15378) StreamFileSystemSink supported mutil hdfs plugins.
[
https://issues.apache.org/jira/browse/FLINK-15378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006557#comment-17006557
]
ouyangwulin edited comment on FLINK-15378 at 1/2/20 3:27 AM:
-
{code:java}
do I understand your problem correctly, that you are trying to use the same
plugin, but with different configs?{code}
yes , but the same plugin how to use different configs when it only create
one FilesystemFactory.
{code:java}
Can not you create a separate plugin but just with a different schema, instead
of adding different identity? {code}
I want to sink to different hdfs cluster. Schema is native same in different
hdfs cluster. So I want add a identify to get the different.
{code:java}
where are the "conf A", "conf B" and hdfs-site.xml files located? Are they
bundled inside the plugin's fat jar? {code}
yes. It can from the fat jar. or From flink config
was (Author: ouyangwuli):
{code:java}
do I understand your problem correctly, that you are trying to use the same
plugin, but with different configs?{code}
yes , but the same plugin how to use different configs when it only create
one FilesystemFactory.
{code:java}
Can not you create a separate plugin but just with a different schema, instead
of adding different identity? {code}
I want to sink to different hdfs cluster. Schema is native same in different
cluster. So I want add a identify to get the different.
{code:java}
where are the "conf A", "conf B" and hdfs-site.xml files located? Are they
bundled inside the plugin's fat jar? {code}
yes. It can from the fat jar. or From flink config
> StreamFileSystemSink supported mutil hdfs plugins.
> --
>
> Key: FLINK-15378
> URL: https://issues.apache.org/jira/browse/FLINK-15378
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem, FileSystems
>Affects Versions: 1.9.2, 1.10.0
>Reporter: ouyangwulin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Attachments: jobmananger.log
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> [As report from
> maillist|[https://lists.apache.org/thread.html/7a6b1e341bde0ef632a82f8d46c9c93da358244b6bac0d8d544d11cb%40%3Cuser.flink.apache.org%3E]]
> Request 1: FileSystem plugins not effect the default yarn dependecies.
> Request 2: StreamFileSystemSink supported mutil hdfs plugins under the same
> schema
> As Problem describe :
> when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
> clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
> implements '*FileSystemFactory*', when jm start, It will call
> FileSystem.initialize(configuration,
> PluginUtils.createPluginManagerFromRootFolder(configuration)) to load
> factories to map FileSystem#**{color}FS_FACTORIES, and the key is only
> schema. When tm/jm use local hadoop conf A , the user code use hadoop conf
> Bin 'filesystem plugin', Conf A and Conf B is used to different hadoop
> cluster. and The Jm will start failed, beacuse of the blodserver in JM will
> load Conf B to get filesystem. the full log add appendix.
>
> AS reslove method:
> use schema and spec identify as key for ' FileSystem#**FS_FACTORIES '
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-15378) StreamFileSystemSink supported mutil hdfs plugins.
[
https://issues.apache.org/jira/browse/FLINK-15378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006557#comment-17006557
]
ouyangwulin edited comment on FLINK-15378 at 1/2/20 3:25 AM:
-
{code:java}
do I understand your problem correctly, that you are trying to use the same
plugin, but with different configs?{code}
yes , but the same plugin how to use different configs when it only create
one FilesystemFactory.
{code:java}
Can not you create a separate plugin but just with a different schema, instead
of adding different identity? {code}
I want to sink to different hdfs cluster. Schema is native same in different
cluster. So I want add a identify to get the different.
{code:java}
where are the "conf A", "conf B" and hdfs-site.xml files located? Are they
bundled inside the plugin's fat jar? {code}
yes. It can from the fat jar. or From flink config
was (Author: ouyangwuli):
{code:java}
do I understand your problem correctly, that you are trying to use the same
plugin, but with different configs?{code}
yes , but the same plugin how to use different configs when it only create
one FilesystemFactory.
{code:java}
Can not you create a separate plugin but just with a different schema, instead
of adding different identity? {code}
I want to sink to different hdfs cluster. Schema is native same in different
cluster. So I want add a identify to get the different.
{code:java}
where are the "conf A", "conf B" and hdfs-site.xml files located? Are they
bundled inside the plugin's fat jar? {code}
yes.
> StreamFileSystemSink supported mutil hdfs plugins.
> --
>
> Key: FLINK-15378
> URL: https://issues.apache.org/jira/browse/FLINK-15378
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem, FileSystems
>Affects Versions: 1.9.2, 1.10.0
>Reporter: ouyangwulin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Attachments: jobmananger.log
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> [As report from
> maillist|[https://lists.apache.org/thread.html/7a6b1e341bde0ef632a82f8d46c9c93da358244b6bac0d8d544d11cb%40%3Cuser.flink.apache.org%3E]]
> Request 1: FileSystem plugins not effect the default yarn dependecies.
> Request 2: StreamFileSystemSink supported mutil hdfs plugins under the same
> schema
> As Problem describe :
> when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
> clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
> implements '*FileSystemFactory*', when jm start, It will call
> FileSystem.initialize(configuration,
> PluginUtils.createPluginManagerFromRootFolder(configuration)) to load
> factories to map FileSystem#**{color}FS_FACTORIES, and the key is only
> schema. When tm/jm use local hadoop conf A , the user code use hadoop conf
> Bin 'filesystem plugin', Conf A and Conf B is used to different hadoop
> cluster. and The Jm will start failed, beacuse of the blodserver in JM will
> load Conf B to get filesystem. the full log add appendix.
>
> AS reslove method:
> use schema and spec identify as key for ' FileSystem#**FS_FACTORIES '
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15378) StreamFileSystemSink supported mutil hdfs plugins.
[
https://issues.apache.org/jira/browse/FLINK-15378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006557#comment-17006557
]
ouyangwulin commented on FLINK-15378:
-
{code:java}
do I understand your problem correctly, that you are trying to use the same
plugin, but with different configs?{code}
yes , but the same plugin how to use different configs when it only create
one FilesystemFactory.
{code:java}
Can not you create a separate plugin but just with a different schema, instead
of adding different identity? {code}
I want to sink to different hdfs cluster. Schema is native same in different
cluster. So I want add a identify to get the different.
{code:java}
where are the "conf A", "conf B" and hdfs-site.xml files located? Are they
bundled inside the plugin's fat jar? {code}
yes.
> StreamFileSystemSink supported mutil hdfs plugins.
> --
>
> Key: FLINK-15378
> URL: https://issues.apache.org/jira/browse/FLINK-15378
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem, FileSystems
>Affects Versions: 1.9.2, 1.10.0
>Reporter: ouyangwulin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Attachments: jobmananger.log
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> [As report from
> maillist|[https://lists.apache.org/thread.html/7a6b1e341bde0ef632a82f8d46c9c93da358244b6bac0d8d544d11cb%40%3Cuser.flink.apache.org%3E]]
> Request 1: FileSystem plugins not effect the default yarn dependecies.
> Request 2: StreamFileSystemSink supported mutil hdfs plugins under the same
> schema
> As Problem describe :
> when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
> clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
> implements '*FileSystemFactory*', when jm start, It will call
> FileSystem.initialize(configuration,
> PluginUtils.createPluginManagerFromRootFolder(configuration)) to load
> factories to map FileSystem#**{color}FS_FACTORIES, and the key is only
> schema. When tm/jm use local hadoop conf A , the user code use hadoop conf
> Bin 'filesystem plugin', Conf A and Conf B is used to different hadoop
> cluster. and The Jm will start failed, beacuse of the blodserver in JM will
> load Conf B to get filesystem. the full log add appendix.
>
> AS reslove method:
> use schema and spec identify as key for ' FileSystem#**FS_FACTORIES '
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-15378) StreamFileSystemSink supported mutil hdfs plugins.
[
https://issues.apache.org/jira/browse/FLINK-15378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17004032#comment-17004032
]
ouyangwulin edited comment on FLINK-15378 at 12/27/19 9:52 AM:
---
[~fly_in_gis] https://issues.apache.org/jira/browse/FLINK-15355 This issue will
fixed the plugin problem.
was (Author: ouyangwuli):
[~fly_in_gis] https://issues.apache.org/jira/browse/FLINK-15355 This issue is
fixed the plugin problem.
> StreamFileSystemSink supported mutil hdfs plugins.
> --
>
> Key: FLINK-15378
> URL: https://issues.apache.org/jira/browse/FLINK-15378
> Project: Flink
> Issue Type: Improvement
> Components: API / Core
>Affects Versions: 1.9.2, 1.11.0
>Reporter: ouyangwulin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Attachments: jobmananger.log
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> [As report from
> maillist|[https://lists.apache.org/thread.html/7a6b1e341bde0ef632a82f8d46c9c93da358244b6bac0d8d544d11cb%40%3Cuser.flink.apache.org%3E]]
> Request 1: FileSystem plugins not effect the default yarn dependecies.
> Request 2: StreamFileSystemSink supported mutil hdfs plugins under the same
> schema
> As Problem describe :
> when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
> clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
> implements '*FileSystemFactory*', when jm start, It will call
> FileSystem.initialize(configuration,
> PluginUtils.createPluginManagerFromRootFolder(configuration)) to load
> factories to map FileSystem#**{color}FS_FACTORIES, and the key is only
> schema. When tm/jm use local hadoop conf A , the user code use hadoop conf
> Bin 'filesystem plugin', Conf A and Conf B is used to different hadoop
> cluster. and The Jm will start failed, beacuse of the blodserver in JM will
> load Conf B to get filesystem. the full log add appendix.
>
> AS reslove method:
> use schema and spec identify as key for ' FileSystem#**FS_FACTORIES '
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-15378) StreamFileSystemSink supported mutil hdfs plugins.
[
https://issues.apache.org/jira/browse/FLINK-15378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17003237#comment-17003237
]
ouyangwulin edited comment on FLINK-15378 at 12/27/19 9:51 AM:
---
[~fly_in_gis]
>> For request 1,
Change plugin to not use '{{classloader.parent-first-patterns.default}} '
is a good idea。
>> For request 2,
Aggregated hdfs-site.xml can support multiple hdfs cluster in same
kerberoes, It create one filesystem instantiate to write to diff hdfs
cluster。But in our sense，We have one hdfs cluster with kerberoes， Other cluster
isn't need kerberoes，So Aggregated hdfs-site.xml is not work 。It also not need
plugin in `$FLINK_HOME/pulgins` ，The user jar can content FileSystem plugin。 I
think I don't interrupt the current plugin mechine.
was (Author: ouyangwuli):
[~fly_in_gis]
>> For request 1,
Change plugin to not use '{{classloader.parent-first-patterns.default}} '
is a good idea。
>> For request 2,
Aggregated hdfs-site.xml can support multiple hdfs cluster in same
kerberoes, It create one filesystem instantiate to write to diff hdfs
cluster。But in our sense，We have one hdfs cluster with kerberoes， Other cluster
isn't need kerberoes，So Aggregated hdfs-site.xml is not work 。It also not need
plugin in `$FLINK_HOME/pulgins` ，The user jar can content FileSystem plugin。
> StreamFileSystemSink supported mutil hdfs plugins.
> --
>
> Key: FLINK-15378
> URL: https://issues.apache.org/jira/browse/FLINK-15378
> Project: Flink
> Issue Type: Improvement
> Components: API / Core
>Affects Versions: 1.9.2, 1.11.0
>Reporter: ouyangwulin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Attachments: jobmananger.log
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> [As report from
> maillist|[https://lists.apache.org/thread.html/7a6b1e341bde0ef632a82f8d46c9c93da358244b6bac0d8d544d11cb%40%3Cuser.flink.apache.org%3E]]
> Request 1: FileSystem plugins not effect the default yarn dependecies.
> Request 2: StreamFileSystemSink supported mutil hdfs plugins under the same
> schema
> As Problem describe :
> when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
> clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
> implements '*FileSystemFactory*', when jm start, It will call
> FileSystem.initialize(configuration,
> PluginUtils.createPluginManagerFromRootFolder(configuration)) to load
> factories to map FileSystem#**{color}FS_FACTORIES, and the key is only
> schema. When tm/jm use local hadoop conf A , the user code use hadoop conf
> Bin 'filesystem plugin', Conf A and Conf B is used to different hadoop
> cluster. and The Jm will start failed, beacuse of the blodserver in JM will
> load Conf B to get filesystem. the full log add appendix.
>
> AS reslove method:
> use schema and spec identify as key for ' FileSystem#**FS_FACTORIES '
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15378) StreamFileSystemSink supported mutil hdfs plugins.
[
https://issues.apache.org/jira/browse/FLINK-15378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17004032#comment-17004032
]
ouyangwulin commented on FLINK-15378:
-
[~fly_in_gis] https://issues.apache.org/jira/browse/FLINK-15355 This issue is
fixed the plugin problem.
> StreamFileSystemSink supported mutil hdfs plugins.
> --
>
> Key: FLINK-15378
> URL: https://issues.apache.org/jira/browse/FLINK-15378
> Project: Flink
> Issue Type: Improvement
> Components: API / Core
>Affects Versions: 1.9.2, 1.11.0
>Reporter: ouyangwulin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Attachments: jobmananger.log
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> [As report from
> maillist|[https://lists.apache.org/thread.html/7a6b1e341bde0ef632a82f8d46c9c93da358244b6bac0d8d544d11cb%40%3Cuser.flink.apache.org%3E]]
> Request 1: FileSystem plugins not effect the default yarn dependecies.
> Request 2: StreamFileSystemSink supported mutil hdfs plugins under the same
> schema
> As Problem describe :
> when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
> clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
> implements '*FileSystemFactory*', when jm start, It will call
> FileSystem.initialize(configuration,
> PluginUtils.createPluginManagerFromRootFolder(configuration)) to load
> factories to map FileSystem#**{color}FS_FACTORIES, and the key is only
> schema. When tm/jm use local hadoop conf A , the user code use hadoop conf
> Bin 'filesystem plugin', Conf A and Conf B is used to different hadoop
> cluster. and The Jm will start failed, beacuse of the blodserver in JM will
> load Conf B to get filesystem. the full log add appendix.
>
> AS reslove method:
> use schema and spec identify as key for ' FileSystem#**FS_FACTORIES '
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15400) elasticsearch sink support dynamic index.
[ https://issues.apache.org/jira/browse/FLINK-15400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17003531#comment-17003531 ] ouyangwulin commented on FLINK-15400: - Please assign this for me > elasticsearch sink support dynamic index. > - > > Key: FLINK-15400 > URL: https://issues.apache.org/jira/browse/FLINK-15400 > Project: Flink > Issue Type: New Feature > Components: Connectors / ElasticSearch >Affects Versions: 1.9.0, 1.9.1, 1.11.0 >Reporter: ouyangwulin >Priority: Major > Fix For: 1.11.0 > > > From > user...@flink.apache.org([https://lists.apache.org/thread.html/ac4e0c068baeb3b070f0213a2e1314e6552b226b8132a4c49d667ecd%40%3Cuser-zh.flink.apache.org%3E]), > Becuase the es 6/7 not support ttl. so User need clean the index by > timestamp. Add dynamic index is a useful function. Add with properties > 'dynamicIndex' as a switch for open dynamicIndex. Add with properties > 'indexField' for the extract time field, Add properties 'indexInterval' for > change cycle mode. > > ||With property||discribe||default||Required|| > |dynamicIndex|Dynamic or not|false(true/false)|false| > |indexField|extract index field| none|dynamicIndex is true , then indexField > is required,only supported type "timestamp","date","long" | > |indexInterval|mode for cycle|d|ddynamicIndex is true , this field is > required ，the value optional is： > d：day > m：mouth > w：week| > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15400) elasticsearch sink support dynamic index.
[ https://issues.apache.org/jira/browse/FLINK-15400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ouyangwulin updated FLINK-15400: Description: >From >user...@flink.apache.org([https://lists.apache.org/thread.html/ac4e0c068baeb3b070f0213a2e1314e6552b226b8132a4c49d667ecd%40%3Cuser-zh.flink.apache.org%3E]), > Becuase the es 6/7 not support ttl. so User need clean the index by >timestamp. Add dynamic index is a useful function. Add with properties >'dynamicIndex' as a switch for open dynamicIndex. Add with properties >'indexField' for the extract time field, Add properties 'indexInterval' for >change cycle mode. ||With property||discribe||default||Required|| |dynamicIndex|Dynamic or not|false(true/false)|false| |indexField|extract index field| none|dynamicIndex is true , then indexField is required,only supported type "timestamp","date","long" | |indexInterval|mode for cycle|d|ddynamicIndex is true , this field is required ，the value optional is： d：day m：mouth w：week| was: >From >user...@flink.apache.org([https://lists.apache.org/thread.html/ac4e0c068baeb3b070f0213a2e1314e6552b226b8132a4c49d667ecd%40%3Cuser-zh.flink.apache.org%3E]), > Becuase the es 6/7 not support ttl. so User need clean the index by >timestamp. Add dynamic index is a useful function. Add with properties >'dynamicIndex' as a switch for open dynamicIndex. Add with properties >'indexField' for the extract time field, Add properties 'indexInterval' for >change cycle mode. ||With property||discribe||default||Required|| |dynamicIndex|Dynamic or not|false(true/false)|false| |indexField|extract index field| none|dynamicIndex is true , then indexField is required,only supported type "timestamp","date","long" | |indexInterval|mode for cycle|d|ddynamicIndex is true , this field is required ，可选参数值如下： d：day m：mouth w：week| > elasticsearch sink support dynamic index. > - > > Key: FLINK-15400 > URL: https://issues.apache.org/jira/browse/FLINK-15400 > Project: Flink > Issue Type: New Feature > Components: Connectors / ElasticSearch >Affects Versions: 1.9.0, 1.9.1, 1.11.0 >Reporter: ouyangwulin >Priority: Major > Fix For: 1.11.0 > > > From > user...@flink.apache.org([https://lists.apache.org/thread.html/ac4e0c068baeb3b070f0213a2e1314e6552b226b8132a4c49d667ecd%40%3Cuser-zh.flink.apache.org%3E]), > Becuase the es 6/7 not support ttl. so User need clean the index by > timestamp. Add dynamic index is a useful function. Add with properties > 'dynamicIndex' as a switch for open dynamicIndex. Add with properties > 'indexField' for the extract time field, Add properties 'indexInterval' for > change cycle mode. > > ||With property||discribe||default||Required|| > |dynamicIndex|Dynamic or not|false(true/false)|false| > |indexField|extract index field| none|dynamicIndex is true , then indexField > is required,only supported type "timestamp","date","long" | > |indexInterval|mode for cycle|d|ddynamicIndex is true , this field is > required ，the value optional is： > d：day > m：mouth > w：week| > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15400) elasticsearch sink support dynamic index.
ouyangwulin created FLINK-15400: --- Summary: elasticsearch sink support dynamic index. Key: FLINK-15400 URL: https://issues.apache.org/jira/browse/FLINK-15400 Project: Flink Issue Type: New Feature Components: Connectors / ElasticSearch Affects Versions: 1.9.1, 1.9.0, 1.11.0 Reporter: ouyangwulin Fix For: 1.11.0 >From >user...@flink.apache.org([https://lists.apache.org/thread.html/ac4e0c068baeb3b070f0213a2e1314e6552b226b8132a4c49d667ecd%40%3Cuser-zh.flink.apache.org%3E]), > Becuase the es 6/7 not support ttl. so User need clean the index by >timestamp. Add dynamic index is a useful function. Add with properties >'dynamicIndex' as a switch for open dynamicIndex. Add with properties >'indexField' for the extract time field, Add properties 'indexInterval' for >change cycle mode. ||With property||discribe||default||Required|| |dynamicIndex|Dynamic or not|false(true/false)|false| |indexField|extract index field| none|dynamicIndex is true , then indexField is required,only supported type "timestamp","date","long" | |indexInterval|mode for cycle|d|ddynamicIndex is true , this field is required ，可选参数值如下： d：day m：mouth w：week| -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-15378) StreamFileSystemSink supported mutil hdfs plugins.
[
https://issues.apache.org/jira/browse/FLINK-15378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17003237#comment-17003237
]
ouyangwulin edited comment on FLINK-15378 at 12/26/19 5:58 AM:
---
[~fly_in_gis]
>> For request 1,
Change plugin to not use '{{classloader.parent-first-patterns.default}} '
is a good idea。
>> For request 2,
Aggregated hdfs-site.xml can support multiple hdfs cluster in same
kerberoes, It create one filesystem instantiate to write to diff hdfs
cluster。But in our sense，We have one hdfs cluster with kerberoes， Other cluster
isn't need kerberoes，So Aggregated hdfs-site.xml is not work 。It also not need
plugin in `$FLINK_HOME/pulgins` ，The user jar can content FileSystem plugin。
was (Author: ouyangwuli):
[~fly_in_gis]
>> For request 1,
Change plugin to not use '{{classloader.parent-first-patterns.default}} '
is a good idea。
>> For request 2,
Aggregated hdfs-site.xml can support multiple hdfs cluster in same
kerberoes, It create one filesystem instantiate to write to diff hdfs
cluster。But in our sense，We have one hdfs cluster with kerberoes， Other cluster
is don't need kerberoes。It also not need plugin in `$FLINK_HOME/pulgins` ，The
user jar can content FileSystem plugin。
> StreamFileSystemSink supported mutil hdfs plugins.
> --
>
> Key: FLINK-15378
> URL: https://issues.apache.org/jira/browse/FLINK-15378
> Project: Flink
> Issue Type: Improvement
> Components: API / Core
>Affects Versions: 1.9.2, 1.11.0
>Reporter: ouyangwulin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Attachments: jobmananger.log
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> [As report from
> maillist|[https://lists.apache.org/thread.html/7a6b1e341bde0ef632a82f8d46c9c93da358244b6bac0d8d544d11cb%40%3Cuser.flink.apache.org%3E]]
> Request 1: FileSystem plugins not effect the default yarn dependecies.
> Request 2: StreamFileSystemSink supported mutil hdfs plugins under the same
> schema
> As Problem describe :
> when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
> clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
> implements '*FileSystemFactory*', when jm start, It will call
> FileSystem.initialize(configuration,
> PluginUtils.createPluginManagerFromRootFolder(configuration)) to load
> factories to map FileSystem#**{color}FS_FACTORIES, and the key is only
> schema. When tm/jm use local hadoop conf A , the user code use hadoop conf
> Bin 'filesystem plugin', Conf A and Conf B is used to different hadoop
> cluster. and The Jm will start failed, beacuse of the blodserver in JM will
> load Conf B to get filesystem. the full log add appendix.
>
> AS reslove method:
> use schema and spec identify as key for ' FileSystem#**FS_FACTORIES '
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-15378) StreamFileSystemSink supported mutil hdfs plugins.
[
https://issues.apache.org/jira/browse/FLINK-15378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17002704#comment-17002704
]
ouyangwulin edited comment on FLINK-15378 at 12/26/19 3:07 AM:
---
Please assign this for me！
was (Author: ouyangwuli):
Please assign the for me！
> StreamFileSystemSink supported mutil hdfs plugins.
> --
>
> Key: FLINK-15378
> URL: https://issues.apache.org/jira/browse/FLINK-15378
> Project: Flink
> Issue Type: Improvement
> Components: API / Core
>Affects Versions: 1.9.2, 1.11.0
>Reporter: ouyangwulin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Attachments: jobmananger.log
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> [As report from
> maillist|[https://lists.apache.org/thread.html/7a6b1e341bde0ef632a82f8d46c9c93da358244b6bac0d8d544d11cb%40%3Cuser.flink.apache.org%3E]]
> Request 1: FileSystem plugins not effect the default yarn dependecies.
> Request 2: StreamFileSystemSink supported mutil hdfs plugins under the same
> schema
> As Problem describe :
> when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
> clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
> implements '*FileSystemFactory*', when jm start, It will call
> FileSystem.initialize(configuration,
> PluginUtils.createPluginManagerFromRootFolder(configuration)) to load
> factories to map FileSystem#**{color}FS_FACTORIES, and the key is only
> schema. When tm/jm use local hadoop conf A , the user code use hadoop conf
> Bin 'filesystem plugin', Conf A and Conf B is used to different hadoop
> cluster. and The Jm will start failed, beacuse of the blodserver in JM will
> load Conf B to get filesystem. the full log add appendix.
>
> AS reslove method:
> use schema and spec identify as key for ' FileSystem#**FS_FACTORIES '
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15378) StreamFileSystemSink supported mutil hdfs plugins.
[
https://issues.apache.org/jira/browse/FLINK-15378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17003237#comment-17003237
]
ouyangwulin commented on FLINK-15378:
-
[~fly_in_gis]
>> For request 1,
Change plugin to not use '{{classloader.parent-first-patterns.default}} '
is a good idea。
>> For request 2,
Aggregated hdfs-site.xml can support multiple hdfs cluster in same
kerberoes, It create one filesystem instantiate to write to diff hdfs
cluster。But in our sense，We have one hdfs cluster with kerberoes， Other cluster
is don't need kerberoes。It also not need plugin in `$FLINK_HOME/pulgins` ，The
user jar can content FileSystem plugin。
> StreamFileSystemSink supported mutil hdfs plugins.
> --
>
> Key: FLINK-15378
> URL: https://issues.apache.org/jira/browse/FLINK-15378
> Project: Flink
> Issue Type: Improvement
> Components: API / Core
>Affects Versions: 1.9.2, 1.11.0
>Reporter: ouyangwulin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Attachments: jobmananger.log
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> [As report from
> maillist|[https://lists.apache.org/thread.html/7a6b1e341bde0ef632a82f8d46c9c93da358244b6bac0d8d544d11cb%40%3Cuser.flink.apache.org%3E]]
> Request 1: FileSystem plugins not effect the default yarn dependecies.
> Request 2: StreamFileSystemSink supported mutil hdfs plugins under the same
> schema
> As Problem describe :
> when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
> clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
> implements '*FileSystemFactory*', when jm start, It will call
> FileSystem.initialize(configuration,
> PluginUtils.createPluginManagerFromRootFolder(configuration)) to load
> factories to map FileSystem#**{color}FS_FACTORIES, and the key is only
> schema. When tm/jm use local hadoop conf A , the user code use hadoop conf
> Bin 'filesystem plugin', Conf A and Conf B is used to different hadoop
> cluster. and The Jm will start failed, beacuse of the blodserver in JM will
> load Conf B to get filesystem. the full log add appendix.
>
> AS reslove method:
> use schema and spec identify as key for ' FileSystem#**FS_FACTORIES '
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15378) StreamFileSystemSink supported mutil hdfs plugins.
[
https://issues.apache.org/jira/browse/FLINK-15378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ouyangwulin updated FLINK-15378:
Description:
[As report from
maillist|[https://lists.apache.org/thread.html/7a6b1e341bde0ef632a82f8d46c9c93da358244b6bac0d8d544d11cb%40%3Cuser.flink.apache.org%3E]]
Request 1: FileSystem plugins not effect the default yarn dependecies.
Request 2: StreamFileSystemSink supported mutil hdfs plugins under the same
schema
As Problem describe :
when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
implements '*FileSystemFactory*', when jm start, It will call
FileSystem.initialize(configuration,
PluginUtils.createPluginManagerFromRootFolder(configuration)) to load factories
to map FileSystem#**{color}FS_FACTORIES, and the key is only schema. When
tm/jm use local hadoop conf A , the user code use hadoop conf Bin 'filesystem
plugin', Conf A and Conf B is used to different hadoop cluster. and The Jm
will start failed, beacuse of the blodserver in JM will load Conf B to get
filesystem. the full log add appendix.
AS reslove method:
use schema and spec identify as key for ' FileSystem#**FS_FACTORIES '
was:
[As report from
maillist|[https://lists.apache.org/thread.html/7a6b1e341bde0ef632a82f8d46c9c93da358244b6bac0d8d544d11cb%40%3Cuser.flink.apache.org%3E]]
Request 1: FileSystem plugins not effect the default yarn dependecies.
Request 2: StreamFileSystemSink supported mutil hdfs plugins.
As Problem describe :
when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
implements '*FileSystemFactory*', when jm start, It will call
FileSystem.initialize(configuration,
PluginUtils.createPluginManagerFromRootFolder(configuration)) to load factories
to map FileSystem#**{color}FS_FACTORIES, and the key is only schema. When
tm/jm use local hadoop conf A , the user code use hadoop conf Bin 'filesystem
plugin', Conf A and Conf B is used to different hadoop cluster. and The Jm
will start failed, beacuse of the blodserver in JM will load Conf B to get
filesystem. the full log add appendix.
AS reslove method:
use schema and spec identify as key for ' FileSystem#**FS_FACTORIES '
> StreamFileSystemSink supported mutil hdfs plugins.
> --
>
> Key: FLINK-15378
> URL: https://issues.apache.org/jira/browse/FLINK-15378
> Project: Flink
> Issue Type: Improvement
> Components: API / Core
>Affects Versions: 1.9.2, 1.11.0
>Reporter: ouyangwulin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Attachments: jobmananger.log
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> [As report from
> maillist|[https://lists.apache.org/thread.html/7a6b1e341bde0ef632a82f8d46c9c93da358244b6bac0d8d544d11cb%40%3Cuser.flink.apache.org%3E]]
> Request 1: FileSystem plugins not effect the default yarn dependecies.
> Request 2: StreamFileSystemSink supported mutil hdfs plugins under the same
> schema
> As Problem describe :
> when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
> clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
> implements '*FileSystemFactory*', when jm start, It will call
> FileSystem.initialize(configuration,
> PluginUtils.createPluginManagerFromRootFolder(configuration)) to load
> factories to map FileSystem#**{color}FS_FACTORIES, and the key is only
> schema. When tm/jm use local hadoop conf A , the user code use hadoop conf
> Bin 'filesystem plugin', Conf A and Conf B is used to different hadoop
> cluster. and The Jm will start failed, beacuse of the blodserver in JM will
> load Conf B to get filesystem. the full log add appendix.
>
> AS reslove method:
> use schema and spec identify as key for ' FileSystem#**FS_FACTORIES '
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15378) StreamFileSystemSink supported mutil hdfs plugins.
[
https://issues.apache.org/jira/browse/FLINK-15378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ouyangwulin updated FLINK-15378:
Description:
[As report from
maillist|[https://lists.apache.org/thread.html/7a6b1e341bde0ef632a82f8d46c9c93da358244b6bac0d8d544d11cb%40%3Cuser.flink.apache.org%3E]]
Request 1: FileSystem plugins not effect the default yarn dependecies.
Request 2: StreamFileSystemSink supported mutil hdfs plugins.
As Problem describe :
when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
implements '*FileSystemFactory*', when jm start, It will call
FileSystem.initialize(configuration,
PluginUtils.createPluginManagerFromRootFolder(configuration)) to load factories
to map FileSystem#**{color}FS_FACTORIES, and the key is only schema. When
tm/jm use local hadoop conf A , the user code use hadoop conf Bin 'filesystem
plugin', Conf A and Conf B is used to different hadoop cluster. and The Jm
will start failed, beacuse of the blodserver in JM will load Conf B to get
filesystem. the full log add appendix.
AS reslove method:
use schema and spec identify as key for ' FileSystem#**FS_FACTORIES '
was:
[As report from
maillist|[https://lists.apache.org/thread.html/7a6b1e341bde0ef632a82f8d46c9c93da358244b6bac0d8d544d11cb%40%3Cuser.flink.apache.org%3E]]
Request 1: FileSystem plugins not effect the default yarn dependecies.
Request 2: StreamFileSystemSink supported mutil hdfs plugins.
As Problem describe :
when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
implements '*FileSystemFactory*', when jm start, It will call
FileSystem.initialize(configuration,
PluginUtils.createPluginManagerFromRootFolder(configuration)) to load factories
to map FileSystem#**{color}FS_FACTORIES, and the key is only schema. When
tm/jm use local hadoop conf A , the user code use hadoop conf Bin 'filesystem
plugin', Conf A and Conf B is used to different hadoop cluster. and The Jm
will start failed, beacuse of the blodserver in JM will load Conf B to get
filesystem. the full log add appendix.
AS reslove method:
use schema and authority as key for ' FileSystem#**FS_FACTORIES '
> StreamFileSystemSink supported mutil hdfs plugins.
> --
>
> Key: FLINK-15378
> URL: https://issues.apache.org/jira/browse/FLINK-15378
> Project: Flink
> Issue Type: Improvement
> Components: API / Core
>Affects Versions: 1.9.2, 1.11.0
>Reporter: ouyangwulin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Attachments: jobmananger.log
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> [As report from
> maillist|[https://lists.apache.org/thread.html/7a6b1e341bde0ef632a82f8d46c9c93da358244b6bac0d8d544d11cb%40%3Cuser.flink.apache.org%3E]]
> Request 1: FileSystem plugins not effect the default yarn dependecies.
> Request 2: StreamFileSystemSink supported mutil hdfs plugins.
> As Problem describe :
> when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
> clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
> implements '*FileSystemFactory*', when jm start, It will call
> FileSystem.initialize(configuration,
> PluginUtils.createPluginManagerFromRootFolder(configuration)) to load
> factories to map FileSystem#**{color}FS_FACTORIES, and the key is only
> schema. When tm/jm use local hadoop conf A , the user code use hadoop conf
> Bin 'filesystem plugin', Conf A and Conf B is used to different hadoop
> cluster. and The Jm will start failed, beacuse of the blodserver in JM will
> load Conf B to get filesystem. the full log add appendix.
>
> AS reslove method:
> use schema and spec identify as key for ' FileSystem#**FS_FACTORIES '
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15378) StreamFileSystemSink supported mutil hdfs plugins.
[
https://issues.apache.org/jira/browse/FLINK-15378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ouyangwulin updated FLINK-15378:
Description:
[As report from
maillist|[https://lists.apache.org/thread.html/7a6b1e341bde0ef632a82f8d46c9c93da358244b6bac0d8d544d11cb%40%3Cuser.flink.apache.org%3E]]
Request 1: FileSystem plugins not effect the default yarn dependecies.
Request 2: StreamFileSystemSink supported mutil hdfs plugins.
As Problem describe :
when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
implements '*FileSystemFactory*', when jm start, It will call
FileSystem.initialize(configuration,
PluginUtils.createPluginManagerFromRootFolder(configuration)) to load factories
to map FileSystem#**{color}FS_FACTORIES, and the key is only schema. When
tm/jm use local hadoop conf A , the user code use hadoop conf Bin 'filesystem
plugin', Conf A and Conf B is used to different hadoop cluster. and The Jm
will start failed, beacuse of the blodserver in JM will load Conf B to get
filesystem. the full log add appendix.
AS reslove method:
use schema and authority as key for ' FileSystem#**FS_FACTORIES '
was:
Request 1: FileSystem plugins not effect the default yarn dependecies.
Request 2: StreamFileSystemSink supported mutil hdfs plugins.
As Problem describe :
when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
implements '*FileSystemFactory*', when jm start, It will call
FileSystem.initialize(configuration,
PluginUtils.createPluginManagerFromRootFolder(configuration)) to load factories
to map FileSystem#**{color}FS_FACTORIES, and the key is only schema. When
tm/jm use local hadoop conf A , the user code use hadoop conf Bin 'filesystem
plugin', Conf A and Conf B is used to different hadoop cluster. and The Jm
will start failed, beacuse of the blodserver in JM will load Conf B to get
filesystem. the full log add appendix.
AS reslove method:
use schema and authority as key for ' FileSystem#**FS_FACTORIES '
> StreamFileSystemSink supported mutil hdfs plugins.
> --
>
> Key: FLINK-15378
> URL: https://issues.apache.org/jira/browse/FLINK-15378
> Project: Flink
> Issue Type: Improvement
> Components: API / Core
>Affects Versions: 1.9.2, 1.11.0
>Reporter: ouyangwulin
>Priority: Major
> Fix For: 1.11.0
>
> Attachments: jobmananger.log
>
>
> [As report from
> maillist|[https://lists.apache.org/thread.html/7a6b1e341bde0ef632a82f8d46c9c93da358244b6bac0d8d544d11cb%40%3Cuser.flink.apache.org%3E]]
> Request 1: FileSystem plugins not effect the default yarn dependecies.
> Request 2: StreamFileSystemSink supported mutil hdfs plugins.
> As Problem describe :
> when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
> clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
> implements '*FileSystemFactory*', when jm start, It will call
> FileSystem.initialize(configuration,
> PluginUtils.createPluginManagerFromRootFolder(configuration)) to load
> factories to map FileSystem#**{color}FS_FACTORIES, and the key is only
> schema. When tm/jm use local hadoop conf A , the user code use hadoop conf
> Bin 'filesystem plugin', Conf A and Conf B is used to different hadoop
> cluster. and The Jm will start failed, beacuse of the blodserver in JM will
> load Conf B to get filesystem. the full log add appendix.
>
> AS reslove method:
> use schema and authority as key for ' FileSystem#**FS_FACTORIES '
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15378) StreamFileSystemSink supported mutil hdfs plugins.
[
https://issues.apache.org/jira/browse/FLINK-15378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17002719#comment-17002719
]
ouyangwulin commented on FLINK-15378:
-
[~pnowojski] When I read the code ClusterEntrypoint#startCluster,It load
plugins in FileSystem#**FS_FACTORIES when startcluster, So the code
'initializeWithoutPlugins' is not work. I think it is a conflict with
‘FileSystem#getUnguardedFileSystem’。
[~wangy] Do you think ‘ use schema and authority as key for '
FileSystem#**FS_FACTORIES ' is suitable?
> StreamFileSystemSink supported mutil hdfs plugins.
> --
>
> Key: FLINK-15378
> URL: https://issues.apache.org/jira/browse/FLINK-15378
> Project: Flink
> Issue Type: Improvement
> Components: API / Core
>Affects Versions: 1.9.2, 1.11.0
>Reporter: ouyangwulin
>Priority: Major
> Fix For: 1.11.0
>
> Attachments: jobmananger.log
>
>
> Request 1: FileSystem plugins not effect the default yarn dependecies.
> Request 2: StreamFileSystemSink supported mutil hdfs plugins.
> As Problem describe :
> when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
> clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
> implements '*FileSystemFactory*', when jm start, It will call
> FileSystem.initialize(configuration,
> PluginUtils.createPluginManagerFromRootFolder(configuration)) to load
> factories to map FileSystem#**{color}FS_FACTORIES, and the key is only
> schema. When tm/jm use local hadoop conf A , the user code use hadoop conf
> Bin 'filesystem plugin', Conf A and Conf B is used to different hadoop
> cluster. and The Jm will start failed, beacuse of the blodserver in JM will
> load Conf B to get filesystem. the full log add appendix.
>
> AS reslove method:
> use schema and authority as key for ' FileSystem#**FS_FACTORIES '
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15378) StreamFileSystemSink supported mutil hdfs plugins.
[
https://issues.apache.org/jira/browse/FLINK-15378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ouyangwulin updated FLINK-15378:
Description:
Request 1: FileSystem plugins not effect the default yarn dependecies.
Request 2: StreamFileSystemSink supported mutil hdfs plugins.
As Problem describe :
when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
implements '*FileSystemFactory*', when jm start, It will call
FileSystem.initialize(configuration,
PluginUtils.createPluginManagerFromRootFolder(configuration)) to load factories
to map FileSystem#**{color}FS_FACTORIES, and the key is only schema. When
tm/jm use local hadoop conf A , the user code use hadoop conf Bin 'filesystem
plugin', Conf A and Conf B is used to different hadoop cluster. and The Jm
will start failed, beacuse of the blodserver in JM will load Conf B to get
filesystem. the full log add appendix.
AS reslove method:
use schema and authority as key for ' FileSystem#**FS_FACTORIES '
was:
Request 1: FileSystem plugins not effect the default yarn dependecies.
Request 2: StreamFileSystemSink supported mutil hdfs plugins.
As Problem describe :
when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
implements '*FileSystemFactory*', when jm start, It will call
FileSystem.initialize(configuration,
PluginUtils.createPluginManagerFromRootFolder(configuration)) to load factories
to map FileSystem#**{color}FS_FACTORIES, and the key is only schema. When
tm/jm use local hadoop conf A , the user code use hadoop conf Bin 'filesystem
plugin', Conf A and Conf B is used to different hadoop cluster. and The Jm
will start failed, beacuse of the blodserver in JM will load Conf B to get
filesystem. the full log add appendix.
AS reslove method:
use schema and authority as key for ' FileSystem#**FS_FACTORIES key'
> StreamFileSystemSink supported mutil hdfs plugins.
> --
>
> Key: FLINK-15378
> URL: https://issues.apache.org/jira/browse/FLINK-15378
> Project: Flink
> Issue Type: Improvement
> Components: API / Core
>Affects Versions: 1.9.2, 1.11.0
>Reporter: ouyangwulin
>Priority: Major
> Fix For: 1.11.0
>
> Attachments: jobmananger.log
>
>
> Request 1: FileSystem plugins not effect the default yarn dependecies.
> Request 2: StreamFileSystemSink supported mutil hdfs plugins.
> As Problem describe :
> when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
> clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
> implements '*FileSystemFactory*', when jm start, It will call
> FileSystem.initialize(configuration,
> PluginUtils.createPluginManagerFromRootFolder(configuration)) to load
> factories to map FileSystem#**{color}FS_FACTORIES, and the key is only
> schema. When tm/jm use local hadoop conf A , the user code use hadoop conf
> Bin 'filesystem plugin', Conf A and Conf B is used to different hadoop
> cluster. and The Jm will start failed, beacuse of the blodserver in JM will
> load Conf B to get filesystem. the full log add appendix.
>
> AS reslove method:
> use schema and authority as key for ' FileSystem#**FS_FACTORIES '
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15378) StreamFileSystemSink supported mutil hdfs plugins.
[
https://issues.apache.org/jira/browse/FLINK-15378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17002704#comment-17002704
]
ouyangwulin commented on FLINK-15378:
-
Please assign the for me！
> StreamFileSystemSink supported mutil hdfs plugins.
> --
>
> Key: FLINK-15378
> URL: https://issues.apache.org/jira/browse/FLINK-15378
> Project: Flink
> Issue Type: Improvement
> Components: API / Core
>Affects Versions: 1.9.2, 1.11.0
>Reporter: ouyangwulin
>Priority: Major
> Fix For: 1.11.0
>
> Attachments: jobmananger.log
>
>
> Request 1: FileSystem plugins not effect the default yarn dependecies.
> Request 2: StreamFileSystemSink supported mutil hdfs plugins.
> As Problem describe :
> when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
> clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
> implements '*FileSystemFactory*', when jm start, It will call
> FileSystem.initialize(configuration,
> PluginUtils.createPluginManagerFromRootFolder(configuration)) to load
> factories to map FileSystem#**{color}FS_FACTORIES, and the key is only
> schema. When tm/jm use local hadoop conf A , the user code use hadoop conf
> Bin 'filesystem plugin', Conf A and Conf B is used to different hadoop
> cluster. and The Jm will start failed, beacuse of the blodserver in JM will
> load Conf B to get filesystem. the full log add appendix.
>
> AS reslove method:
> use schema and authority as key for ' FileSystem#**FS_FACTORIES key'
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15378) StreamFileSystemSink supported mutil hdfs plugins.
[
https://issues.apache.org/jira/browse/FLINK-15378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ouyangwulin updated FLINK-15378:
Description:
Request 1: FileSystem plugins not effect the default yarn dependecies.
Request 2: StreamFileSystemSink supported mutil hdfs plugins.
As Problem describe :
when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
implements '*FileSystemFactory*', when jm start, It will call
FileSystem.initialize(configuration,
PluginUtils.createPluginManagerFromRootFolder(configuration)) to load factories
to map FileSystem#**{color}FS_FACTORIES, and the key is only schema. When
tm/jm use local hadoop conf A , the user code use hadoop conf Bin 'filesystem
plugin', Conf A and Conf B is used to different hadoop cluster. and The Jm
will start failed, beacuse of the blodserver in JM will load Conf B to get
filesystem. the full log add appendix.
AS reslove method:
use schema and authority as key for ' FileSystem#**FS_FACTORIES key'
was:
Request 1: FileSystem plugins not effect the default yarn dependecies.
Request 2: StreamFileSystemSink supported mutil hdfs plugins.
As Problem describe :
when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
implements '*FileSystemFactory*', when jm start, It will call
FileSystem.initialize(configuration,
PluginUtils.createPluginManagerFromRootFolder(configuration)) to load factories
to map FileSystem#**{color}FS_FACTORIES, and the key is only schema. When
tm/jm use local hadoop conf A , the user code use hadoop conf Bin 'filesystem
plugin', Conf A and Conf B is used to different hadoop cluster. and The Jm
will start failed, beacuse of the blodserver in JM will load Conf B to get
filesystem. the full log add appendix.
> StreamFileSystemSink supported mutil hdfs plugins.
> --
>
> Key: FLINK-15378
> URL: https://issues.apache.org/jira/browse/FLINK-15378
> Project: Flink
> Issue Type: Improvement
> Components: API / Core
>Affects Versions: 1.9.2, 1.11.0
>Reporter: ouyangwulin
>Priority: Major
> Fix For: 1.11.0
>
> Attachments: jobmananger.log
>
>
> Request 1: FileSystem plugins not effect the default yarn dependecies.
> Request 2: StreamFileSystemSink supported mutil hdfs plugins.
> As Problem describe :
> when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
> clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
> implements '*FileSystemFactory*', when jm start, It will call
> FileSystem.initialize(configuration,
> PluginUtils.createPluginManagerFromRootFolder(configuration)) to load
> factories to map FileSystem#**{color}FS_FACTORIES, and the key is only
> schema. When tm/jm use local hadoop conf A , the user code use hadoop conf
> Bin 'filesystem plugin', Conf A and Conf B is used to different hadoop
> cluster. and The Jm will start failed, beacuse of the blodserver in JM will
> load Conf B to get filesystem. the full log add appendix.
>
> AS reslove method:
> use schema and authority as key for ' FileSystem#**FS_FACTORIES key'
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15378) StreamFileSystemSink supported mutil hdfs plugins.
[
https://issues.apache.org/jira/browse/FLINK-15378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ouyangwulin updated FLINK-15378:
Attachment: jobmananger.log
> StreamFileSystemSink supported mutil hdfs plugins.
> --
>
> Key: FLINK-15378
> URL: https://issues.apache.org/jira/browse/FLINK-15378
> Project: Flink
> Issue Type: Improvement
> Components: API / Core
>Affects Versions: 1.9.2, 1.11.0
>Reporter: ouyangwulin
>Priority: Major
> Fix For: 1.11.0
>
> Attachments: jobmananger.log
>
>
> Request 1: FileSystem plugins not effect the default yarn dependecies.
> Request 2: StreamFileSystemSink supported mutil hdfs plugins.
>
> As Problem describe :
> when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
> clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
> implements '*FileSystemFactory*', when jm start, It will call
> FileSystem.initialize(configuration,
> PluginUtils.createPluginManagerFromRootFolder(configuration)) to load
> factories to map FileSystem#**{color}FS_FACTORIES, and the key is only
> schema. When tm/jm use local hadoop conf A , the user code use hadoop conf
> Bin 'filesystem plugin', Conf A and Conf B is used to different hadoop
> cluster. and The Jm will start failed, beacuse of the blodserver in JM will
> load Conf B to get filesystem. the full log add appendix.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15378) StreamFileSystemSink supported mutil hdfs plugins.
[
https://issues.apache.org/jira/browse/FLINK-15378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ouyangwulin updated FLINK-15378:
Description:
Request 1: FileSystem plugins not effect the default yarn dependecies.
Request 2: StreamFileSystemSink supported mutil hdfs plugins.
As Problem describe :
when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
implements '*FileSystemFactory*', when jm start, It will call
FileSystem.initialize(configuration,
PluginUtils.createPluginManagerFromRootFolder(configuration)) to load factories
to map FileSystem#**{color}FS_FACTORIES, and the key is only schema. When
tm/jm use local hadoop conf A , the user code use hadoop conf Bin 'filesystem
plugin', Conf A and Conf B is used to different hadoop cluster. and The Jm
will start failed, beacuse of the blodserver in JM will load Conf B to get
filesystem. the full log add appendix.
was:
Request 1: FileSystem plugins not effect the default yarn dependecies.
Request 2: StreamFileSystemSink supported mutil hdfs plugins.
> StreamFileSystemSink supported mutil hdfs plugins.
> --
>
> Key: FLINK-15378
> URL: https://issues.apache.org/jira/browse/FLINK-15378
> Project: Flink
> Issue Type: Improvement
> Components: API / Core
>Affects Versions: 1.9.2, 1.11.0
>Reporter: ouyangwulin
>Priority: Major
> Fix For: 1.11.0
>
>
> Request 1: FileSystem plugins not effect the default yarn dependecies.
> Request 2: StreamFileSystemSink supported mutil hdfs plugins.
>
> As Problem describe :
> when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
> clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
> implements '*FileSystemFactory*', when jm start, It will call
> FileSystem.initialize(configuration,
> PluginUtils.createPluginManagerFromRootFolder(configuration)) to load
> factories to map FileSystem#**{color}FS_FACTORIES, and the key is only
> schema. When tm/jm use local hadoop conf A , the user code use hadoop conf
> Bin 'filesystem plugin', Conf A and Conf B is used to different hadoop
> cluster. and The Jm will start failed, beacuse of the blodserver in JM will
> load Conf B to get filesystem. the full log add appendix.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-15378) StreamFileSystemSink supported mutil hdfs plugins.
ouyangwulin created FLINK-15378: --- Summary: StreamFileSystemSink supported mutil hdfs plugins. Key: FLINK-15378 URL: https://issues.apache.org/jira/browse/FLINK-15378 Project: Flink Issue Type: Improvement Components: API / Core Affects Versions: 1.9.2, 1.11.0 Reporter: ouyangwulin Fix For: 1.11.0 Request 1: FileSystem plugins not effect the default yarn dependecies. Request 2: StreamFileSystemSink supported mutil hdfs plugins. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-14831) When edit flink-metrics-influxdb, need add metrics.md by hand
[
https://issues.apache.org/jira/browse/FLINK-14831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ouyangwulin closed FLINK-14831.
---
Resolution: Won't Fix
> When edit flink-metrics-influxdb, need add metrics.md by hand
> --
>
> Key: FLINK-14831
> URL: https://issues.apache.org/jira/browse/FLINK-14831
> Project: Flink
> Issue Type: Improvement
> Components: Documentation, Runtime / Metrics
>Reporter: ouyangwulin
>Priority: Major
>
> When edit flink-metrics-influxdb, need add metrics.md by hand. AND
> {code:java}
> mvn package -Dgenerate-config-docs -pl flink-docs -am -nsu -DskipTests{code}
> , Is not work
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14831) When edit flink-metrics-influxdb, need add metrics.md by hand
[
https://issues.apache.org/jira/browse/FLINK-14831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16995493#comment-16995493
]
ouyangwulin commented on FLINK-14831:
-
I will close this jira, I know your mean.
> When edit flink-metrics-influxdb, need add metrics.md by hand
> --
>
> Key: FLINK-14831
> URL: https://issues.apache.org/jira/browse/FLINK-14831
> Project: Flink
> Issue Type: Improvement
> Components: Documentation, Runtime / Metrics
>Reporter: ouyangwulin
>Priority: Major
>
> When edit flink-metrics-influxdb, need add metrics.md by hand. AND
> {code:java}
> mvn package -Dgenerate-config-docs -pl flink-docs -am -nsu -DskipTests{code}
> , Is not work
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14831) When edit flink-metrics-influxdb, need add metrics.md by hand
[
https://issues.apache.org/jira/browse/FLINK-14831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16995283#comment-16995283
]
ouyangwulin commented on FLINK-14831:
-
[~rmetzger] I have try to fixed this problems. and add
'ConfigOptionsDocGenerator', but It conflict with 'metrics.prometheus' when
validate ‘host’。 So the metric report only 'prometheus' was generator doc auto
> When edit flink-metrics-influxdb, need add metrics.md by hand
> --
>
> Key: FLINK-14831
> URL: https://issues.apache.org/jira/browse/FLINK-14831
> Project: Flink
> Issue Type: Improvement
> Components: Documentation, Runtime / Metrics
>Reporter: ouyangwulin
>Priority: Major
>
> When edit flink-metrics-influxdb, need add metrics.md by hand. AND
> {code:java}
> mvn package -Dgenerate-config-docs -pl flink-docs -am -nsu -DskipTests{code}
> , Is not work
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14172) Implement KubeClient with official Java client library for kubernetes
[ https://issues.apache.org/jira/browse/FLINK-14172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993247#comment-16993247 ] ouyangwulin commented on FLINK-14172: - Please assign this to me > Implement KubeClient with official Java client library for kubernetes > - > > Key: FLINK-14172 > URL: https://issues.apache.org/jira/browse/FLINK-14172 > Project: Flink > Issue Type: Sub-task > Components: Deployment / Kubernetes >Reporter: Yang Wang >Priority: Major > > Official Java client library for kubernetes is become more and more active. > The new features(such as leader election) and some client > implementations(informer, lister, cache) are better. So we should use the > official java client for kubernetes in flink. > https://github.com/kubernetes-client/java -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14986) Support to get detailed Kubernetes cluster description
[ https://issues.apache.org/jira/browse/FLINK-14986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16992498#comment-16992498 ] ouyangwulin commented on FLINK-14986: - Please assign this for me [~fly_in_gis] > Support to get detailed Kubernetes cluster description > -- > > Key: FLINK-14986 > URL: https://issues.apache.org/jira/browse/FLINK-14986 > Project: Flink > Issue Type: Sub-task >Reporter: Yang Wang >Priority: Major > > Currently Flink supports get yarn cluster description by > `YarnClusterDescriptor#getClusterDescription`. We should support the same > behavior in Kubernetes cluster. > Usually the cluster description contains the "total resources, available > resources, etc." -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15176) Add '--job-classname' to flink-container 'job-cluster-job.yaml.template'
ouyangwulin created FLINK-15176: --- Summary: Add '--job-classname' to flink-container 'job-cluster-job.yaml.template' Key: FLINK-15176 URL: https://issues.apache.org/jira/browse/FLINK-15176 Project: Flink Issue Type: Improvement Components: Deployment / Docker Affects Versions: 1.11.0 Reporter: ouyangwulin Fix For: 1.11.0 As from 'user...@flink.apache.org'. When use 'job-cluster-job.yaml.template' deploy a job, the template don't have a good sense about how to use '--job-classname' -- This message was sent by Atlassian Jira (v8.3.4#803005)
