[GitHub] [flink] flinkbot edited a comment on pull request #14684: [FLINK-20460][Connector-HBase] Support async lookup for HBase connector
flinkbot edited a comment on pull request #14684: URL: https://github.com/apache/flink/pull/14684#issuecomment-762224447 ## CI report: * 0375eb67f1ccb8460862baea733795874ba102f3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14352) * c34d423032cb900ffad161afdb7f0b14fe7a824a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15192: [FLINK-21777][network] Replace the 4M data writing cache of sort-merge shuffle with writev system call
flinkbot edited a comment on pull request #15192: URL: https://github.com/apache/flink/pull/15192#issuecomment-798828549 ## CI report: * b33ebbeb181bb30510dcd28361fb9fff39a140a2 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14606) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15136: [FLINK-21622][table-planner] Introduce function TO_TIMESTAMP_LTZ(numeric [, precision])
flinkbot edited a comment on pull request #15136: URL: https://github.com/apache/flink/pull/15136#issuecomment-795141489 ## CI report: * 30df8333ffb8cb76b6bf46043a3cc8dc1b2d8420 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14578) * f8626abdcd33ea2e5ea91cceb82ed8ecb049fe1a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14609) * faff60c6e9ec1eedd27c3f1d221d7fc2de8f99c7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-21467) Document possible recommended usage of Bounded{One/Multi}Input.endInput and emphasize that they could be called multiple times
[
https://issues.apache.org/jira/browse/FLINK-21467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17301066#comment-17301066
]
Kezhu Wang commented on FLINK-21467:
Hi [~pnowojski], I guess it depends on various subtleties:
# "MAX_WATERMARK" could come from last unaligned checkpoint.
# Last unaligned checkpoint considered as completed but fail at
"notifyCheckpointComplete" phase".
# Recovered subtask gets splits assigned from either source enumerator or
redistributed operator list state.
The key unknown questions are:
# Will "MAX_WATERMARK" be persisted in unaligned checkpoint ?
# When an operator is considered finished ?
# A recovered finishing subtask could get new splits assigned ?
> Document possible recommended usage of Bounded{One/Multi}Input.endInput and
> emphasize that they could be called multiple times
> --
>
> Key: FLINK-21467
> URL: https://issues.apache.org/jira/browse/FLINK-21467
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.13.0
>Reporter: Kezhu Wang
>Priority: Major
>
> It is too tempting to use these api, especially {{BoundedOneInput.endInput}},
> to commit final result before FLIP-147 delivered. And this will cause
> re-commit after failover as [~gaoyunhaii] has pointed out in FLINK-21132.
> I have
> [pointed|https://github.com/apache/iceberg/issues/2033#issuecomment-784153620]
> this out in
> [apache/iceberg#2033|https://github.com/apache/iceberg/issues/2033], please
> correct me if I was wrong.
> cc [~aljoscha] [~pnowojski] [~roman_khachatryan]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #15136: [FLINK-21622][table-planner] Introduce function TO_TIMESTAMP_LTZ(numeric [, precision])
flinkbot edited a comment on pull request #15136: URL: https://github.com/apache/flink/pull/15136#issuecomment-795141489 ## CI report: * 30df8333ffb8cb76b6bf46043a3cc8dc1b2d8420 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14578) * f8626abdcd33ea2e5ea91cceb82ed8ecb049fe1a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] SteNicholas commented on a change in pull request #15185: [FLINK-21768][clients]Optimize system.exit() logic of CliFrontend
SteNicholas commented on a change in pull request #15185:
URL: https://github.com/apache/flink/pull/15185#discussion_r593852102
##
File path:
flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java
##
@@ -1124,20 +1124,21 @@ public static void main(final String[] args) {
final List customCommandLines =
loadCustomCommandLines(configuration, configurationDirectory);
+int retCode = -1;
try {
final CliFrontend cli = new CliFrontend(configuration,
customCommandLines);
SecurityUtils.install(new
SecurityConfiguration(cli.configuration));
-int retCode =
+retCode =
SecurityUtils.getInstalledContext().runSecured(() ->
cli.parseAndRun(args));
-System.exit(retCode);
} catch (Throwable t) {
final Throwable strippedThrowable =
ExceptionUtils.stripException(t,
UndeclaredThrowableException.class);
LOG.error("Fatal error while running command line interface.",
strippedThrowable);
strippedThrowable.printStackTrace();
System.exit(31);
Review comment:
retCode = 31, and call `System#exit` method in finally block.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15174: [FLINK-21651][sql-client] Migrate module related tests in LocalExecutorITCase to the new integration test framework
flinkbot edited a comment on pull request #15174: URL: https://github.com/apache/flink/pull/15174#issuecomment-797861091 ## CI report: * faa510486a60de5f71d9cf924ae47fcac04560ff Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14576) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14572) * 20dc7144e6ab5afc640368bdea2786c1b3eaa51d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14608) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15174: [FLINK-21651][sql-client] Migrate module related tests in LocalExecutorITCase to the new integration test framework
flinkbot edited a comment on pull request #15174: URL: https://github.com/apache/flink/pull/15174#issuecomment-797861091 ## CI report: * faa510486a60de5f71d9cf924ae47fcac04560ff Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14576) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14572) * 20dc7144e6ab5afc640368bdea2786c1b3eaa51d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-21466) Make "embedded" parameter optional when start sql-client.sh
[
https://issues.apache.org/jira/browse/FLINK-21466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17301048#comment-17301048
]
Shengkai Fang commented on FLINK-21466:
---
I think you are right. Make "embedded" as the default parameter to strat the
client. Please ignore the "-i init1.sql,init2.sql" that belongs to FLINK-20320.
> Make "embedded" parameter optional when start sql-client.sh
> ---
>
> Key: FLINK-21466
> URL: https://issues.apache.org/jira/browse/FLINK-21466
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Shengkai Fang
>Assignee: zck
>Priority: Major
> Labels: starter
>
> Users can use command
> {code:java}
> >./sql-client.sh
> {code}
> to start the sql client and don't need to add {{embedded}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21466) Make "embedded" parameter optional when start sql-client.sh
[
https://issues.apache.org/jira/browse/FLINK-21466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shengkai Fang updated FLINK-21466:
--
Description:
Users can use command
{code:java}
>./sql-client.sh
{code}
to start the sql client and don't need to add {{embedded}}.
was:
Users can use command
{code:java}
>./sql-client.sh embedded -i init1.sql,init2.sql
{code}
to start the sql client and don't need to add {{embedded}}.
> Make "embedded" parameter optional when start sql-client.sh
> ---
>
> Key: FLINK-21466
> URL: https://issues.apache.org/jira/browse/FLINK-21466
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Shengkai Fang
>Assignee: zck
>Priority: Major
> Labels: starter
>
> Users can use command
> {code:java}
> >./sql-client.sh
> {code}
> to start the sql client and don't need to add {{embedded}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-21743) JdbcXaSinkFunction throws XAER_RMFAIL when calling snapshotState and beginTx
[
https://issues.apache.org/jira/browse/FLINK-21743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17301045#comment-17301045
]
Wei Hao edited comment on FLINK-21743 at 3/14/21, 5:26 AM:
---
The config 'max_prepared_transactions' seems to be specific to Postgresql
server, which seems to be an issue in [exactly-once jdbc
implementation|https://github.com/apache/flink/pull/10847]. For example, MySQL
server doesn't support simultaneous prepared transactions. Is there a
work-around here?
was (Author: weihao):
The config 'max_prepared_transactions' seems to be specific to Postgresql
server, which seems to be an issue in [exactly-once jdbc
implementation|https://github.com/apache/flink/pull/10847].
> JdbcXaSinkFunction throws XAER_RMFAIL when calling snapshotState and beginTx
>
>
> Key: FLINK-21743
> URL: https://issues.apache.org/jira/browse/FLINK-21743
> Project: Flink
> Issue Type: Test
> Components: Connectors / JDBC
>Affects Versions: 1.13.0
> Environment: org.apache.flink:flink-streaming-java_2.11:1.12.1
> org.apache.flink:flink-connector-jdbc_2.11:1.13-SNAPSHOT
>Reporter: Wei Hao
>Assignee: Roman Khachatryan
>Priority: Major
>
> {code:java}
> public void snapshotState(FunctionSnapshotContext context) throws Exception {
> LOG.debug("snapshot state, checkpointId={}", context.getCheckpointId());
> this.rollbackPreparedFromCheckpoint(context.getCheckpointId());
> this.prepareCurrentTx(context.getCheckpointId());
> this.beginTx(context.getCheckpointId() + 1L);
> this.stateHandler.store(JdbcXaSinkFunctionState.of(this.preparedXids,
> this.hangingXids));
> }
> {code}
> When checkpointing starts, it calls snapshotState(), which ends and prepares
> the current transaction. The issue I found is with beginTx(), where a new Xid
> is generated and xaFacade will run command like 'xa start new_xid', which
> will throw the exception as shown below and causes checkpointing failure.
> {code:java}
> Caused by: org.apache.flink.connector.jdbc.xa.XaFacade$TransientXaException:
> com.mysql.cj.jdbc.MysqlXAException: XAER_RMFAIL: The command cannot be
> executed when global transaction is in the PREPARED stateCaused by:
> org.apache.flink.connector.jdbc.xa.XaFacade$TransientXaException:
> com.mysql.cj.jdbc.MysqlXAException: XAER_RMFAIL: The command cannot be
> executed when global transaction is in the PREPARED state at
> org.apache.flink.connector.jdbc.xa.XaFacadeImpl.wrapException(XaFacadeImpl.java:353)
> at
> org.apache.flink.connector.jdbc.xa.XaFacadeImpl.access$800(XaFacadeImpl.java:66)
> at
> org.apache.flink.connector.jdbc.xa.XaFacadeImpl$Command.lambda$fromRunnable$0(XaFacadeImpl.java:288)
> at
> org.apache.flink.connector.jdbc.xa.XaFacadeImpl$Command.lambda$fromRunnable$4(XaFacadeImpl.java:327)
> at
> org.apache.flink.connector.jdbc.xa.XaFacadeImpl.execute(XaFacadeImpl.java:267)
> at
> org.apache.flink.connector.jdbc.xa.XaFacadeImpl.start(XaFacadeImpl.java:160)
> at
> org.apache.flink.connector.jdbc.xa.JdbcXaSinkFunction.beginTx(JdbcXaSinkFunction.java:302)
> at
> org.apache.flink.connector.jdbc.xa.JdbcXaSinkFunction.snapshotState(JdbcXaSinkFunction.java:241)
> at
> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.trySnapshotFunctionState(StreamingFunctionUtils.java:118)
> at
> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.snapshotFunctionState(StreamingFunctionUtils.java:99)
> at
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.snapshotState(AbstractUdfStreamOperator.java:89)
> at
> org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:205)
> ... 23 more
> {code}
> I think the scenario is quite predictable because it is how xa transaction
> works.
> The MySQL shell example below behaves quite similar to how JdbcXaSinkFunction
> works.
> {code:java}
> xa start "";
> # Inserting some rows
> # end the current transaction
> xa end "";
> xa prepare "";
> # start a new transaction with the same connection while the previous one is
> PREPARED
> xa prepare "";
> {code}
> This also produces error 'SQL Error [1399] [XAE07]: XAER_RMFAIL: The command
> cannot be executed when global transaction is in the PREPARED state'.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] LadyForest commented on pull request #15174: [FLINK-21651][sql-client] Migrate module related tests in LocalExecutorITCase to the new integration test framework
LadyForest commented on pull request #15174: URL: https://github.com/apache/flink/pull/15174#issuecomment-798834936 > We can also remove createModuleReplaceVars and assertShowResult methods in LocalExecutorITCase. My bad:( I forgot to remove them... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15193: [FLINK-21778][network] Use heap memory instead of direct memory as index entry cache for sort-merge shuffle
flinkbot edited a comment on pull request #15193: URL: https://github.com/apache/flink/pull/15193#issuecomment-798833303 ## CI report: * 4841aec2ecec361d616c7b35aacdf9bb5835 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14607) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-21743) JdbcXaSinkFunction throws XAER_RMFAIL when calling snapshotState and beginTx
[
https://issues.apache.org/jira/browse/FLINK-21743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17301045#comment-17301045
]
Wei Hao commented on FLINK-21743:
-
The config 'max_prepared_transactions' seems to be specific to Postgresql
server, which seems to be an issue in [exactly-once jdbc
implementation|https://github.com/apache/flink/pull/10847].
> JdbcXaSinkFunction throws XAER_RMFAIL when calling snapshotState and beginTx
>
>
> Key: FLINK-21743
> URL: https://issues.apache.org/jira/browse/FLINK-21743
> Project: Flink
> Issue Type: Test
> Components: Connectors / JDBC
>Affects Versions: 1.13.0
> Environment: org.apache.flink:flink-streaming-java_2.11:1.12.1
> org.apache.flink:flink-connector-jdbc_2.11:1.13-SNAPSHOT
>Reporter: Wei Hao
>Assignee: Roman Khachatryan
>Priority: Major
>
> {code:java}
> public void snapshotState(FunctionSnapshotContext context) throws Exception {
> LOG.debug("snapshot state, checkpointId={}", context.getCheckpointId());
> this.rollbackPreparedFromCheckpoint(context.getCheckpointId());
> this.prepareCurrentTx(context.getCheckpointId());
> this.beginTx(context.getCheckpointId() + 1L);
> this.stateHandler.store(JdbcXaSinkFunctionState.of(this.preparedXids,
> this.hangingXids));
> }
> {code}
> When checkpointing starts, it calls snapshotState(), which ends and prepares
> the current transaction. The issue I found is with beginTx(), where a new Xid
> is generated and xaFacade will run command like 'xa start new_xid', which
> will throw the exception as shown below and causes checkpointing failure.
> {code:java}
> Caused by: org.apache.flink.connector.jdbc.xa.XaFacade$TransientXaException:
> com.mysql.cj.jdbc.MysqlXAException: XAER_RMFAIL: The command cannot be
> executed when global transaction is in the PREPARED stateCaused by:
> org.apache.flink.connector.jdbc.xa.XaFacade$TransientXaException:
> com.mysql.cj.jdbc.MysqlXAException: XAER_RMFAIL: The command cannot be
> executed when global transaction is in the PREPARED state at
> org.apache.flink.connector.jdbc.xa.XaFacadeImpl.wrapException(XaFacadeImpl.java:353)
> at
> org.apache.flink.connector.jdbc.xa.XaFacadeImpl.access$800(XaFacadeImpl.java:66)
> at
> org.apache.flink.connector.jdbc.xa.XaFacadeImpl$Command.lambda$fromRunnable$0(XaFacadeImpl.java:288)
> at
> org.apache.flink.connector.jdbc.xa.XaFacadeImpl$Command.lambda$fromRunnable$4(XaFacadeImpl.java:327)
> at
> org.apache.flink.connector.jdbc.xa.XaFacadeImpl.execute(XaFacadeImpl.java:267)
> at
> org.apache.flink.connector.jdbc.xa.XaFacadeImpl.start(XaFacadeImpl.java:160)
> at
> org.apache.flink.connector.jdbc.xa.JdbcXaSinkFunction.beginTx(JdbcXaSinkFunction.java:302)
> at
> org.apache.flink.connector.jdbc.xa.JdbcXaSinkFunction.snapshotState(JdbcXaSinkFunction.java:241)
> at
> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.trySnapshotFunctionState(StreamingFunctionUtils.java:118)
> at
> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.snapshotFunctionState(StreamingFunctionUtils.java:99)
> at
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.snapshotState(AbstractUdfStreamOperator.java:89)
> at
> org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:205)
> ... 23 more
> {code}
> I think the scenario is quite predictable because it is how xa transaction
> works.
> The MySQL shell example below behaves quite similar to how JdbcXaSinkFunction
> works.
> {code:java}
> xa start "";
> # Inserting some rows
> # end the current transaction
> xa end "";
> xa prepare "";
> # start a new transaction with the same connection while the previous one is
> PREPARED
> xa prepare "";
> {code}
> This also produces error 'SQL Error [1399] [XAE07]: XAER_RMFAIL: The command
> cannot be executed when global transaction is in the PREPARED state'.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #15193: [FLINK-21778][network] Use heap memory instead of direct memory as index entry cache for sort-merge shuffle
flinkbot commented on pull request #15193: URL: https://github.com/apache/flink/pull/15193#issuecomment-798833303 ## CI report: * 4841aec2ecec361d616c7b35aacdf9bb5835 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20740) Use managed memory to avoid direct memory OOM error for sort-merge shuffle (introduce a separated buffer pool)
[ https://issues.apache.org/jira/browse/FLINK-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-20740: Summary: Use managed memory to avoid direct memory OOM error for sort-merge shuffle (introduce a separated buffer pool) (was: Use managed memory (network memory) to avoid direct memory OOM error for sort-merge shuffle (introduce a separated buffer pool)) > Use managed memory to avoid direct memory OOM error for sort-merge shuffle > (introduce a separated buffer pool) > -- > > Key: FLINK-20740 > URL: https://issues.apache.org/jira/browse/FLINK-20740 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Network >Affects Versions: 1.12.0 >Reporter: Yingjie Cao >Priority: Major > Fix For: 1.13.0 > > > Currently, sort-merge blocking shuffle uses some unmanaged memory for data > writing and reading, which means users must increase the size of direct > memory, otherwise, one may encounter direct memory OOM error, which is really > bad for usability. > As discussed in the mailing list, a separated buffer pool will be introduced > to be used by the shuffle reading phase, this buffer pool is part of Flink > framework and has different life cycle with the tasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #15193: [FLINK-21778][network] Use heap memory instead of direct memory as index entry cache for sort-merge shuffle
flinkbot commented on pull request #15193: URL: https://github.com/apache/flink/pull/15193#issuecomment-798831053 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 4841aec2ecec361d616c7b35aacdf9bb5835 (Sun Mar 14 04:29:23 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-21778).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-21778) Use heap memory instead of direct memory as index entry cache for sort-merge shuffle
[ https://issues.apache.org/jira/browse/FLINK-21778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-21778: --- Labels: pull-request-available (was: ) > Use heap memory instead of direct memory as index entry cache for sort-merge > shuffle > > > Key: FLINK-21778 > URL: https://issues.apache.org/jira/browse/FLINK-21778 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Network >Affects Versions: 1.12.0 >Reporter: Yingjie Cao >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > > Currently, the sort-merge shuffle implementation uses a piece of direct > memory as index entry cache for acceleration. We can use heap memory instead > to reduce the usage of direct memory which further reduces the possibility of > OutOfMemoryError. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wsry opened a new pull request #15193: [FLINK-21778][network] Use heap memory instead of direct memory as index entry cache for sort-merge shuffle
wsry opened a new pull request #15193: URL: https://github.com/apache/flink/pull/15193 ## What is the purpose of the change Currently, the sort-merge shuffle implementation uses a piece of direct memory as index entry cache for acceleration. This patch switches to heap memory instead to reduce the consumption of direct memory which can further reduce the possibility of OutOfMemoryError. ## Brief change log - Use heap memory instead of direct memory as index entry cache for sort-merge shuffle. ## Verifying this change This change is already covered by existing tests, such as PartitionedFileWriteReadTest, SortMergeResultPartitionTest and BlockingShuffleITCase. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15192: [FLINK-21777][network] Replace the 4M data writing cache of sort-merge shuffle with writev system call
flinkbot edited a comment on pull request #15192: URL: https://github.com/apache/flink/pull/15192#issuecomment-798828549 ## CI report: * b33ebbeb181bb30510dcd28361fb9fff39a140a2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14606) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-21778) Use heap memory instead of direct memory as index entry cache for sort-merge shuffle
Yingjie Cao created FLINK-21778: --- Summary: Use heap memory instead of direct memory as index entry cache for sort-merge shuffle Key: FLINK-21778 URL: https://issues.apache.org/jira/browse/FLINK-21778 Project: Flink Issue Type: Sub-task Components: Runtime / Network Affects Versions: 1.12.0 Reporter: Yingjie Cao Fix For: 1.13.0 Currently, the sort-merge shuffle implementation uses a piece of direct memory as index entry cache for acceleration. We can use heap memory instead to reduce the usage of direct memory which further reduces the possibility of OutOfMemoryError. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #15192: [FLINK-21777][network] Replace the 4M data writing cache of sort-merge shuffle with writev system call
flinkbot commented on pull request #15192: URL: https://github.com/apache/flink/pull/15192#issuecomment-798828549 ## CI report: * b33ebbeb181bb30510dcd28361fb9fff39a140a2 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15192: [FLINK-21777][network] Replace the 4M data writing cache of sort-merge shuffle with writev system call
flinkbot commented on pull request #15192: URL: https://github.com/apache/flink/pull/15192#issuecomment-798827237 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit b33ebbeb181bb30510dcd28361fb9fff39a140a2 (Sun Mar 14 03:40:17 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-21777).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-21777) Replace the 4M data writing cache of sort-merge shuffle with writev system call
[ https://issues.apache.org/jira/browse/FLINK-21777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-21777: --- Labels: pull-request-available (was: ) > Replace the 4M data writing cache of sort-merge shuffle with writev system > call > --- > > Key: FLINK-21777 > URL: https://issues.apache.org/jira/browse/FLINK-21777 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Network >Affects Versions: 1.12.0 >Reporter: Yingjie Cao >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > > Currently, the sort-merge shuffle implementation uses 4M unmanaged direct > memory as cache for data writing. It can be replaced by the writev system > call which can reduce the unmanaged direct memory usage without any > performance loss. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wsry opened a new pull request #15192: [FLINK-21777][network] Replace the 4M data writing cache of sort-merge shuffle with writev system call
wsry opened a new pull request #15192: URL: https://github.com/apache/flink/pull/15192 ## What is the purpose of the change Currently, the sort-merge shuffle implementation uses 4M unmanaged direct memory as cache for data writing. This patch replaces the cache with writev system call which can reduce the unmanaged direct memory usage without any performance loss. ## Brief change log - Remove the data writing cache in PartitionedFileWriter; - Allocate a collection of network buffers as write buffer; - Bulk write the data out when the write buffer is full. ## Verifying this change This change is already covered by existing tests, such as PartitionedFileWriteReadTest, SortMergeResultPartitionTest and BlockingShuffleITCase. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15175: [FLINK-16829] Refactored Prometheus Metric Reporters to use construct…
flinkbot edited a comment on pull request #15175: URL: https://github.com/apache/flink/pull/15175#issuecomment-797871668 ## CI report: * dfe37f37eb20046f59a7bf1aeb2f15cfc1b7a8b7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14605) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15189: [FLINK-21773][metrics][statsd] Remove flink-runtime dependency
flinkbot edited a comment on pull request #15189: URL: https://github.com/apache/flink/pull/15189#issuecomment-798459150 ## CI report: * f7089222a33530ec555b266b06f0d94b238288a9 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14604) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20388) Supports users setting operators' metrics name
[ https://issues.apache.org/jira/browse/FLINK-20388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17301040#comment-17301040 ] tim yu commented on FLINK-20388: Hi [~jark] [~chesnay], What is the conclusion through above comments ? Is the conclusion the following content ? # Introduce a dedicated operator description which we display in the UI when hovering or something. # If the generated operator name is greater than 80 characters, we truncate it simply. The full operator name is assigned to operator description. > Supports users setting operators' metrics name > -- > > Key: FLINK-20388 > URL: https://issues.apache.org/jira/browse/FLINK-20388 > Project: Flink > Issue Type: Improvement > Components: API / DataStream, Runtime / Metrics >Reporter: hailong wang >Priority: Major > > Currently, we only support users setting operators name. > And we use those in the topology to distinguish operators, at the same time, > as the operator metrics name. > If the operator name length is larger than 80, we truncate it simply. > I think we can allow users to set operator metrics name like operators name. > If the user is not set, use the current way. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-21777) Replace the 4M data writing cache of sort-merge shuffle with writev system call
[ https://issues.apache.org/jira/browse/FLINK-21777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-21777: Description: Currently, the sort-merge shuffle implementation uses 4M unmanaged direct memory as cache for data writing. It can be replaced by the writev system call which can reduce the unmanaged direct memory usage without any performance loss. (was: Currently, the sort-merge shuffle implementation uses 4M unmanned direct memory as cache for data written. It can be replaced by the writev system call which can reduce the unmanned direct memory usage without any performance loss.) > Replace the 4M data writing cache of sort-merge shuffle with writev system > call > --- > > Key: FLINK-21777 > URL: https://issues.apache.org/jira/browse/FLINK-21777 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Network >Affects Versions: 1.12.0 >Reporter: Yingjie Cao >Priority: Major > Fix For: 1.13.0 > > > Currently, the sort-merge shuffle implementation uses 4M unmanaged direct > memory as cache for data writing. It can be replaced by the writev system > call which can reduce the unmanaged direct memory usage without any > performance loss. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #15175: [FLINK-16829] Refactored Prometheus Metric Reporters to use construct…
flinkbot edited a comment on pull request #15175: URL: https://github.com/apache/flink/pull/15175#issuecomment-797871668 ## CI report: * 62141a2aff3348088ceb5eb6c449e9abfe5f9870 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14573) * dfe37f37eb20046f59a7bf1aeb2f15cfc1b7a8b7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rionmonster commented on a change in pull request #15175: [FLINK-16829] Refactored Prometheus Metric Reporters to use construct…
rionmonster commented on a change in pull request #15175:
URL: https://github.com/apache/flink/pull/15175#discussion_r593831333
##
File path:
flink-metrics/flink-metrics-prometheus/src/main/java/org/apache/flink/metrics/prometheus/PrometheusPushGatewayReporter.java
##
@@ -51,75 +42,21 @@
"org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporterFactory")
public class PrometheusPushGatewayReporter extends AbstractPrometheusReporter
implements Scheduled {
-private PushGateway pushGateway;
-private String jobName;
-private boolean deleteOnShutdown;
-private Map groupingKey;
-
-@Override
-public void open(MetricConfig config) {
-super.open(config);
-
-String host = config.getString(HOST.key(), HOST.defaultValue());
-int port = config.getInteger(PORT.key(), PORT.defaultValue());
-String configuredJobName = config.getString(JOB_NAME.key(),
JOB_NAME.defaultValue());
-boolean randomSuffix =
-config.getBoolean(
-RANDOM_JOB_NAME_SUFFIX.key(),
RANDOM_JOB_NAME_SUFFIX.defaultValue());
-deleteOnShutdown =
-config.getBoolean(DELETE_ON_SHUTDOWN.key(),
DELETE_ON_SHUTDOWN.defaultValue());
-groupingKey =
-parseGroupingKey(config.getString(GROUPING_KEY.key(),
GROUPING_KEY.defaultValue()));
-
-if (host == null || host.isEmpty() || port < 1) {
-throw new IllegalArgumentException(
-"Invalid host/port configuration. Host: " + host + " Port:
" + port);
-}
-
-if (randomSuffix) {
-this.jobName = configuredJobName + new AbstractID();
-} else {
-this.jobName = configuredJobName;
-}
-
-pushGateway = new PushGateway(host + ':' + port);
-log.info(
-"Configured PrometheusPushGatewayReporter with {host:{},
port:{}, jobName:{}, randomJobNameSuffix:{}, deleteOnShutdown:{},
groupingKey:{}}",
-host,
-port,
-jobName,
-randomSuffix,
-deleteOnShutdown,
-groupingKey);
-}
-
-Map parseGroupingKey(final String groupingKeyConfig) {
-if (!groupingKeyConfig.isEmpty()) {
-Map groupingKey = new HashMap<>();
-String[] kvs = groupingKeyConfig.split(";");
-for (String kv : kvs) {
-int idx = kv.indexOf("=");
-if (idx < 0) {
-log.warn("Invalid prometheusPushGateway groupingKey:{},
will be ignored", kv);
-continue;
-}
-
-String labelKey = kv.substring(0, idx);
-String labelValue = kv.substring(idx + 1);
-if (StringUtils.isNullOrWhitespaceOnly(labelKey)
-|| StringUtils.isNullOrWhitespaceOnly(labelValue)) {
-log.warn(
-"Invalid groupingKey {labelKey:{}, labelValue:{}}
must not be empty",
-labelKey,
-labelValue);
-continue;
-}
-groupingKey.put(labelKey, labelValue);
-}
-
-return groupingKey;
-}
-return Collections.emptyMap();
+private final PushGateway pushGateway;
+private final String jobName;
+private final Map groupingKey;
+private final boolean deleteOnShutdown;
+
+PrometheusPushGatewayReporter(
+@Nullable PushGateway pushGateway,
+@Nullable String jobName,
Review comment:
Updated accordingly (although `pushGateway` is no longer an argument).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] rionmonster commented on a change in pull request #15175: [FLINK-16829] Refactored Prometheus Metric Reporters to use construct…
rionmonster commented on a change in pull request #15175:
URL: https://github.com/apache/flink/pull/15175#discussion_r593831319
##
File path:
flink-metrics/flink-metrics-prometheus/src/main/java/org/apache/flink/metrics/prometheus/PrometheusReporter.java
##
@@ -50,30 +49,11 @@ int getPort() {
return port;
}
-@Override
-public void open(MetricConfig config) {
-super.open(config);
-
-String portsConfig = config.getString(ARG_PORT, DEFAULT_PORT);
-Iterator ports = NetUtils.getPortRangeFromString(portsConfig);
-
-while (ports.hasNext()) {
-int port = ports.next();
-try {
-// internally accesses CollectorRegistry.defaultRegistry
-httpServer = new HTTPServer(port);
-this.port = port;
-log.info("Started PrometheusReporter HTTP server on port {}.",
port);
-break;
-} catch (IOException ioe) { // assume port conflict
-log.debug("Could not start PrometheusReporter HTTP server on
port {}.", port, ioe);
-}
-}
-if (httpServer == null) {
-throw new RuntimeException(
-"Could not start PrometheusReporter HTTP server on any
configured port. Ports: "
-+ portsConfig);
-}
+PrometheusReporter(
+final int port,
+@Nullable HTTPServer httpServer) {
+this.httpServer = Preconditions.checkNotNull(httpServer);
Review comment:
Makes sense, although the `httpServer` is no longer passed in since we
pass in the ports iterator now. Done!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] rionmonster commented on a change in pull request #15175: [FLINK-16829] Refactored Prometheus Metric Reporters to use construct…
rionmonster commented on a change in pull request #15175:
URL: https://github.com/apache/flink/pull/15175#discussion_r593831274
##
File path:
flink-metrics/flink-metrics-prometheus/src/main/java/org/apache/flink/metrics/prometheus/PrometheusReporterFactory.java
##
@@ -17,18 +17,58 @@
package org.apache.flink.metrics.prometheus;
+import io.prometheus.client.exporter.HTTPServer;
+
+import org.apache.flink.metrics.MetricConfig;
import org.apache.flink.metrics.reporter.InterceptInstantiationViaReflection;
import org.apache.flink.metrics.reporter.MetricReporterFactory;
+import org.apache.flink.runtime.metrics.MetricRegistryConfiguration;
+import org.apache.flink.util.NetUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.Iterator;
import java.util.Properties;
/** {@link MetricReporterFactory} for {@link PrometheusReporter}. */
@InterceptInstantiationViaReflection(
reporterClassName =
"org.apache.flink.metrics.prometheus.PrometheusReporter")
public class PrometheusReporterFactory implements MetricReporterFactory {
+private static final Logger LOG =
LoggerFactory.getLogger(PrometheusReporterFactory.class);
+
+static final String ARG_PORT = "port";
+private static final String DEFAULT_PORT = "9249";
+
@Override
public PrometheusReporter createMetricReporter(Properties properties) {
-return new PrometheusReporter();
+MetricConfig metricConfig = (MetricConfig)properties;
+String portsConfig = metricConfig.getString(ARG_PORT, DEFAULT_PORT);
+Iterator ports = NetUtils.getPortRangeFromString(portsConfig);
+Integer port = null;
+HTTPServer httpServer = null;
+while (ports.hasNext()) {
+port = ports.next();
+try {
+// internally accesses CollectorRegistry.defaultRegistry
+httpServer = new HTTPServer(port);
Review comment:
Done!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] rionmonster commented on a change in pull request #15175: [FLINK-16829] Refactored Prometheus Metric Reporters to use construct…
rionmonster commented on a change in pull request #15175:
URL: https://github.com/apache/flink/pull/15175#discussion_r593831263
##
File path:
flink-metrics/flink-metrics-prometheus/src/main/java/org/apache/flink/metrics/prometheus/PrometheusPushGatewayReporterFactory.java
##
@@ -17,18 +17,102 @@
package org.apache.flink.metrics.prometheus;
+import io.prometheus.client.exporter.PushGateway;
+
+import org.apache.flink.metrics.MetricConfig;
import org.apache.flink.metrics.reporter.InterceptInstantiationViaReflection;
import org.apache.flink.metrics.reporter.MetricReporterFactory;
+import org.apache.flink.util.AbstractID;
+import org.apache.flink.util.StringUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
import java.util.Properties;
+import static
org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporterOptions.DELETE_ON_SHUTDOWN;
+import static
org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporterOptions.GROUPING_KEY;
+import static
org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporterOptions.HOST;
+import static
org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporterOptions.JOB_NAME;
+import static
org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporterOptions.PORT;
+import static
org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporterOptions.RANDOM_JOB_NAME_SUFFIX;
+
/** {@link MetricReporterFactory} for {@link PrometheusPushGatewayReporter}. */
@InterceptInstantiationViaReflection(
reporterClassName =
"org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter")
public class PrometheusPushGatewayReporterFactory implements
MetricReporterFactory {
+private static final Logger LOG =
+
LoggerFactory.getLogger(PrometheusPushGatewayReporterFactory.class);
+
@Override
public PrometheusPushGatewayReporter createMetricReporter(Properties
properties) {
-return new PrometheusPushGatewayReporter();
+MetricConfig metricConfig = (MetricConfig)properties;
+
+String host = metricConfig.getString(HOST.key(), HOST.defaultValue());
+int port = metricConfig.getInteger(PORT.key(), PORT.defaultValue());
+String configuredJobName = metricConfig.getString(JOB_NAME.key(),
JOB_NAME.defaultValue());
+boolean randomSuffix =
+metricConfig.getBoolean(
+RANDOM_JOB_NAME_SUFFIX.key(),
RANDOM_JOB_NAME_SUFFIX.defaultValue());
+boolean deleteOnShutdown =
+metricConfig.getBoolean(DELETE_ON_SHUTDOWN.key(),
DELETE_ON_SHUTDOWN.defaultValue());
+Map groupingKey =
+parseGroupingKey(metricConfig.getString(GROUPING_KEY.key(),
GROUPING_KEY.defaultValue()));
+
+if (host == null || host.isEmpty() || port < 1) {
+throw new IllegalArgumentException(
+"Invalid host/port configuration. Host: " + host + " Port:
" + port);
+}
+
+String jobName = configuredJobName;
+if (randomSuffix) {
+jobName = configuredJobName + new AbstractID();
+}
+
+PushGateway pushGateway = new PushGateway(host + ':' + port);
Review comment:
Done! Moved this (specifically the `PushGateway` initialization) into
the constructor as opposed to the factory, which makes total sense. The
`jobName` resolution is still within the factory, but we could adjust that if
you preferred and pass in a flag to generate the name within the constructor.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15182: [FLINK-21761][metrics][influxdb] Remove flink-runtime dependency
flinkbot edited a comment on pull request #15182: URL: https://github.com/apache/flink/pull/15182#issuecomment-798247313 ## CI report: * 90e71e1cacac3b0be387b451caf6b02582d10196 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14603) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-21777) Replace the 4M data writing cache of sort-merge shuffle with writev system call
Yingjie Cao created FLINK-21777: --- Summary: Replace the 4M data writing cache of sort-merge shuffle with writev system call Key: FLINK-21777 URL: https://issues.apache.org/jira/browse/FLINK-21777 Project: Flink Issue Type: Sub-task Components: Runtime / Network Affects Versions: 1.12.0 Reporter: Yingjie Cao Fix For: 1.13.0 Currently, the sort-merge shuffle implementation uses 4M unmanned direct memory as cache for data written. It can be replaced by the writev system call which can reduce the unmanned direct memory usage without any performance loss. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] Wosin commented on pull request #15156: [FLINK-21393] [formats] Implement ParquetAvroInputFormat
Wosin commented on pull request #15156: URL: https://github.com/apache/flink/pull/15156#issuecomment-798814244 Hey :) Thanks for CC. Looks good! One thing to consider, is if there is anything stopping us from going even futher and adding possibility to deserialize `SpecificRecord` instances too ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #15175: [FLINK-16829] Refactored Prometheus Metric Reporters to use construct…
zentol commented on a change in pull request #15175:
URL: https://github.com/apache/flink/pull/15175#discussion_r593813599
##
File path:
flink-metrics/flink-metrics-prometheus/src/main/java/org/apache/flink/metrics/prometheus/PrometheusReporter.java
##
@@ -50,30 +49,11 @@ int getPort() {
return port;
}
-@Override
-public void open(MetricConfig config) {
-super.open(config);
-
-String portsConfig = config.getString(ARG_PORT, DEFAULT_PORT);
-Iterator ports = NetUtils.getPortRangeFromString(portsConfig);
-
-while (ports.hasNext()) {
-int port = ports.next();
-try {
-// internally accesses CollectorRegistry.defaultRegistry
-httpServer = new HTTPServer(port);
-this.port = port;
-log.info("Started PrometheusReporter HTTP server on port {}.",
port);
-break;
-} catch (IOException ioe) { // assume port conflict
-log.debug("Could not start PrometheusReporter HTTP server on
port {}.", port, ioe);
-}
-}
-if (httpServer == null) {
-throw new RuntimeException(
-"Could not start PrometheusReporter HTTP server on any
configured port. Ports: "
-+ portsConfig);
-}
+PrometheusReporter(
+final int port,
+@Nullable HTTPServer httpServer) {
+this.httpServer = Preconditions.checkNotNull(httpServer);
Review comment:
`@Nullable` to us implies that null is an expected case that should be
acceptable. Hence, it doesn't make to annotate it as such if we then right away
enforce that it is in fact not null.
##
File path:
flink-metrics/flink-metrics-prometheus/src/main/java/org/apache/flink/metrics/prometheus/PrometheusReporterFactory.java
##
@@ -17,18 +17,58 @@
package org.apache.flink.metrics.prometheus;
+import io.prometheus.client.exporter.HTTPServer;
+
+import org.apache.flink.metrics.MetricConfig;
import org.apache.flink.metrics.reporter.InterceptInstantiationViaReflection;
import org.apache.flink.metrics.reporter.MetricReporterFactory;
+import org.apache.flink.runtime.metrics.MetricRegistryConfiguration;
+import org.apache.flink.util.NetUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.Iterator;
import java.util.Properties;
/** {@link MetricReporterFactory} for {@link PrometheusReporter}. */
@InterceptInstantiationViaReflection(
reporterClassName =
"org.apache.flink.metrics.prometheus.PrometheusReporter")
public class PrometheusReporterFactory implements MetricReporterFactory {
+private static final Logger LOG =
LoggerFactory.getLogger(PrometheusReporterFactory.class);
+
+static final String ARG_PORT = "port";
+private static final String DEFAULT_PORT = "9249";
+
@Override
public PrometheusReporter createMetricReporter(Properties properties) {
-return new PrometheusReporter();
+MetricConfig metricConfig = (MetricConfig)properties;
+String portsConfig = metricConfig.getString(ARG_PORT, DEFAULT_PORT);
+Iterator ports = NetUtils.getPortRangeFromString(portsConfig);
+Integer port = null;
+HTTPServer httpServer = null;
+while (ports.hasNext()) {
+port = ports.next();
+try {
+// internally accesses CollectorRegistry.defaultRegistry
+httpServer = new HTTPServer(port);
Review comment:
let's move this into the constructor, and have it accept the ports
iterator
##
File path:
flink-metrics/flink-metrics-prometheus/src/main/java/org/apache/flink/metrics/prometheus/PrometheusPushGatewayReporter.java
##
@@ -51,75 +42,21 @@
"org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporterFactory")
public class PrometheusPushGatewayReporter extends AbstractPrometheusReporter
implements Scheduled {
-private PushGateway pushGateway;
-private String jobName;
-private boolean deleteOnShutdown;
-private Map groupingKey;
-
-@Override
-public void open(MetricConfig config) {
-super.open(config);
-
-String host = config.getString(HOST.key(), HOST.defaultValue());
-int port = config.getInteger(PORT.key(), PORT.defaultValue());
-String configuredJobName = config.getString(JOB_NAME.key(),
JOB_NAME.defaultValue());
-boolean randomSuffix =
-config.getBoolean(
-RANDOM_JOB_NAME_SUFFIX.key(),
RANDOM_JOB_NAME_SUFFIX.defaultValue());
-deleteOnShutdown =
-config.getBoolean(DELETE_ON_SHUTDOWN.key(),
DELETE_ON_SHUTDOWN.defaultValue());
-groupingKey =
-parseGroupingKey(config.getString(GROUPING_KEY.key(),
GROUPING_KEY.defaultValue()));
-
-
[GitHub] [flink] flinkbot edited a comment on pull request #15180: [FLINK-21762][metrics][prometheus] Remove flink-runtime dependency
flinkbot edited a comment on pull request #15180: URL: https://github.com/apache/flink/pull/15180#issuecomment-798246847 ## CI report: * ac91363f476031fe77b25495ffe48fae2de45d77 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14602) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15182: [FLINK-21761][metrics][influxdb] Remove flink-runtime dependency
flinkbot edited a comment on pull request #15182: URL: https://github.com/apache/flink/pull/15182#issuecomment-798247313 ## CI report: * 41a479d510718a9403630b8c317b960cd03cf3bc Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14588) * 90e71e1cacac3b0be387b451caf6b02582d10196 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14603) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15189: [FLINK-21773][metrics][statsd] Remove flink-runtime dependency
flinkbot edited a comment on pull request #15189: URL: https://github.com/apache/flink/pull/15189#issuecomment-798459150 ## CI report: * 77bf284b8395d960b0c53425bc6a663cd3103b2a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14596) * f7089222a33530ec555b266b06f0d94b238288a9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14604) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15180: [FLINK-21762][metrics][prometheus] Remove flink-runtime dependency
flinkbot edited a comment on pull request #15180: URL: https://github.com/apache/flink/pull/15180#issuecomment-798246847 ## CI report: * f20c629faf179681a10575c41a96adc16d3459c0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14586) * ac91363f476031fe77b25495ffe48fae2de45d77 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14602) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15189: [FLINK-21773][metrics][statsd] Remove flink-runtime dependency
flinkbot edited a comment on pull request #15189: URL: https://github.com/apache/flink/pull/15189#issuecomment-798459150 ## CI report: * 77bf284b8395d960b0c53425bc6a663cd3103b2a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14596) * f7089222a33530ec555b266b06f0d94b238288a9 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15182: [FLINK-21761][metrics][influxdb] Remove flink-runtime dependency
flinkbot edited a comment on pull request #15182: URL: https://github.com/apache/flink/pull/15182#issuecomment-798247313 ## CI report: * 41a479d510718a9403630b8c317b960cd03cf3bc Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14588) * 90e71e1cacac3b0be387b451caf6b02582d10196 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15180: [FLINK-21762][metrics][prometheus] Remove flink-runtime dependency
flinkbot edited a comment on pull request #15180: URL: https://github.com/apache/flink/pull/15180#issuecomment-798246847 ## CI report: * f20c629faf179681a10575c41a96adc16d3459c0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14586) * ac91363f476031fe77b25495ffe48fae2de45d77 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] LeonBein commented on a change in pull request #15109: [FLINK-20955][connectors/hbase] HBase connector using new connector API
LeonBein commented on a change in pull request #15109: URL: https://github.com/apache/flink/pull/15109#discussion_r593793297 ## File path: flink-connectors/flink-connector-hbase/pom.xml ## @@ -0,0 +1,163 @@ + + +http://maven.apache.org/POM/4.0.0; +xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; +xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> + + flink-connectors + org.apache.flink + 1.13-SNAPSHOT + .. + + 4.0.0 + + flink-connector-hbase-new Review comment: Another consideration was to call this module `flink-connector-hbase-unified`, with `unified` as in [Unified Sink API](https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API) However, we leave this decision open for now This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] LeonBein commented on a change in pull request #15109: [FLINK-20955][connectors/hbase] HBase connector using new connector API
LeonBein commented on a change in pull request #15109:
URL: https://github.com/apache/flink/pull/15109#discussion_r593793004
##
File path:
flink-connectors/flink-connector-hbase/src/test/java/org/apache/flink/connector/hbase/testutil/HBaseTestClusterUtil.java
##
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.hbase.testutil;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.HBaseTestingUtility;
+import org.apache.hadoop.hbase.MiniHBaseCluster;
+import org.apache.hadoop.hbase.StartMiniClusterOption;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.Delete;
+import org.apache.hadoop.hbase.client.HBaseAdmin;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.replication.ReplicationPeerDescription;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hdfs.protocol.HdfsConstants;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.w3c.dom.Document;
+import org.w3c.dom.Element;
+import org.w3c.dom.NodeList;
+import org.xml.sax.SAXException;
+
+import javax.xml.parsers.DocumentBuilderFactory;
+import javax.xml.parsers.ParserConfigurationException;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.List;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+/** Provides static access to a {@link MiniHBaseCluster} for testing. */
+public class HBaseTestClusterUtil {
Review comment:
✅
Done in eec8db2246f521afb66068a509bf3bddebf14e3f
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] LeonBein commented on a change in pull request #15109: [FLINK-20955][connectors/hbase] HBase connector using new connector API
LeonBein commented on a change in pull request #15109:
URL: https://github.com/apache/flink/pull/15109#discussion_r593792660
##
File path:
flink-connectors/flink-connector-hbase/src/main/java/org/apache/flink/connector/hbase/source/enumerator/HBaseSplitEnumerator.java
##
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.hbase.source.enumerator;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SplitEnumerator;
+import org.apache.flink.api.connector.source.SplitEnumeratorContext;
+import org.apache.flink.connector.hbase.source.split.HBaseSourceSplit;
+import org.apache.flink.connector.hbase.util.HBaseConfigurationUtil;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptor;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Queue;
+
+/** The enumerator class for Hbase source. */
+@Internal
+public class HBaseSplitEnumerator
+implements SplitEnumerator> {
+private final SplitEnumeratorContext context;
+private final Queue remainingSplits;
+private final String table;
+private final byte[] serializedConfig;
+
+public HBaseSplitEnumerator(
+SplitEnumeratorContext context,
+byte[] serializedConfig,
+String table) {
+this.context = context;
+this.remainingSplits = new ArrayDeque<>();
+this.table = table;
+this.serializedConfig = serializedConfig;
+}
+
+@Override
+public void start() {
+Configuration hbaseConfiguration =
+
HBaseConfigurationUtil.deserializeConfiguration(this.serializedConfig, null);
+try (Connection connection =
ConnectionFactory.createConnection(hbaseConfiguration);
+Admin admin = connection.getAdmin()) {
+ColumnFamilyDescriptor[] colFamDes =
+
admin.getDescriptor(TableName.valueOf(this.table)).getColumnFamilies();
+List splits = new ArrayList<>();
+
+int parallelism = context.currentParallelism();
+int colFamilies = colFamDes.length;
+
+if (parallelism > colFamilies) {
Review comment:
✅
Done by @caustt in adaa06071dc4a6f74a6fe2fd25f7d5c207515f46
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] LeonBein commented on a change in pull request #15109: [FLINK-20955][connectors/hbase] HBase connector using new connector API
LeonBein commented on a change in pull request #15109:
URL: https://github.com/apache/flink/pull/15109#discussion_r593792517
##
File path:
flink-connectors/flink-connector-hbase/src/test/java/org/apache/flink/connector/hbase/source/HBaseSourceITCase.java
##
@@ -0,0 +1,313 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.hbase.source;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.functions.RichFlatMapFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.connector.hbase.source.reader.HBaseEvent;
+import org.apache.flink.connector.hbase.source.reader.HBaseSourceDeserializer;
+import org.apache.flink.connector.hbase.testutil.FailureSink;
+import org.apache.flink.connector.hbase.testutil.HBaseTestClusterUtil;
+import org.apache.flink.connector.hbase.testutil.Util;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.minicluster.MiniCluster;
+import org.apache.flink.runtime.minicluster.MiniClusterJobClient;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.util.Collector;
+
+import org.apache.hadoop.hbase.client.Put;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.xml.sax.SAXException;
+
+import javax.xml.parsers.ParserConfigurationException;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+
+import static
org.apache.flink.connector.hbase.testutil.FileSignal.awaitSignalThrowOnFailure;
+import static
org.apache.flink.connector.hbase.testutil.FileSignal.awaitSuccess;
+import static
org.apache.flink.connector.hbase.testutil.FileSignal.cleanupFolder;
+import static
org.apache.flink.connector.hbase.testutil.FileSignal.cleanupSignal;
+import static org.apache.flink.connector.hbase.testutil.FileSignal.makeFolder;
+import static org.apache.flink.connector.hbase.testutil.FileSignal.signal;
+import static
org.apache.flink.connector.hbase.testutil.FileSignal.signalFailure;
+import static
org.apache.flink.connector.hbase.testutil.FileSignal.signalSuccess;
+import static org.junit.Assert.assertArrayEquals;
+
+/** Tests the most basic use cases of the source with a mocked HBase system. */
+public class HBaseSourceITCase extends TestsWithTestHBaseCluster {
+
+private DataStream streamFromHBaseSource(
+StreamExecutionEnvironment environment, String tableName)
+throws ParserConfigurationException, SAXException, IOException {
+return streamFromHBaseSource(environment, tableName, 1);
+}
+
+private DataStream streamFromHBaseSource(
+StreamExecutionEnvironment environment, String tableName, int
parallelism)
+throws ParserConfigurationException, SAXException, IOException {
+HBaseStringDeserializationScheme deserializationScheme =
+new HBaseStringDeserializationScheme();
+HBaseSource source =
+new HBaseSource<>(deserializationScheme, tableName,
cluster.getConfig());
+environment.setParallelism(parallelism);
+DataStream stream =
+environment.fromSource(
+source,
+WatermarkStrategy.noWatermarks(),
+"hbaseSourceITCase",
+deserializationScheme.getProducedType());
+return stream;
+}
+
+private static void expectFirstValuesToBe(
+DataStream stream, T[] expectedValues, String message) {
+
+List collectedValues = new ArrayList<>();
+stream.flatMap(
+new RichFlatMapFunction() {
+
+@Override
+public void flatMap(T value, Collector out) {
+System.out.println("Test collected: " + value);
+collectedValues.add(value);
+if (collectedValues.size() ==
[GitHub] [flink] flinkbot edited a comment on pull request #15191: [FLINK-21776][metrics] Migrate JobManagerJobMG instatiations to factory method
flinkbot edited a comment on pull request #15191: URL: https://github.com/apache/flink/pull/15191#issuecomment-798521197 ## CI report: * 4759dbafe9cf13ab47617dba7bac08703418422e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14598) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15190: [FLINK-21775][metrics] Migrate JobManagerMG instatiations to factory method
flinkbot edited a comment on pull request #15190: URL: https://github.com/apache/flink/pull/15190#issuecomment-798496630 ## CI report: * 694f77c5efd6d4bd53279451152fe9e311463e6e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14597) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15189: [FLINK-21773][metrics][slf4j] Remove flink-runtime dependency
flinkbot edited a comment on pull request #15189: URL: https://github.com/apache/flink/pull/15189#issuecomment-798459150 ## CI report: * 77bf284b8395d960b0c53425bc6a663cd3103b2a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14596) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15188: [FLINK-21772][metrics][slf4j] Remove flink-runtime dependency
flinkbot edited a comment on pull request #15188: URL: https://github.com/apache/flink/pull/15188#issuecomment-798424651 ## CI report: * 6b3c0efce0ab7cd55e1023d08ef4e4077c7bfd27 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14595) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15186: [FLINK-21770][metrics] JobManagerMG#addJob accepts job ID and name
flinkbot edited a comment on pull request #15186: URL: https://github.com/apache/flink/pull/15186#issuecomment-798412260 ## CI report: * f0997a4fb9540f80536f86175e7b37095d375231 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14593) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15187: [FLINK-21771][metrics][tests] Replace StreamTaskTestHarness#TestTaskMetricGroup
flinkbot edited a comment on pull request #15187: URL: https://github.com/apache/flink/pull/15187#issuecomment-798412370 ## CI report: * b3112ce7fa1b1955362ef7b3a0b2de15bb1a197f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14594) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15179: [FLINK-21757][metrics] Add LogicalScopeProvider interface
flinkbot edited a comment on pull request #15179: URL: https://github.com/apache/flink/pull/15179#issuecomment-798246629 ## CI report: * a3b1f3391fea3be3c40c92c95b0aab51398a91ad Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14585) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14943: [FLINK-21354][statebackend] Introduce ChangelogStateBackend to delegate state access
flinkbot edited a comment on pull request #14943: URL: https://github.com/apache/flink/pull/14943#issuecomment-779315285 ## CI report: * f80b97c479ec82663e971553c54c8a6cda3122ff UNKNOWN * 8584b9ca7b9cf29f0f845aafe422f356e068cfe6 UNKNOWN * 8e3149727880b6f2d8ce7f2b851f51be5b341b49 UNKNOWN * 990cd12a4db5d50780c607576f883515c81cdcd2 UNKNOWN * 8fb00ef123f5b348a16032840bf51241b3ee9a57 UNKNOWN * 18275c13b2db6a16aaf5a471e1d075690daf0c35 UNKNOWN * ead18d44ff0ad3632a074dfb3261e2ce5cc84dd4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14583) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15183: [FLINK-21764][metrics][dropwizard] Remove flink-runtime dependency
flinkbot edited a comment on pull request #15183: URL: https://github.com/apache/flink/pull/15183#issuecomment-798304217 ## CI report: * b1cdadcdfcd2f3ba4aacbe70187795db2657e4a8 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14590) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15184: [FLINK-21766][metrics] Remove OperatorMetricGroup#parent()
flinkbot edited a comment on pull request #15184: URL: https://github.com/apache/flink/pull/15184#issuecomment-798333157 ## CI report: * cd4c5c934602f7c6b04bfe4657a27919341e0eec Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14591) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-21466) Make "embedded" parameter optional when start sql-client.sh
[
https://issues.apache.org/jira/browse/FLINK-21466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17300919#comment-17300919
]
zck commented on FLINK-21466:
-
[~jark]
plz check my thoughts. just set mode parameter as the default value "embedded"?
But it cant "-i init1.sql,init2.sql",i will do a function?
> Make "embedded" parameter optional when start sql-client.sh
> ---
>
> Key: FLINK-21466
> URL: https://issues.apache.org/jira/browse/FLINK-21466
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Shengkai Fang
>Assignee: zck
>Priority: Major
> Labels: starter
>
> Users can use command
> {code:java}
> >./sql-client.sh embedded -i init1.sql,init2.sql
> {code}
> to start the sql client and don't need to add {{embedded}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #15181: [FLINK-21760][metrics][jmx] Remove flink-runtime dependency
flinkbot edited a comment on pull request #15181: URL: https://github.com/apache/flink/pull/15181#issuecomment-798247104 ## CI report: * 6e7942b9c3bb252df92eda86ec0bbef38dc2a3d2 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14587) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15191: [FLINK-21776][metrics] Migrate JobManagerJobMG instatiations to factory method
flinkbot edited a comment on pull request #15191: URL: https://github.com/apache/flink/pull/15191#issuecomment-798521197 ## CI report: * 4759dbafe9cf13ab47617dba7bac08703418422e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14598) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15190: [FLINK-21775][metrics] Migrate JobManagerMG instatiations to factory method
flinkbot edited a comment on pull request #15190: URL: https://github.com/apache/flink/pull/15190#issuecomment-798496630 ## CI report: * 694f77c5efd6d4bd53279451152fe9e311463e6e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14597) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15178: [FLINK-21756][jmx] Move JMXServer to flink-core
flinkbot edited a comment on pull request #15178: URL: https://github.com/apache/flink/pull/15178#issuecomment-798246427 ## CI report: * f3a1184b3981ad9a087d83c0ee784c5dd719d310 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14584) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15191: [FLINK-21776][metrics] Migrate JobManagerJobMG instatiations to factory method
flinkbot commented on pull request #15191: URL: https://github.com/apache/flink/pull/15191#issuecomment-798521197 ## CI report: * 4759dbafe9cf13ab47617dba7bac08703418422e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15190: [FLINK-21775][metrics] Migrate JobManagerMG instatiations to factory method
flinkbot commented on pull request #15190: URL: https://github.com/apache/flink/pull/15190#issuecomment-798496630 ## CI report: * 694f77c5efd6d4bd53279451152fe9e311463e6e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15177: [FLINK-21759][metrics][tests] Add TestMetricGroup
flinkbot edited a comment on pull request #15177: URL: https://github.com/apache/flink/pull/15177#issuecomment-798218232 ## CI report: * b099399b0f17a96f7ab6116c056577762f81771d UNKNOWN * e3c6775999035e893e0e0360eaa7862e5d967f5d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14581) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15176: [FLINK-21758][metrics] Add default implementations for int methods
flinkbot edited a comment on pull request #15176: URL: https://github.com/apache/flink/pull/15176#issuecomment-798218009 ## CI report: * f1edd8148ee94d5a45b1f2842549f81ec6fec234 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14580) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-21626) Consider shaping newly introduced RuntimeContext.getJobId to return JobID with no Optional wrapper
[
https://issues.apache.org/jira/browse/FLINK-21626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kezhu Wang updated FLINK-21626:
---
Fix Version/s: 1.13.0
> Consider shaping newly introduced RuntimeContext.getJobId to return JobID
> with no Optional wrapper
> --
>
> Key: FLINK-21626
> URL: https://issues.apache.org/jira/browse/FLINK-21626
> Project: Flink
> Issue Type: Improvement
> Components: API / Core
>Affects Versions: 1.13.0
>Reporter: Kezhu Wang
>Priority: Major
> Fix For: 1.13.0
>
>
> Currently, this newly introduced {{RuntimeContext.getJobId()}} returns
> {{Optional}}. The only path where it returns no job id is
> {{RuntimeUDFContext}}(through {{CollectionExecutor}} through
> {{CollectionEnvironment}}).
> But after {{DataSet}} dropped, there will be no paths to return no job id.
> Both FLINK-21581 and [my
> comment|https://github.com/apache/flink/pull/15053#issuecomment-789410967]
> raised this concern. But different with FLINK-21581, I think we could return
> an environment/executor/plan level unique job id in {{RuntimeUDFContext}} for
> this new api. This way there will be no breaking change after {{DataSet}}
> dropped. And more importantly, a careful chosen job id does not hurt callers
> of {{RuntimeUDFContext}} in my opinion.
> cc [~chesnay] [~roman_khachatryan] [~aljoscha] [~sewen]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21578) Closeable Sink Committer/GlobalCommitter were created to function in onestep during job graph composition
[
https://issues.apache.org/jira/browse/FLINK-21578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17300887#comment-17300887
]
Kezhu Wang commented on FLINK-21578:
[~gaoyunhaii] Not exactly. The problem is interference between {{createXyz}}
and invocation of {{createXyz}} in
{{SinkTransformationTranslator.createCommitter}}.
{{SinkTransformationTranslator}} is called in job composition phase, there
should be not resource consuming procedure.
This is should be considered as bug(or flaw or something else) in api.
> Closeable Sink Committer/GlobalCommitter were created to function in onestep
> during job graph composition
> -
>
> Key: FLINK-21578
> URL: https://issues.apache.org/jira/browse/FLINK-21578
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.13.0
>Reporter: Kezhu Wang
>Priority: Major
>
> Normally, functions/operators are created in job graph composition phase for
> serialization and transmission. Them are "opened" in flink cluster to
> function. This two steps procedure succeed in that there will be no
> resource-cleanup requirement in job graph composition phase.
> While {{Committer}} and {{GlobalCommitter}} has no such "open" operatin but
> they were created in job graph composition phase.
> Following are fixes I could image if we converge to "this is problematic".
> # Add {{open}} or similar method for these two classes.
> # Add {{hasCommitter}}, {{hasGlobalCommitter}} to {{Sink}} and make
> {{createCommitter}} and others not optional(enforce this in runtime).
> Personally, I am a bit preferring second approach for possible less code path
> touching in job graph composition phase. But first approach has advantage
> that it could be an no breaking change.
> There might be other approaches though.
> cc [~guoweima] [~gaoyunhaii] [~aljoscha] [~kkl0u]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #15191: [FLINK-21776][metrics] Migrate JobManagerJobMG instatiations to factory method
flinkbot commented on pull request #15191: URL: https://github.com/apache/flink/pull/15191#issuecomment-798481211 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 4759dbafe9cf13ab47617dba7bac08703418422e (Sat Mar 13 15:00:49 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15189: [FLINK-21773][metrics][slf4j] Remove flink-runtime dependency
flinkbot edited a comment on pull request #15189: URL: https://github.com/apache/flink/pull/15189#issuecomment-798459150 ## CI report: * 77bf284b8395d960b0c53425bc6a663cd3103b2a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14596) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol opened a new pull request #15191: [FLINK-21776][metrics] Migrate JobManagerJobMG instatiations to factory method
zentol opened a new pull request #15191: URL: https://github.com/apache/flink/pull/15191 Based on #15190. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15190: [FLINK-21775][metrics] Migrate JobManagerMG instatiations to factory method
flinkbot commented on pull request #15190: URL: https://github.com/apache/flink/pull/15190#issuecomment-798478026 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 694f77c5efd6d4bd53279451152fe9e311463e6e (Sat Mar 13 14:57:20 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14841: [FLINK-21232][YARN][Common] Introduce pluggable Hadoop delegation token providers
flinkbot edited a comment on pull request #14841: URL: https://github.com/apache/flink/pull/14841#issuecomment-772182984 ## CI report: * 60e427103cca73ac1fc29fb18893e3996edb1c7a UNKNOWN * 83aa947d7ab9528a7c8b3418316066b19f0e6916 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14579) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-21776) Migrate all JobManagerJobMG instantiations to factory method
Chesnay Schepler created FLINK-21776: Summary: Migrate all JobManagerJobMG instantiations to factory method Key: FLINK-21776 URL: https://issues.apache.org/jira/browse/FLINK-21776 Project: Flink Issue Type: Sub-task Components: Runtime / Metrics Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.13.0 Modify all existing usages of the JobManagerMG constructor to use the (to-be-moved) MetricUtils factory method, for consistency and to make constructor changes easier. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-21776) Migrate all JobManagerJobMG instantiations to factory method
[ https://issues.apache.org/jira/browse/FLINK-21776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-21776: - Description: Modify all existing usages of the JobManageJobrMG constructor to use runtime Apis, for consistency and to make constructor changes easier. (was: Modify all existing usages of the JobManagerMG constructor to use the (to-be-moved) MetricUtils factory method, for consistency and to make constructor changes easier.) > Migrate all JobManagerJobMG instantiations to factory method > > > Key: FLINK-21776 > URL: https://issues.apache.org/jira/browse/FLINK-21776 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Metrics >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > > Modify all existing usages of the JobManageJobrMG constructor to use runtime > Apis, for consistency and to make constructor changes easier. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-21775) Migrate all JobManagerMG instantiations to factory method
[ https://issues.apache.org/jira/browse/FLINK-21775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-21775: --- Labels: pull-request-available (was: ) > Migrate all JobManagerMG instantiations to factory method > - > > Key: FLINK-21775 > URL: https://issues.apache.org/jira/browse/FLINK-21775 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Metrics >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > > Modify all existing usages of the JobManagerMG constructor to use the > (to-be-moved) MetricUtils factory method, for consistency and to make > constructor changes easier. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] zentol opened a new pull request #15190: [FLINK-21775][metrics] Migrate JobManagerMG instatiations to factory method
zentol opened a new pull request #15190: URL: https://github.com/apache/flink/pull/15190 Based on #15186. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-21775) Migrate all JobManagerMG instantiations to factory method
Chesnay Schepler created FLINK-21775: Summary: Migrate all JobManagerMG instantiations to factory method Key: FLINK-21775 URL: https://issues.apache.org/jira/browse/FLINK-21775 Project: Flink Issue Type: Sub-task Components: Runtime / Metrics Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.13.0 Modify all existing usages of the JobManagerMG constructor to use the (to-be-moved) MetricUtils factory method, for consistency and to make constructor changes easier. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #15189: [FLINK-21773][metrics][slf4j] Remove flink-runtime dependency
flinkbot commented on pull request #15189: URL: https://github.com/apache/flink/pull/15189#issuecomment-798459150 ## CI report: * 77bf284b8395d960b0c53425bc6a663cd3103b2a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15188: [FLINK-21772][metrics][slf4j] Remove flink-runtime dependency
flinkbot edited a comment on pull request #15188: URL: https://github.com/apache/flink/pull/15188#issuecomment-798424651 ## CI report: * 6b3c0efce0ab7cd55e1023d08ef4e4077c7bfd27 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14595) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-21774) Do not display column names when retrun set is emtpy in SQL Client
Jark Wu created FLINK-21774:
---
Summary: Do not display column names when retrun set is emtpy in
SQL Client
Key: FLINK-21774
URL: https://issues.apache.org/jira/browse/FLINK-21774
Project: Flink
Issue Type: Sub-task
Components: Table SQL / Client
Reporter: Jark Wu
Currently, SQL Client will display column names even if the return set is empty:
{code}
SHOW MODULES;
+-+
| module name |
+-+
0 row in set
!ok
{code}
In mature databases, e.g. MySQL, they only show "Empty Set" instead of column
names:
{code}
mysql> show tables;
Empty set (0.00 sec)
{code}
We can improve this by simply omit the column names header.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] wuchong commented on pull request #15174: [FLINK-21651][sql-client] Migrate module related tests in LocalExecutorITCase to the new integration test framework
wuchong commented on pull request #15174: URL: https://github.com/apache/flink/pull/15174#issuecomment-798430818 @fsk119 I think the branch alread based on the lastest master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15188: [FLINK-21772][metrics][slf4j] Remove flink-runtime dependency
flinkbot commented on pull request #15188: URL: https://github.com/apache/flink/pull/15188#issuecomment-798424651 ## CI report: * 6b3c0efce0ab7cd55e1023d08ef4e4077c7bfd27 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15186: [FLINK-21770][metrics] JobManagerMG#addJob accepts job ID and name
flinkbot edited a comment on pull request #15186: URL: https://github.com/apache/flink/pull/15186#issuecomment-798412260 ## CI report: * f0997a4fb9540f80536f86175e7b37095d375231 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14593) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15187: [FLINK-21771][metrics][tests] Replace StreamTaskTestHarness#TestTaskMetricGroup
flinkbot edited a comment on pull request #15187: URL: https://github.com/apache/flink/pull/15187#issuecomment-798412370 ## CI report: * b3112ce7fa1b1955362ef7b3a0b2de15bb1a197f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14594) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15189: [FLINK-21773][metrics][slf4j] Remove flink-runtime dependency
flinkbot commented on pull request #15189: URL: https://github.com/apache/flink/pull/15189#issuecomment-798422527 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 77bf284b8395d960b0c53425bc6a663cd3103b2a (Sat Mar 13 14:10:35 UTC 2021) **Warnings:** * **1 pom.xml files were touched**: Check for build and licensing issues. * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-21773) Remove dependency from flink-metrics-statsd
[ https://issues.apache.org/jira/browse/FLINK-21773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-21773: --- Labels: pull-request-available (was: ) > Remove dependency from flink-metrics-statsd > --- > > Key: FLINK-21773 > URL: https://issues.apache.org/jira/browse/FLINK-21773 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Metrics >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] zentol opened a new pull request #15189: [FLINK-21773][metrics][slf4j] Remove flink-runtime dependency
zentol opened a new pull request #15189: URL: https://github.com/apache/flink/pull/15189 Based on #15177. - reworks the tests to work directly against the reporter This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15188: [FLINK-21772][metrics][slf4j] Remove flink-runtime dependency
flinkbot commented on pull request #15188: URL: https://github.com/apache/flink/pull/15188#issuecomment-798417786 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 6b3c0efce0ab7cd55e1023d08ef4e4077c7bfd27 (Sat Mar 13 14:02:00 UTC 2021) **Warnings:** * **1 pom.xml files were touched**: Check for build and licensing issues. * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-21773) Remove dependency from flink-metrics-statsd
Chesnay Schepler created FLINK-21773: Summary: Remove dependency from flink-metrics-statsd Key: FLINK-21773 URL: https://issues.apache.org/jira/browse/FLINK-21773 Project: Flink Issue Type: Sub-task Components: Runtime / Metrics Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.13.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-21772) Remove dependency from flink-metrics-slf4j
[ https://issues.apache.org/jira/browse/FLINK-21772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-21772: --- Labels: pull-request-available (was: ) > Remove dependency from flink-metrics-slf4j > -- > > Key: FLINK-21772 > URL: https://issues.apache.org/jira/browse/FLINK-21772 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Metrics >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] zentol opened a new pull request #15188: [FLINK-21772][metrics][slf4j] Remove flink-runtime dependency
zentol opened a new pull request #15188: URL: https://github.com/apache/flink/pull/15188 Based on #15177. - reworks the tests to work directly against the reporter This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-21578) Closeable Sink Committer/GlobalCommitter were created to function in onestep during job graph composition
[
https://issues.apache.org/jira/browse/FLINK-21578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17300863#comment-17300863
]
Yun Gao commented on FLINK-21578:
-
Hi [~kezhuw], sorry I might not fully got the issue, are you thinking that
booting resources in `createXX` has problems ?
For example, I think the pattern would be like
{code:java}
class XXSink implements Sink {
...
XXCommitter createCommiter() {
ResourceA a = new ResourceA();
ResourceB b = new ResourceB();
return new XXCommitter(a, b);
}
}
class XXCommitter implements Committer {
private final ResourceA a;
private final ResourceB b;
public XXCommitter(ResourceA a, ResourceB b) {
...
}
void commit(...) {
...
}
@Override
void close() {
a.close();
b.close();
}
}
{code}
> Closeable Sink Committer/GlobalCommitter were created to function in onestep
> during job graph composition
> -
>
> Key: FLINK-21578
> URL: https://issues.apache.org/jira/browse/FLINK-21578
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.13.0
>Reporter: Kezhu Wang
>Priority: Major
>
> Normally, functions/operators are created in job graph composition phase for
> serialization and transmission. Them are "opened" in flink cluster to
> function. This two steps procedure succeed in that there will be no
> resource-cleanup requirement in job graph composition phase.
> While {{Committer}} and {{GlobalCommitter}} has no such "open" operatin but
> they were created in job graph composition phase.
> Following are fixes I could image if we converge to "this is problematic".
> # Add {{open}} or similar method for these two classes.
> # Add {{hasCommitter}}, {{hasGlobalCommitter}} to {{Sink}} and make
> {{createCommitter}} and others not optional(enforce this in runtime).
> Personally, I am a bit preferring second approach for possible less code path
> touching in job graph composition phase. But first approach has advantage
> that it could be an no breaking change.
> There might be other approaches though.
> cc [~guoweima] [~gaoyunhaii] [~aljoscha] [~kkl0u]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #15187: [FLINK-21771][metrics][tests] Replace StreamTaskTestHarness#TestTaskMetricGroup
flinkbot commented on pull request #15187: URL: https://github.com/apache/flink/pull/15187#issuecomment-798412370 ## CI report: * b3112ce7fa1b1955362ef7b3a0b2de15bb1a197f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15186: [FLINK-21770][metrics] JobManagerMG#addJob accepts job ID and name
flinkbot commented on pull request #15186: URL: https://github.com/apache/flink/pull/15186#issuecomment-798412260 ## CI report: * f0997a4fb9540f80536f86175e7b37095d375231 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-21772) Remove dependency from flink-metrics-slf4j
Chesnay Schepler created FLINK-21772: Summary: Remove dependency from flink-metrics-slf4j Key: FLINK-21772 URL: https://issues.apache.org/jira/browse/FLINK-21772 Project: Flink Issue Type: Sub-task Components: Runtime / Metrics Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.13.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #15187: [FLINK-21771][metrics][tests] Replace StreamTaskTestHarness#TestTaskMetricGroup
flinkbot commented on pull request #15187: URL: https://github.com/apache/flink/pull/15187#issuecomment-798403944 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit b3112ce7fa1b1955362ef7b3a0b2de15bb1a197f (Sat Mar 13 13:46:38 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-21771) Replace StreamTaskTestHarness#TestTaskMetricGroup
[ https://issues.apache.org/jira/browse/FLINK-21771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-21771: --- Labels: pull-request-available (was: ) > Replace StreamTaskTestHarness#TestTaskMetricGroup > - > > Key: FLINK-21771 > URL: https://issues.apache.org/jira/browse/FLINK-21771 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Metrics, Runtime / Task, Tests >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > > Reduce reliance on MetricGroup constructors by creating the metric groups > through the usual API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
