[GitHub] [flink] flinkbot commented on issue #9796: [FLINK-14253][table-planner-blink] Add hash distribution and sort grouping only when dynamic partition insert
flinkbot commented on issue #9796: [FLINK-14253][table-planner-blink] Add hash distribution and sort grouping only when dynamic partition insert URL: https://github.com/apache/flink/pull/9796#issuecomment-536253430 ## CI report: * 3b84408bb5ee764b1a64103b1f4df3e0ceacaba7 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] hequn8128 commented on a change in pull request #9766: [FLINK-14018][python] Add Python building blocks to make sure the basic functionality of Python ScalarFunction could work
hequn8128 commented on a change in pull request #9766: [FLINK-14018][python] Add Python building blocks to make sure the basic functionality of Python ScalarFunction could work URL: https://github.com/apache/flink/pull/9766#discussion_r329333978 ## File path: flink-python/pyflink/fn_execution/coders.py ## @@ -0,0 +1,67 @@ + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +from apache_beam.coders import Coder, VarIntCoder +from apache_beam.coders.coders import FastCoder + +from pyflink.fn_execution import coder_impl +from pyflink.fn_execution import flink_fn_execution_pb2 + +FLINK_SCHEMA_CODER_URN = "flink:coder:schema:v1" + + +class RowCoder(FastCoder): +""" +Coder for Row. +""" + +def __init__(self, field_coders): +self._field_coders = field_coders + +def _create_impl(self): +return coder_impl.RowCoderImpl([c.get_impl() for c in self._field_coders]) + +def is_deterministic(self): +return all(c.is_deterministic() for c in self._field_coders) + +def to_type_hint(self): +from pyflink.table import Row +return Row + +def __repr__(self): Review comment: Also implement `__eq__` and `__hash__` methods? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] hequn8128 commented on a change in pull request #9766: [FLINK-14018][python] Add Python building blocks to make sure the basic functionality of Python ScalarFunction could work
hequn8128 commented on a change in pull request #9766: [FLINK-14018][python] Add Python building blocks to make sure the basic functionality of Python ScalarFunction could work URL: https://github.com/apache/flink/pull/9766#discussion_r329334111 ## File path: flink-python/pyflink/fn_execution/coders.py ## @@ -0,0 +1,67 @@ + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +from apache_beam.coders import Coder, VarIntCoder +from apache_beam.coders.coders import FastCoder + +from pyflink.fn_execution import coder_impl +from pyflink.fn_execution import flink_fn_execution_pb2 + +FLINK_SCHEMA_CODER_URN = "flink:coder:schema:v1" + Review comment: `__all__ = ['RowCoder']` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14179) Wrong description of SqlCommand.SHOW_FUNCTIONS
[
https://issues.apache.org/jira/browse/FLINK-14179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940248#comment-16940248
]
Canbin Zheng commented on FLINK-14179:
--
[~Tison] [~jark] It's a quite minor change, could any take a look to merge then
close this ticket.
> Wrong description of SqlCommand.SHOW_FUNCTIONS
> --
>
> Key: FLINK-14179
> URL: https://issues.apache.org/jira/browse/FLINK-14179
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.9.0
>Reporter: Canbin Zheng
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Attachments: image-2019-09-24-10-59-26-286.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently '*SHOW FUNCTIONS*' lists not only user-defined functions, but also
> system-defined ones, the description {color:#172b4d}*'Shows all registered
> user-defined functions.'* not correctly depicts this functionality. I think
> we can change the description to '*Shows all system-defined and user-defined
> functions.*'{color}
>
> {color:#172b4d}!image-2019-09-24-10-59-26-286.png!{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on issue #9796: [FLINK-14253][table-planner-blink] Add hash distribution and sort grouping only when dynamic partition insert
flinkbot commented on issue #9796: [FLINK-14253][table-planner-blink] Add hash distribution and sort grouping only when dynamic partition insert URL: https://github.com/apache/flink/pull/9796#issuecomment-536252529 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 3b84408bb5ee764b1a64103b1f4df3e0ceacaba7 (Sun Sep 29 05:45:52 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-14253).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-14253) Add hash distribution and sort grouping only when dynamic partition insert
[ https://issues.apache.org/jira/browse/FLINK-14253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-14253: --- Labels: pull-request-available (was: ) > Add hash distribution and sort grouping only when dynamic partition insert > -- > > Key: FLINK-14253 > URL: https://issues.apache.org/jira/browse/FLINK-14253 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Jingsong Lee >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > > Now in BatchExecSinkRule, we don't have static partitions, if it is a > partitioned table, will add hash distribution and sort grouping. It is wrong: > # Group only when dynamic partition insert (not all partition fields are > static partitions) > # We can just hash and sort dynamic partition fields instead of all > partition fields. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] JingsongLi opened a new pull request #9796: [FLINK-14253][table-planner-blink] Add hash distribution and sort grouping only when dynamic partition insert
JingsongLi opened a new pull request #9796: [FLINK-14253][table-planner-blink] Add hash distribution and sort grouping only when dynamic partition insert URL: https://github.com/apache/flink/pull/9796 ## What is the purpose of the change Now in BatchExecSinkRule, we don't have static partitions, if it is a partitioned table, will add hash distribution and sort grouping. It is wrong: 1. Group only when dynamic partition insert (not all partition fields are static partitions) 2. We can just hash and sort dynamic partition fields instead of all partition fields. ## Verifying this change stream and batch PartitionedSinkTest ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9762: !IGNORE! ZK 3.5.5 shaded migration
flinkbot edited a comment on issue #9762: !IGNORE! ZK 3.5.5 shaded migration URL: https://github.com/apache/flink/pull/9762#issuecomment-534689957 ## CI report: * cefac57542dbd85a6693f6a1b7c8c90d91e85d30 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/128980426) * f04ce4e35191731da657f170c6949ba70d87b6e3 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129575680) * 0250869c17414f0c0047b34fdf6a6ad4b57bf017 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129576444) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9762: !IGNORE! ZK 3.5.5 shaded migration
flinkbot edited a comment on issue #9762: !IGNORE! ZK 3.5.5 shaded migration URL: https://github.com/apache/flink/pull/9762#issuecomment-534689957 ## CI report: * cefac57542dbd85a6693f6a1b7c8c90d91e85d30 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/128980426) * f04ce4e35191731da657f170c6949ba70d87b6e3 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129575680) * 0250869c17414f0c0047b34fdf6a6ad4b57bf017 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-14268) YARN AM endless restarts when using wrong checkpoint path or wrong checkpoint
[
https://issues.apache.org/jira/browse/FLINK-14268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lsw_aka_laplace updated FLINK-14268:
Description:
I tried to start a streaming task and restore from checkpoint which it was
stored in HDFS.
I set a wrong checkpoint path and sth unexpected happened: YARN AM restarted
again and again. Since we have already set some restart strategy to prevent
endless restart, it should have been restarted with limited times.
Since we made sure that restart strategy works, we dived into source code and
did some change mainly in _ClusterEntrypoint_.
{code:java}
//代码占位符
//before
@Override
public void onFatalError(Throwable exception) {
LOG.error("Fatal error occurred in the cluster entrypoint.", exception);
System.exit(RUNTIME_FAILURE_RETURN_CODE);
}
//after
@Override
public void onFatalError(Throwable exception) {
LOG.error("Fatal error occurred in the cluster entrypoint.", exception);
if(ExceptionUtils.findThrowable(exception,PerJobFatalException.class).isPresent()){
//PerJobFatalException is the FLAG
//在perjob模式有些致命的异常出现，am会一直重启，不能失败掉
LOG.error("perjob fatal error");
System.exit(STARTUP_FAILURE_RETURN_CODE);
}
System.exit(RUNTIME_FAILURE_RETURN_CODE);
}
{code}
We forced to make the FAILURE_RETURN_CODE as STARTUP_FAILURE_RETURN_CODE
rather than RUNTIME_FAILURE_RETURN_CODE in some condition and *it DID WORK*.
After discussing with [~Tison], I knew that FAILURE_RETURN_CODE seems only to
be used to debug, so I submitted this issue and look forward to ANY solution~
was:
I tried to start a streaming task and restore from checkpoint which it was
stored in HDFS.
I set a wrong checkpoint path and sth unexpected happened: YARN AM restarted
again and again. Since we have already set some restart strategy to prevent
endless restart, it should have been restarted with limited times.
Since we made sure that restart strategy works, we dived into source code and
did some change mainly in _ClusterEntrypoint_.
{code:java}
//代码占位符
//before
@Override
public void onFatalError(Throwable exception) {
LOG.error("Fatal error occurred in the cluster entrypoint.", exception);
System.exit(RUNTIME_FAILURE_RETURN_CODE);
}
//after
@Override
public void onFatalError(Throwable exception) {
LOG.error("Fatal error occurred in the cluster entrypoint.", exception);
// PerJobFatalException is the FLAG
if(ExceptionUtils.findThrowable(exception,PerJobFatalException.class).isPresent()){//在perjob模式有些致命的异常出现，am会一直重启，不能失败掉
LOG.error("perjob fatal error");
System.exit(STARTUP_FAILURE_RETURN_CODE);
}
System.exit(RUNTIME_FAILURE_RETURN_CODE);
}
{code}
We forced to make the FAILURE_RETURN_CODE as STARTUP_FAILURE_RETURN_CODE
rather than RUNTIME_FAILURE_RETURN_CODE in some condition and *it DID WORK*.
After discussing with [~Tison], I knew that FAILURE_RETURN_CODE seems only to
be used to debug, so I submitted this issue and look forward to ANY solution~
> YARN AM endless restarts when using wrong checkpoint path or wrong checkpoint
> -
>
> Key: FLINK-14268
> URL: https://issues.apache.org/jira/browse/FLINK-14268
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.7.2
> Environment: Flink: 1.7.2
> Deloyment: YARN Per Job
> YARN:2.7.2
> State backend:FSStateBackend with HDFS
>
>Reporter: Lsw_aka_laplace
>Priority: Critical
>
> I tried to start a streaming task and restore from checkpoint which it was
> stored in HDFS.
> I set a wrong checkpoint path and sth unexpected happened: YARN AM restarted
> again and again. Since we have already set some restart strategy to prevent
> endless restart, it should have been restarted with limited times.
> Since we made sure that restart strategy works, we dived into source code and
> did some change mainly in _ClusterEntrypoint_.
>
> {code:java}
> //代码占位符
> //before
> @Override
> public void onFatalError(Throwable exception) {
>LOG.error("Fatal error occurred in the cluster entrypoint.", exception);
>System.exit(RUNTIME_FAILURE_RETURN_CODE);
> }
> //after
> @Override
> public void onFatalError(Throwable exception) {
>LOG.error("Fatal error occurred in the cluster entrypoint.", exception);
>
> if(ExceptionUtils.findThrowable(exception,PerJobFatalException.class).isPresent()){
> //PerJobFatalException is the FLAG
> //在perjob模式有些致命的异常出现，am会一直重启，不能失败掉
> LOG.error("perjob fatal error");
> System.exit(STARTUP_FAILURE_RETURN_CODE);
>}
>System.exit(RUNTIME_FAILURE_RETURN_CODE);
> }
> {code}
> We forced to make the FAILURE_RETURN_CODE as STARTUP_FAILURE_RETURN_CODE
> rather than RUNTIME_FAILURE_RETURN_CODE in some condition and *it DID
[jira] [Updated] (FLINK-14268) YARN AM endless restarts when using wrong checkpoint path or wrong checkpoint
[
https://issues.apache.org/jira/browse/FLINK-14268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lsw_aka_laplace updated FLINK-14268:
Description:
I tried to start a streaming task and restore from checkpoint which it was
stored in HDFS.
I set a wrong checkpoint path and sth unexpected happened: YARN AM restarted
again and again. Since we have already set some restart strategy to prevent
endless restart, it should have been restarted with limited times.
Since we made sure that restart strategy works, we dived into source code and
did some change mainly in _ClusterEntrypoint_.
{code:java}
//代码占位符
//before
@Override
public void onFatalError(Throwable exception) {
LOG.error("Fatal error occurred in the cluster entrypoint.", exception);
System.exit(RUNTIME_FAILURE_RETURN_CODE);
}
//after
@Override
public void onFatalError(Throwable exception) {
LOG.error("Fatal error occurred in the cluster entrypoint.", exception);
// PerJobFatalException is the FLAG
if(ExceptionUtils.findThrowable(exception,PerJobFatalException.class).isPresent()){//在perjob模式有些致命的异常出现，am会一直重启，不能失败掉
LOG.error("perjob fatal error");
System.exit(STARTUP_FAILURE_RETURN_CODE);
}
System.exit(RUNTIME_FAILURE_RETURN_CODE);
}
{code}
We forced to make the FAILURE_RETURN_CODE as STARTUP_FAILURE_RETURN_CODE
rather than RUNTIME_FAILURE_RETURN_CODE in some condition and *it DID WORK*.
After discussing with [~Tison], I knew that FAILURE_RETURN_CODE seems only to
be used to debug, so I submitted this issue and look forward to ANY solution~
was:
I tried to start a streaming task and restore from checkpoint which it was
stored in HDFS.
I set a wrong checkpoint path and sth unexpected happened: YARN AM restarted
again and again. Since we have already set some restart strategy to prevent
endless restart, it should have been restarted with limited times.
Since we made sure that restart strategy works, we dived into source code and
did some change mainly in _ClusterEntrypoint_.
{code:java}
//代码占位符
//before
@Override
public void onFatalError(Throwable exception) {
LOG.error("Fatal error occurred in the cluster entrypoint.", exception);
System.exit(RUNTIME_FAILURE_RETURN_CODE);
}
//after
@Override
public void onFatalError(Throwable exception) {
LOG.error("Fatal error occurred in the cluster entrypoint.", exception);
// PerJobFatalException is the FLAG
if(ExceptionUtils.findThrowable(exception,PerJobFatalException.class).isPresent()){//在perjob模式有些致命的异常出现，am会一直重启，不能失败掉
LOG.error("perjob fatal error");
System.exit(STARTUP_FAILURE_RETURN_CODE);
}
System.exit(RUNTIME_FAILURE_RETURN_CODE);
}
{code}
We forced to make the FAILURE_RETURN_CODE as STARTUP_FAILURE_RETURN_CODE
rather than RUNTIME_FAILURE_RETURN_CODE in some condition and *it DID WORK*.
After discussing with [~Tison], I knew that FAILURE_RETURN_CODE seems only to
be used by debug, so I submitted this issue and look forward to ANY solution~
> YARN AM endless restarts when using wrong checkpoint path or wrong checkpoint
> -
>
> Key: FLINK-14268
> URL: https://issues.apache.org/jira/browse/FLINK-14268
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.7.2
> Environment: Flink: 1.7.2
> Deloyment: YARN Per Job
> YARN:2.7.2
> State backend:FSStateBackend with HDFS
>
>Reporter: Lsw_aka_laplace
>Priority: Critical
>
> I tried to start a streaming task and restore from checkpoint which it was
> stored in HDFS.
> I set a wrong checkpoint path and sth unexpected happened: YARN AM restarted
> again and again. Since we have already set some restart strategy to prevent
> endless restart, it should have been restarted with limited times.
> Since we made sure that restart strategy works, we dived into source code and
> did some change mainly in _ClusterEntrypoint_.
>
> {code:java}
> //代码占位符
> //before
> @Override
> public void onFatalError(Throwable exception) {
>LOG.error("Fatal error occurred in the cluster entrypoint.", exception);
>System.exit(RUNTIME_FAILURE_RETURN_CODE);
> }
> //after
> @Override
> public void onFatalError(Throwable exception) {
>LOG.error("Fatal error occurred in the cluster entrypoint.", exception);
> // PerJobFatalException is the FLAG
> if(ExceptionUtils.findThrowable(exception,PerJobFatalException.class).isPresent()){//在perjob模式有些致命的异常出现，am会一直重启，不能失败掉
> LOG.error("perjob fatal error");
> System.exit(STARTUP_FAILURE_RETURN_CODE);
>}
>System.exit(RUNTIME_FAILURE_RETURN_CODE);
> }
> {code}
> We forced to make the FAILURE_RETURN_CODE as STARTUP_FAILURE_RETURN_CODE
> rather than RUNTIME_FAILURE_RETURN_CODE in some condition and *it DID WORK*.
[jira] [Created] (FLINK-14268) YARN AM endless restarts when using wrong checkpoint path or wrong checkpoint
Lsw_aka_laplace created FLINK-14268:
---
Summary: YARN AM endless restarts when using wrong checkpoint path
or wrong checkpoint
Key: FLINK-14268
URL: https://issues.apache.org/jira/browse/FLINK-14268
Project: Flink
Issue Type: Bug
Components: Deployment / YARN
Affects Versions: 1.7.2
Environment: Flink: 1.7.2
Deloyment: YARN Per Job
YARN:2.7.2
State backend:FSStateBackend with HDFS
Reporter: Lsw_aka_laplace
I tried to start a streaming task and restore from checkpoint which it was
stored in HDFS.
I set a wrong checkpoint path and sth unexpected happened: YARN AM restarted
again and again. Since we have already set some restart strategy to prevent
endless restart, it should have been restarted with limited times.
Since we made sure that restart strategy works, we dived into source code and
did some change mainly in _ClusterEntrypoint_.
{code:java}
//代码占位符
//before
@Override
public void onFatalError(Throwable exception) {
LOG.error("Fatal error occurred in the cluster entrypoint.", exception);
System.exit(RUNTIME_FAILURE_RETURN_CODE);
}
//after
@Override
public void onFatalError(Throwable exception) {
LOG.error("Fatal error occurred in the cluster entrypoint.", exception);
// PerJobFatalException is the FLAG
if(ExceptionUtils.findThrowable(exception,PerJobFatalException.class).isPresent()){//在perjob模式有些致命的异常出现，am会一直重启，不能失败掉
LOG.error("perjob fatal error");
System.exit(STARTUP_FAILURE_RETURN_CODE);
}
System.exit(RUNTIME_FAILURE_RETURN_CODE);
}
{code}
We forced to make the FAILURE_RETURN_CODE as STARTUP_FAILURE_RETURN_CODE
rather than RUNTIME_FAILURE_RETURN_CODE in some condition and *it DID WORK*.
After discussing with [~Tison], I knew that FAILURE_RETURN_CODE seems only to
be used by debug, so I submitted this issue and look forward to ANY solution~
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9762: !IGNORE! ZK 3.5.5 shaded migration
flinkbot edited a comment on issue #9762: !IGNORE! ZK 3.5.5 shaded migration URL: https://github.com/apache/flink/pull/9762#issuecomment-534689957 ## CI report: * cefac57542dbd85a6693f6a1b7c8c90d91e85d30 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/128980426) * f04ce4e35191731da657f170c6949ba70d87b6e3 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129575680) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9762: !IGNORE! ZK 3.5.5 shaded migration
flinkbot edited a comment on issue #9762: !IGNORE! ZK 3.5.5 shaded migration URL: https://github.com/apache/flink/pull/9762#issuecomment-534689957 ## CI report: * cefac57542dbd85a6693f6a1b7c8c90d91e85d30 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/128980426) * f04ce4e35191731da657f170c6949ba70d87b6e3 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14117) Translate changes on documentation index page to Chinese
[ https://issues.apache.org/jira/browse/FLINK-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940236#comment-16940236 ] Ricco Chen commented on FLINK-14117: Hi, [~fhueske], i want to contribute to Flink. Would u assign this translation work to me? Thanks. > Translate changes on documentation index page to Chinese > > > Key: FLINK-14117 > URL: https://issues.apache.org/jira/browse/FLINK-14117 > Project: Flink > Issue Type: Task > Components: chinese-translation, Documentation >Affects Versions: 1.10.0 >Reporter: Fabian Hueske >Priority: Major > > The changes of commit > [ee0d6fdf0604d74bd1cf9a6eb9cf5338ac1aa4f9|https://github.com/apache/flink/commit/ee0d6fdf0604d74bd1cf9a6eb9cf5338ac1aa4f9#diff-1a523bd9fa0dbf998008b37579210e12] > on the documentation index page should be translated to Chinese. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-13360) Add documentation for HBase connector for Table API & SQL
[ https://issues.apache.org/jira/browse/FLINK-13360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940231#comment-16940231 ] Jark Wu commented on FLINK-13360: - Hi [~lzljs3620320], do you still have time to do this? There are many users are asking how to use HBase and MySQL in DDL in user mailing list an DingTalk group. > Add documentation for HBase connector for Table API & SQL > - > > Key: FLINK-13360 > URL: https://issues.apache.org/jira/browse/FLINK-13360 > Project: Flink > Issue Type: Sub-task > Components: Connectors / HBase, Documentation >Reporter: Jark Wu >Assignee: Jingsong Lee >Priority: Major > Fix For: 1.10.0, 1.9.2 > > > Add documentation for HBase connector for Table API & SQL > - “Connect to External Systems”: Add DDL for HBase in “Table Connector” > section. HBase support batch-source & lookup & sink. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-13361) Add documentation for JDBC connector for Table API & SQL
[ https://issues.apache.org/jira/browse/FLINK-13361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940232#comment-16940232 ] Jark Wu commented on FLINK-13361: - Hi [~lzljs3620320], do you still have time to do this? There are many users are asking how to use HBase and MySQL in DDL in user mailing list an DingTalk group. > Add documentation for JDBC connector for Table API & SQL > > > Key: FLINK-13361 > URL: https://issues.apache.org/jira/browse/FLINK-13361 > Project: Flink > Issue Type: Sub-task > Components: Connectors / JDBC, Documentation >Reporter: Jark Wu >Assignee: Jingsong Lee >Priority: Major > Fix For: 1.10.0, 1.9.2 > > > Add documentation for JDBC connector for Table API & SQL > - “Connect to External Systems”: Add DDL for JDBC in “Table Connector” > section. JDBC support batch-source & lookup & sink. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14266) Introduce RowCsvInputFormat to new CSV module
[
https://issues.apache.org/jira/browse/FLINK-14266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940230#comment-16940230
]
Jark Wu commented on FLINK-14266:
-
Can we leverage the existing {{CsvRowDeserializationSchema}} for this?
> Introduce RowCsvInputFormat to new CSV module
> -
>
> Key: FLINK-14266
> URL: https://issues.apache.org/jira/browse/FLINK-14266
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / FileSystem
>Reporter: Jingsong Lee
>Priority: Major
> Fix For: 1.10.0
>
>
> Now, we have an old CSV, but that is not standard CSV support. we should
> support the RFC-compliant CSV format for table/sql.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9781: [FLINK-14098] Support multiple statements for TableEnvironment.
flinkbot edited a comment on issue #9781: [FLINK-14098] Support multiple statements for TableEnvironment. URL: https://github.com/apache/flink/pull/9781#issuecomment-535813967 ## CI report: * ed0e04d1cd481b096d68d0c9bf36a997dcca2550 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129400381) * ec8fccdeb53c5e2e0f089708753be33839706e2a : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129441309) * ad27f8e71d91534867d093f00cde3d5b81534282 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129526289) * 5247e6b03266d0d25d42a185e0a447643116d96f : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129545862) * bdad4f2c28154dd4d9d4ab05b9202c47644d4c9f : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129573368) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9749: [FLINK-14115][docs-zh] Translate DataStream Code Walkthrough to Chinese
flinkbot edited a comment on issue #9749: [FLINK-14115][docs-zh] Translate DataStream Code Walkthrough to Chinese URL: https://github.com/apache/flink/pull/9749#issuecomment-534149758 ## CI report: * 141eb6db531af4b12a9506489f280f180c02a408 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/128770115) * 0b96f066d276e4a483785f01335d36876e0e971c : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/128887957) * 5d7e1626653293ea20f935829d553de3b3e421fe : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129282782) * 16fab72a9ee8ea4940082d9fd32f0331f6271540 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/129455836) * 087f60aef7141fc77a9dd632ca01bf6319bbf29d : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129459090) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] dianfu commented on a change in pull request #9766: [FLINK-14018][python] Add Python building blocks to make sure the basic functionality of Python ScalarFunction could work
dianfu commented on a change in pull request #9766: [FLINK-14018][python] Add Python building blocks to make sure the basic functionality of Python ScalarFunction could work URL: https://github.com/apache/flink/pull/9766#discussion_r329335830 ## File path: flink-python/pyflink/table/udf.py ## @@ -0,0 +1,217 @@ + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import collections +import functools +import inspect +from abc import ABCMeta, abstractmethod + +from pyflink.java_gateway import get_gateway +from pyflink.table.types import DataType, _to_java_type +from pyflink.util import utils + +__all__ = ['FunctionContext', 'ScalarFunction', 'udf'] + + +class FunctionContext(object): +""" +Used to obtain global runtime information about the context in which the +user-defined function is executed. The information includes the metric group, +and global job parameters, etc. +""" +pass + + +class UserDefinedFunction(object): +""" +Base interface for user-defined function. +""" +__metaclass__ = ABCMeta Review comment: It's a best practice to declare the ABCMeta if it's an abstract class in Python. Quoted from [StackOverFlow](https://stackoverflow.com/questions/3570796/why-use-abstract-base-classes-in-python): "A handy feature of ABCs is that if you don't implement all necessary methods (and properties) you get an error upon instantiation, rather than an AttributeError, potentially much later, when you actually try to use the missing method.". I'll add a test case. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-14145) getLatestCheckpoint(true) returns wrong checkpoint
[
https://issues.apache.org/jira/browse/FLINK-14145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-14145:
Fix Version/s: 1.9.1
1.10.0
> getLatestCheckpoint(true) returns wrong checkpoint
> --
>
> Key: FLINK-14145
> URL: https://issues.apache.org/jira/browse/FLINK-14145
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.9.0
>Reporter: Ufuk Celebi
>Assignee: Gyula Fora
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0, 1.9.1
>
> Attachments:
> 0001-FLINK-14145-runtime-Add-getLatestCheckpoint-test.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The flag to prefer checkpoints for recovery introduced in FLINK-11159 returns
> the wrong checkpoint if:
> * checkpoints are preferred ({{getLatestCheckpoint(true)}}),
> * the latest checkpoint is *not* a savepoint,
> * more than a single checkpoint is retained.
> The current implementation assumes that the latest checkpoint is a savepoint
> and skips over it. I attached a patch for
> {{StandaloneCompletedCheckpointStoreTest}} that demonstrates the issue.
> You can apply the patch via {{git am -3 < *.patch}}.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14145) getLatestCheckpoint(true) returns wrong checkpoint
[
https://issues.apache.org/jira/browse/FLINK-14145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940229#comment-16940229
]
Jark Wu commented on FLINK-14145:
-
1.10.0: de92c698bbba464d15de9b70546f1b1ceee5b48b
1.9.1: 3aeeb5fede885f9efeb66fe271828b25ffd6571e
> getLatestCheckpoint(true) returns wrong checkpoint
> --
>
> Key: FLINK-14145
> URL: https://issues.apache.org/jira/browse/FLINK-14145
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.9.0
>Reporter: Ufuk Celebi
>Assignee: Gyula Fora
>Priority: Major
> Labels: pull-request-available
> Attachments:
> 0001-FLINK-14145-runtime-Add-getLatestCheckpoint-test.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The flag to prefer checkpoints for recovery introduced in FLINK-11159 returns
> the wrong checkpoint if:
> * checkpoints are preferred ({{getLatestCheckpoint(true)}}),
> * the latest checkpoint is *not* a savepoint,
> * more than a single checkpoint is retained.
> The current implementation assumes that the latest checkpoint is a savepoint
> and skips over it. I attached a patch for
> {{StandaloneCompletedCheckpointStoreTest}} that demonstrates the issue.
> You can apply the patch via {{git am -3 < *.patch}}.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14235) Kafka010ProducerITCase>KafkaProducerTestBase.testOneToOneAtLeastOnceCustomOperator fails on travis
[
https://issues.apache.org/jira/browse/FLINK-14235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940228#comment-16940228
]
gaofeilong commented on FLINK-14235:
Another instance [https://api.travis-ci.com/v3/job/239836223/log.txt]
> Kafka010ProducerITCase>KafkaProducerTestBase.testOneToOneAtLeastOnceCustomOperator
> fails on travis
> --
>
> Key: FLINK-14235
> URL: https://issues.apache.org/jira/browse/FLINK-14235
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.10.0
>Reporter: Piotr Nowojski
>Priority: Blocker
> Labels: test-stability
> Fix For: 1.10.0
>
>
> Fails with following message:
> {noformat}
> Expected to contain all of: <[0]>, but was: <[]>
> {noformat}
> with the following stack trace:
> {noformat}
> Test
> testOneToOneAtLeastOnceCustomOperator(org.apache.flink.streaming.connectors.kafka.Kafka010ProducerITCase)
> failed with:
> java.lang.AssertionError: Expected to contain all of: <[0]>, but was: <[]>
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaTestBase.assertAtLeastOnceForTopic(KafkaTestBase.java:235)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaProducerTestBase.testOneToOneAtLeastOnce(KafkaProducerTestBase.java:289)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaProducerTestBase.testOneToOneAtLeastOnceCustomOperator(KafkaProducerTestBase.java:214)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {noformat}
> https://api.travis-ci.com/v3/job/239300010/log.txt
> This might be related to https://issues.apache.org/jira/browse/FLINK-14224
> but the failure message is different (here it's data loss, there the job has
> failed)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13942) Add Overview page for Getting Started section
[ https://issues.apache.org/jira/browse/FLINK-13942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-13942: Fix Version/s: 1.9.1 1.10.0 > Add Overview page for Getting Started section > - > > Key: FLINK-13942 > URL: https://issues.apache.org/jira/browse/FLINK-13942 > Project: Flink > Issue Type: Sub-task > Components: Documentation >Affects Versions: 1.9.0, 1.10.0 >Reporter: Fabian Hueske >Assignee: Fabian Hueske >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1 > > Time Spent: 20m > Remaining Estimate: 0h > > The Getting Started section provide different types of tutorials that target > users with different interests and backgrounds. > We should add a brief overview page that describes the different tutorials > such that users easily find the material that they need to get started with > Flink. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-10672) Task stuck while writing output to flink
[
https://issues.apache.org/jira/browse/FLINK-10672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940226#comment-16940226
]
Yun Gao commented on FLINK-10672:
-
Hi [~mxm], very thanks for the explanation. Besides, is it possible to also
provide the topology of the underlying Flink Job and the status of each job
vertex when the job get stuck? I think with these information, we may be able
to analyze if there are cyclic dependency among tasks.
> Task stuck while writing output to flink
>
>
> Key: FLINK-10672
> URL: https://issues.apache.org/jira/browse/FLINK-10672
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.5.4
> Environment: OS: Debuan rodente 4.17
> Flink version: 1.5.4
> ||Key||Value||
> |jobmanager.heap.mb|1024|
> |jobmanager.rpc.address|localhost|
> |jobmanager.rpc.port|6123|
> |metrics.reporter.jmx.class|org.apache.flink.metrics.jmx.JMXReporter|
> |metrics.reporter.jmx.port|9250-9260|
> |metrics.reporters|jmx|
> |parallelism.default|1|
> |rest.port|8081|
> |taskmanager.heap.mb|1024|
> |taskmanager.numberOfTaskSlots|1|
> |web.tmpdir|/tmp/flink-web-bdb73d6c-5b9e-47b5-9ebf-eed0a7c82c26|
>
> h1. Overview
> ||Data Port||All Slots||Free Slots||CPU Cores||Physical Memory||JVM Heap
> Size||Flink Managed Memory||
> |43501|1|0|12|62.9 GB|922 MB|642 MB|
> h1. Memory
> h2. JVM (Heap/Non-Heap)
> ||Type||Committed||Used||Maximum||
> |Heap|922 MB|575 MB|922 MB|
> |Non-Heap|68.8 MB|64.3 MB|-1 B|
> |Total|991 MB|639 MB|922 MB|
> h2. Outside JVM
> ||Type||Count||Used||Capacity||
> |Direct|3,292|105 MB|105 MB|
> |Mapped|0|0 B|0 B|
> h1. Network
> h2. Memory Segments
> ||Type||Count||
> |Available|3,194|
> |Total|3,278|
> h1. Garbage Collection
> ||Collector||Count||Time||
> |G1_Young_Generation|13|336|
> |G1_Old_Generation|1|21|
>Reporter: Ankur Goenka
>Assignee: Yun Gao
>Priority: Major
> Labels: beam
> Attachments: 1uruvakHxBu.png, 3aDKQ24WvKk.png, Po89UGDn58V.png,
> jmx_dump.json, jmx_dump_detailed.json, jstack_129827.log, jstack_163822.log,
> jstack_66985.log
>
>
> I am running a fairly complex pipleline with 200+ task.
> The pipeline works fine with small data (order of 10kb input) but gets stuck
> with a slightly larger data (300kb input).
>
> The task gets stuck while writing the output toFlink, more specifically it
> gets stuck while requesting memory segment in local buffer pool. The Task
> manager UI shows that it has enough memory and memory segments to work with.
> The relevant stack trace is
> {quote}"grpc-default-executor-0" #138 daemon prio=5 os_prio=0
> tid=0x7fedb0163800 nid=0x30b7f in Object.wait() [0x7fedb4f9]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at (C/C++) 0x7fef201c7dae (Unknown Source)
> at (C/C++) 0x7fef1f2aea07 (Unknown Source)
> at (C/C++) 0x7fef1f241cd3 (Unknown Source)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xf6d56450> (a java.util.ArrayDeque)
> at
> org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestMemorySegment(LocalBufferPool.java:247)
> - locked <0xf6d56450> (a java.util.ArrayDeque)
> at
> org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBufferBuilderBlocking(LocalBufferPool.java:204)
> at
> org.apache.flink.runtime.io.network.api.writer.RecordWriter.requestNewBufferBuilder(RecordWriter.java:213)
> at
> org.apache.flink.runtime.io.network.api.writer.RecordWriter.sendToTarget(RecordWriter.java:144)
> at
> org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:107)
> at
> org.apache.flink.runtime.operators.shipping.OutputCollector.collect(OutputCollector.java:65)
> at
> org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
> at
> org.apache.beam.runners.flink.translation.functions.FlinkExecutableStagePruningFunction.flatMap(FlinkExecutableStagePruningFunction.java:42)
> at
> org.apache.beam.runners.flink.translation.functions.FlinkExecutableStagePruningFunction.flatMap(FlinkExecutableStagePruningFunction.java:26)
> at
> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:80)
> at
> org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
> at
> org.apache.beam.runners.flink.translation.functions.FlinkExecutableStageFunction$MyDataReceiver.accept(FlinkExecutableStageFunction.java:230)
> - locked <0xf6a60bd0> (a java.lang.Object)
> at
> org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver.accept(BeamFnDataInboundObserver.java:81)
> at
> org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver.accept(BeamFnDataInboundObserver.java:32)
> at
>
[jira] [Commented] (FLINK-13230) Retry acknowledgement calls
[ https://issues.apache.org/jira/browse/FLINK-13230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940225#comment-16940225 ] Jark Wu commented on FLINK-13230: - I moved this issue to 1.9.2 to unblock 1.9.1 as this is not a critical issue. > Retry acknowledgement calls > --- > > Key: FLINK-13230 > URL: https://issues.apache.org/jira/browse/FLINK-13230 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Google Cloud PubSub >Reporter: Richard Deurwaarder >Assignee: Richard Deurwaarder >Priority: Minor > Labels: pull-request-available > Fix For: 1.9.2 > > Time Spent: 10m > Remaining Estimate: 0h > > Currently when a pubsub/grpc pull fails we retry based on configuration given > by the user. > We should do the same for Acknowledgement calls -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14206) Let fullRestart metric count fine grained restarts as well
[
https://issues.apache.org/jira/browse/FLINK-14206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940224#comment-16940224
]
Jark Wu commented on FLINK-14206:
-
I moved this issue to 1.9.2 to unblock 1.9.1 as this is not a critical issue.
> Let fullRestart metric count fine grained restarts as well
> --
>
> Key: FLINK-14206
> URL: https://issues.apache.org/jira/browse/FLINK-14206
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Zhu Zhu
>Assignee: Zhu Zhu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0, 1.9.2
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> With fine grained recovery introduced in 1.9.0, the {{fullRestart}} metric
> only counts how many times the entire graph has been restarted, not including
> the number of fine grained failure restarts.
> As many users leverage this metric for failure detecting monitoring and
> alerting, I'd propose to make it also count fine grained restarts.
> The concrete proposal is:
> - Add a counter {{numberOfRestartsCounter}} in {{ExecutionGraph}} to count
> all restarts. The counter is not to be registered to metric groups.
> - Let fullRestart query the value of the counter, instead of
> {{ExecutionGraph#globalModVersion}}
> - increment {{numberOfRestartsCounter}} in
> {{ExecutionGraph#incrementGlobalModVersion()}}
> - increment {{numberOfRestartsCounter}} in
> {{AdaptedRestartPipelinedRegionStrategyNG#restartTasks(...)}}, to ensure that
> the fine grained recovery really happens
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13230) Retry acknowledgement calls
[ https://issues.apache.org/jira/browse/FLINK-13230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-13230: Fix Version/s: (was: 1.9.1) 1.9.2 > Retry acknowledgement calls > --- > > Key: FLINK-13230 > URL: https://issues.apache.org/jira/browse/FLINK-13230 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Google Cloud PubSub >Reporter: Richard Deurwaarder >Assignee: Richard Deurwaarder >Priority: Minor > Labels: pull-request-available > Fix For: 1.9.2 > > Time Spent: 10m > Remaining Estimate: 0h > > Currently when a pubsub/grpc pull fails we retry based on configuration given > by the user. > We should do the same for Acknowledgement calls -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13809) The log directory of Flink Python API is unwritable if it is installed via "sudo"
[
https://issues.apache.org/jira/browse/FLINK-13809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-13809:
Fix Version/s: (was: 1.9.1)
1.9.2
> The log directory of Flink Python API is unwritable if it is installed via
> "sudo"
> --
>
> Key: FLINK-13809
> URL: https://issues.apache.org/jira/browse/FLINK-13809
> Project: Flink
> Issue Type: Improvement
> Components: API / Python
>Affects Versions: 1.9.0
>Reporter: Wei Zhong
>Assignee: Wei Zhong
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.10.0, 1.9.2
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, if the python apache-flink package is installed via "sudo", an
> exception will be thrown when starting the flink python shell:
> {code:java}
> log4j:ERROR setFile(null,false) call failed. java.io.FileNotFoundException:
> /Library/Python/2.7/site-packages/pyflink/log/flink-zhongwei-python-zhongweideMacBook-Pro.local.log
> (Permission denied) at java.io.FileOutputStream.open0(Native Method) at
> java.io.FileOutputStream.open(FileOutputStream.java:270) at
> java.io.FileOutputStream.(FileOutputStream.java:213) at
> java.io.FileOutputStream.(FileOutputStream.java:133) at
> org.apache.log4j.FileAppender.setFile(FileAppender.java:294) at
> org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165) at
> org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307) at
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
> at
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
> at
> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
> at
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
> at
> org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
> at
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
> at
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
> at
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
> at org.apache.log4j.LogManager.(LogManager.java:127) at
> org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:81) at
> org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329) at
> org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349) at
> org.apache.flink.api.java.ExecutionEnvironment.(ExecutionEnvironment.java:102)
> at java.lang.Class.forName0(Native Method) at
> java.lang.Class.forName(Class.java:348) at
> org.apache.flink.api.python.shaded.py4j.reflection.CurrentThreadClassLoadingStrategy.classForName(CurrentThreadClassLoadingStrategy.java:40)
> at
> org.apache.flink.api.python.shaded.py4j.reflection.ReflectionUtil.classForName(ReflectionUtil.java:51)
> at
> org.apache.flink.api.python.shaded.py4j.reflection.TypeUtil.forName(TypeUtil.java:243)
> at
> org.apache.flink.api.python.shaded.py4j.commands.ReflectionCommand.getUnknownMember(ReflectionCommand.java:175)
> at
> org.apache.flink.api.python.shaded.py4j.commands.ReflectionCommand.execute(ReflectionCommand.java:87)
> at
> org.apache.flink.api.python.shaded.py4j.GatewayConnection.run(GatewayConnection.java:238)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> It does not affect the running of flink python shell but it would be better
> if we can fix it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14206) Let fullRestart metric count fine grained restarts as well
[
https://issues.apache.org/jira/browse/FLINK-14206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-14206:
Fix Version/s: (was: 1.9.1)
1.9.2
> Let fullRestart metric count fine grained restarts as well
> --
>
> Key: FLINK-14206
> URL: https://issues.apache.org/jira/browse/FLINK-14206
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Zhu Zhu
>Assignee: Zhu Zhu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0, 1.9.2
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> With fine grained recovery introduced in 1.9.0, the {{fullRestart}} metric
> only counts how many times the entire graph has been restarted, not including
> the number of fine grained failure restarts.
> As many users leverage this metric for failure detecting monitoring and
> alerting, I'd propose to make it also count fine grained restarts.
> The concrete proposal is:
> - Add a counter {{numberOfRestartsCounter}} in {{ExecutionGraph}} to count
> all restarts. The counter is not to be registered to metric groups.
> - Let fullRestart query the value of the counter, instead of
> {{ExecutionGraph#globalModVersion}}
> - increment {{numberOfRestartsCounter}} in
> {{ExecutionGraph#incrementGlobalModVersion()}}
> - increment {{numberOfRestartsCounter}} in
> {{AdaptedRestartPipelinedRegionStrategyNG#restartTasks(...)}}, to ensure that
> the fine grained recovery really happens
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13358) Add documentation for InputFormatTableSource and OutputFormatTableSink
[ https://issues.apache.org/jira/browse/FLINK-13358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-13358: Fix Version/s: (was: 1.9.1) 1.9.2 > Add documentation for InputFormatTableSource and OutputFormatTableSink > -- > > Key: FLINK-13358 > URL: https://issues.apache.org/jira/browse/FLINK-13358 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Table SQL / API >Reporter: Jark Wu >Priority: Minor > Fix For: 1.9.2 > > > Add documentation for InputFormatTableSource and OutputFormatTableSink > - “User-defined Sources & Sinks”: what is InputFormatTableSource and > OutputFormatTableSink, and how to use/implement them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14200) Temporal Table Function Joins do not work on Tables (only TableSources) on the query side
[
https://issues.apache.org/jira/browse/FLINK-14200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-14200:
Fix Version/s: (was: 1.9.1)
1.9.2
> Temporal Table Function Joins do not work on Tables (only TableSources) on
> the query side
> -
>
> Key: FLINK-14200
> URL: https://issues.apache.org/jira/browse/FLINK-14200
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.9.0
> Environment: Java 8, Scala 2.11, Flink 1.9
>Reporter: Benoît Paris
>Priority: Major
> Fix For: 1.9.2
>
> Attachments:
> temporal-table-function-query-side-as-not-table-source.zip
>
>
> This only affects the Blink planner. The legacy planner works fine.
> With Orders as a TableSource, and Orders2 as a Table with the same content:
> {code:java}
> tEnv.registerTableSource("Orders", new FooSource(new String[] {"o_currency",
> "o_amount", "o_proctime"}));
> Table orders2 = tEnv.sqlQuery("SELECT * FROM Orders");
> tEnv.registerTable("Orders2", orders2);{code}
> This works (TableSource on the query side):
> {code:java}
> SELECT
> o_amount * r_amount AS amount
> FROM Orders
> , LATERAL TABLE (Rates(o_proctime))
> WHERE r_currency = o_currency{code}
> While this does not (Table on the query side):
> {code:java}
> SELECT
> o_amount * r_amount AS amount
> FROM Orders2
> , LATERAL TABLE (Rates(o_proctime))
> WHERE r_currency = o_currency{code}
> Throwing an NPE in FlinkRelBuilder, called from
> LogicalCorrelateToJoinFromTemporalTableFunctionRule. Attached is Java code
> for reproduction, along with the full log and stacktrace, and a pom.xml.
> EDIT: This may not be Table vs TableSource, maybe more a projection or a
> decorrelate issue? Don't know how the planner works well enough to
> characterize it better.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13520) Add documentation for the different behavior of row count sliding window in blink planner
[ https://issues.apache.org/jira/browse/FLINK-13520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-13520: Fix Version/s: (was: 1.9.1) 1.9.2 > Add documentation for the different behavior of row count sliding window in > blink planner > - > > Key: FLINK-13520 > URL: https://issues.apache.org/jira/browse/FLINK-13520 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Table SQL / Planner >Reporter: Jark Wu >Priority: Minor > Fix For: 1.10.0, 1.9.2 > > > As discussed in FLINK-13446, we want to keep the current behavior of row > count sliding window in blink planner. This is different with flink planner. > So it would be nice to document it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13357) Add documentation for PartitionableTableSource and PartitionableTableSink
[ https://issues.apache.org/jira/browse/FLINK-13357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-13357: Fix Version/s: (was: 1.9.1) 1.9.2 > Add documentation for PartitionableTableSource and PartitionableTableSink > - > > Key: FLINK-13357 > URL: https://issues.apache.org/jira/browse/FLINK-13357 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Table SQL / API >Reporter: Jark Wu >Priority: Minor > Fix For: 1.9.2 > > > Add documentation for PartitionableTableSource and PartitionableTableSink > - “User-defined Sources & Sinks”: what is PartitionableTableSource and > PartitionableTableSink, and how to use/implement them -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13083) Various improvements PubSub Connector
[ https://issues.apache.org/jira/browse/FLINK-13083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-13083: Fix Version/s: (was: 1.9.1) 1.9.2 > Various improvements PubSub Connector > - > > Key: FLINK-13083 > URL: https://issues.apache.org/jira/browse/FLINK-13083 > Project: Flink > Issue Type: Improvement > Components: Connectors / Google Cloud PubSub >Affects Versions: 1.9.0 >Reporter: Richard Deurwaarder >Assignee: Richard Deurwaarder >Priority: Minor > Fix For: 1.9.2 > > > Umbrella task to keep track of issues remaining when FLINK-9311 was merged. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14177) Bump Curator From 2.12.0 to 4.2.0
[ https://issues.apache.org/jira/browse/FLINK-14177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-14177: Fix Version/s: (was: 1.9.1) 1.9.2 > Bump Curator From 2.12.0 to 4.2.0 > - > > Key: FLINK-14177 > URL: https://issues.apache.org/jira/browse/FLINK-14177 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hadoop Compatibility, Runtime / > Checkpointing >Affects Versions: 1.8.1, 1.9.0 >Reporter: lamber-ken >Assignee: lamber-ken >Priority: Major > Labels: pull-request-available > Fix For: 1.9.2 > > Time Spent: 10m > Remaining Estimate: 0h > > According to FLINK-10052 and FLINK-13417, we needs to upgrade the version of > CuratorFramework firstly. > Curator4.2.0 supports > 1) zk3.4.* and zk3.5.* > 2) connectionStateErrorPolicy -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14030) IS_NULL is optimized to incorrect results
[
https://issues.apache.org/jira/browse/FLINK-14030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-14030:
Summary: IS_NULL is optimized to incorrect results (was: Nonequivalent
conversion happens in Table planner )
> IS_NULL is optimized to incorrect results
> -
>
> Key: FLINK-14030
> URL: https://issues.apache.org/jira/browse/FLINK-14030
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.9.0
>Reporter: Leonard Xu
>Priority: Major
>
> *testAllApis()* unit tests will run fail because planner make a conversion
> from *[ifThenElse(isNull(plus(f0, f1)), 'null', 'not null')]*
> to *[CASE(OR(IS NULL($0), IS NULL($1)), _UTF-16LE'null', _UTF-16LE'not
> null')]*
> which is not a equivalence conversion. The result of expression 'f0 + 'f1
> should be null
> when the result overflows even if its two operands both are not null.
> It's easy to reproduce as following:
> testAllApis(
> 'f0 + 'f1,
> "f1 + f1",
> "f1 + f1",
> "null")// the result should be null because overflow
> override def testData: Row =
> { val testData = new Row(2) testData.setField(0,
> BigDecimal("1e10").bigDecimal) testData.setField(1,
> BigDecimal("0").bigDecimal) testData }
> override def typeInfo: RowTypeInfo =
> { new RowTypeInfo( /* 0 */ fromLogicalTypeToTypeInfo(DECIMAL(38, 10)), /* 1
> */ fromLogicalTypeToTypeInfo(DECIMAL(38, 28)) ) }
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9781: [FLINK-14098] Support multiple statements for TableEnvironment.
flinkbot edited a comment on issue #9781: [FLINK-14098] Support multiple statements for TableEnvironment. URL: https://github.com/apache/flink/pull/9781#issuecomment-535813967 ## CI report: * ed0e04d1cd481b096d68d0c9bf36a997dcca2550 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129400381) * ec8fccdeb53c5e2e0f089708753be33839706e2a : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129441309) * ad27f8e71d91534867d093f00cde3d5b81534282 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129526289) * 5247e6b03266d0d25d42a185e0a447643116d96f : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129545862) * bdad4f2c28154dd4d9d4ab05b9202c47644d4c9f : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/129573368) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-14030) Nonequivalent conversion happens in Table planner
[
https://issues.apache.org/jira/browse/FLINK-14030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-14030:
Fix Version/s: (was: 1.9.1)
> Nonequivalent conversion happens in Table planner
> --
>
> Key: FLINK-14030
> URL: https://issues.apache.org/jira/browse/FLINK-14030
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.9.0
>Reporter: Leonard Xu
>Priority: Major
>
> *testAllApis()* unit tests will run fail because planner make a conversion
> from *[ifThenElse(isNull(plus(f0, f1)), 'null', 'not null')]*
> to *[CASE(OR(IS NULL($0), IS NULL($1)), _UTF-16LE'null', _UTF-16LE'not
> null')]*
> which is not a equivalence conversion. The result of expression 'f0 + 'f1
> should be null
> when the result overflows even if its two operands both are not null.
> It's easy to reproduce as following:
> testAllApis(
> 'f0 + 'f1,
> "f1 + f1",
> "f1 + f1",
> "null")// the result should be null because overflow
> override def testData: Row =
> { val testData = new Row(2) testData.setField(0,
> BigDecimal("1e10").bigDecimal) testData.setField(1,
> BigDecimal("0").bigDecimal) testData }
> override def typeInfo: RowTypeInfo =
> { new RowTypeInfo( /* 0 */ fromLogicalTypeToTypeInfo(DECIMAL(38, 10)), /* 1
> */ fromLogicalTypeToTypeInfo(DECIMAL(38, 28)) ) }
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13827) shell variable should be escaped in start-scala-shell.sh
[
https://issues.apache.org/jira/browse/FLINK-13827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-13827:
Fix Version/s: (was: 1.9.1)
1.9.2
> shell variable should be escaped in start-scala-shell.sh
>
>
> Key: FLINK-13827
> URL: https://issues.apache.org/jira/browse/FLINK-13827
> Project: Flink
> Issue Type: Bug
> Components: Scala Shell
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Zili Chen
>Priority: Major
> Fix For: 1.10.0, 1.9.2
>
>
> {code:java}
> diff --git a/flink-scala-shell/start-script/start-scala-shell.sh
> b/flink-scala-shell/start-script/start-scala-shell.sh
> index b6da81af72..65b9045584 100644
> --- a/flink-scala-shell/start-script/start-scala-shell.sh
> +++ b/flink-scala-shell/start-script/start-scala-shell.sh
> @@ -97,9 +97,9 @@ log_setting="-Dlog.file="$LOG"
> -Dlog4j.configuration=file:"$FLINK_CONF_DIR"/$LOG
>
> if ${EXTERNAL_LIB_FOUND}
> then
> -$JAVA_RUN -Dscala.color -cp "$FLINK_CLASSPATH" $log_setting
> org.apache.flink.api.scala.FlinkShell $@ --addclasspath "$EXT_CLASSPATH"
> +$JAVA_RUN -Dscala.color -cp "$FLINK_CLASSPATH" "$log_setting"
> org.apache.flink.api.scala.FlinkShell $@ --addclasspath "$EXT_CLASSPATH"
> else
> -$JAVA_RUN -Dscala.color -cp "$FLINK_CLASSPATH" $log_setting
> org.apache.flink.api.scala.FlinkShell $@
> +$JAVA_RUN -Dscala.color -cp "$FLINK_CLASSPATH" "$log_setting"
> org.apache.flink.api.scala.FlinkShell $@
> fi
>
> #restore echo
> {code}
> otherwise it is error prone when {{$log_setting}} contain arbitrary content.
> For example, if the parent dir contain whitespace, said {{flink-1.9.0 2}},
> then {{bin/start-scala-shell.sh local}} will fail with
> {{Error: Could not find or load main class
> 2.log.flink\-\*\-scala\-shell\-local\-\*.log}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13692) Make CompletedCheckpointStore backwards compatible?
[
https://issues.apache.org/jira/browse/FLINK-13692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-13692:
Fix Version/s: (was: 1.9.1)
1.9.2
> Make CompletedCheckpointStore backwards compatible?
> ---
>
> Key: FLINK-13692
> URL: https://issues.apache.org/jira/browse/FLINK-13692
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Till Rohrmann
>Priority: Major
> Fix For: 1.10.0, 1.9.2
>
>
> With FLINK-11159 we changed the signature of
> {{CompletedCheckpointStore#getLatestCheckpoint}} from
> {{getLatestCheckpoint()}} to {{getLatestCheckpoint(boolean)}}. This is an API
> breaking change which causes users of this interface having to adapt their
> implementation and recompile it. Since {{CompletedCheckpointStore}} is no
> public API, this change is technically ok, but not so nice.
> If we decide to provide backwards compatibility, then we should still keep
> {{getLatestCheckpoint()}} and add a deprecation annotation to it. At least,
> we should update the JavaDocs of the
> {{CompletedCheckpointStore#getLatestCheckpoint}} method to reflect what the
> passed parameter states.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13940) S3RecoverableWriter causes job to get stuck in recovery
[ https://issues.apache.org/jira/browse/FLINK-13940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-13940: Fix Version/s: (was: 1.9.1) 1.9.2 > S3RecoverableWriter causes job to get stuck in recovery > --- > > Key: FLINK-13940 > URL: https://issues.apache.org/jira/browse/FLINK-13940 > Project: Flink > Issue Type: Bug > Components: Connectors / FileSystem >Affects Versions: 1.8.0, 1.8.1, 1.9.0 >Reporter: Jimmy Weibel Rasmussen >Assignee: Kostas Kloudas >Priority: Major > Fix For: 1.10.0, 1.9.2 > > > > The cleaning up of tmp files in S3 introduced by this ticket/PR: > https://issues.apache.org/jira/browse/FLINK-10963 > is preventing the flink job from being able to recover under some > circumstances. > > This is what seems to be happening: > When the jobs tries to recover, it will call initializeState() on all > operators, which results in the Bucket.restoreInProgressFile method being > called. > This will download the part_tmp file mentioned in the checkpoint that we're > restoring from, and finally it will call fsWriter.cleanupRecoverableState > which deletes the part_tmp file in S3. > Now the open() method is called on all operators. If the open() call fails > for one of the operators (this might happen if the issue that caused the job > to fail and restart is still unresolved), the job will fail again and try to > restart from the same checkpoint as before. This time however, downloading > the part_tmp file mentioned in the checkpoint fails because it was deleted > during the last recover attempt. > The bug is critical because it results in data loss. > > > > I discovered the bug because I have a flink job with a RabbitMQ source and a > StreamingFileSink that writes to S3 (and therefore uses the > S3RecoverableWriter). > Occasionally I have some RabbitMQ connection issues which causes the job to > fail and restart, sometimes the first few restart attempts fail because > rabbitmq is unreachable when flink tries to reconnect. > > This is what I was seeing: > RabbitMQ goes down > Job fails because of a RabbitMQ ConsumerCancelledException > Job attempts to restart but fails with a Rabbitmq connection exception (x > number of times) > RabbitMQ is back up > Job attempts to restart but fails with a FileNotFoundException due to some > _part_tmp file missing in S3. > > The job will be unable to restart and only option is to cancel and restart > the job (and loose all state) > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13925) ClassLoader in BlobLibraryCacheManager is not using context class loader
[ https://issues.apache.org/jira/browse/FLINK-13925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-13925: Fix Version/s: (was: 1.9.1) 1.9.2 > ClassLoader in BlobLibraryCacheManager is not using context class loader > > > Key: FLINK-13925 > URL: https://issues.apache.org/jira/browse/FLINK-13925 > Project: Flink > Issue Type: Bug >Affects Versions: 1.8.1, 1.9.0 >Reporter: Jan Lukavský >Assignee: Jan Lukavský >Priority: Major > Labels: pull-request-available > Fix For: 1.8.3, 1.9.2 > > Time Spent: 10m > Remaining Estimate: 0h > > Use thread's current context classloader as parent class loader of flink user > code class loaders. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13503) Add contract in `LookupableTableSource` to specify the behavior when lookupKeys contains null
[ https://issues.apache.org/jira/browse/FLINK-13503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-13503: Fix Version/s: (was: 1.9.1) 1.9.2 > Add contract in `LookupableTableSource` to specify the behavior when > lookupKeys contains null > - > > Key: FLINK-13503 > URL: https://issues.apache.org/jira/browse/FLINK-13503 > Project: Flink > Issue Type: Task > Components: Connectors / JDBC, Table SQL / API >Affects Versions: 1.9.0, 1.10.0 >Reporter: Jing Zhang >Assignee: Jing Zhang >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0, 1.9.2 > > Time Spent: 10m > Remaining Estimate: 0h > > I think we should add contract in `LookupableTableSource` to specify expected > behavior when the lookupKeys contains null value. > For example, one input record of eval method is (null,1) which means to look > up data in (a,b) columns which key satisfy the requirement. there are at > least three possibility here. > * to ignore null value, that is, in the above example, only looks `b = 1` > * to lookup `is value`, that is, in the above example, only looks `a is > null and b = 1` > * to return empty records, that is, in the above example, only looks `a = > null and b = 1` > In fact, there are different behavior in current code. > For example, in Jdbc connector, > The query template in `JdbcLookUpFunction` like: > SELECT c, d, e, f from T where a = ? and b = ? > If pass (null, 1) to `eval` method, it will generate the following query: > SELECT c, d, e, f from T where a = null and b = 1 > Which always outputs empty records. > BTW, Is this behavior reasonable? > and the `InMemoryLookupableTableSource` behaviors like point 2 in the above > list. > some private connector in Blink behaviors like point 1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13361) Add documentation for JDBC connector for Table API & SQL
[ https://issues.apache.org/jira/browse/FLINK-13361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-13361: Fix Version/s: (was: 1.9.1) 1.9.2 > Add documentation for JDBC connector for Table API & SQL > > > Key: FLINK-13361 > URL: https://issues.apache.org/jira/browse/FLINK-13361 > Project: Flink > Issue Type: Sub-task > Components: Connectors / JDBC, Documentation >Reporter: Jark Wu >Assignee: Jingsong Lee >Priority: Major > Fix For: 1.10.0, 1.9.2 > > > Add documentation for JDBC connector for Table API & SQL > - “Connect to External Systems”: Add DDL for JDBC in “Table Connector” > section. JDBC support batch-source & lookup & sink. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13360) Add documentation for HBase connector for Table API & SQL
[ https://issues.apache.org/jira/browse/FLINK-13360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-13360: Fix Version/s: (was: 1.9.1) 1.9.2 > Add documentation for HBase connector for Table API & SQL > - > > Key: FLINK-13360 > URL: https://issues.apache.org/jira/browse/FLINK-13360 > Project: Flink > Issue Type: Sub-task > Components: Connectors / HBase, Documentation >Reporter: Jark Wu >Assignee: Jingsong Lee >Priority: Major > Fix For: 1.10.0, 1.9.2 > > > Add documentation for HBase connector for Table API & SQL > - “Connect to External Systems”: Add DDL for HBase in “Table Connector” > section. HBase support batch-source & lookup & sink. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13298) write Chinese documentation and quickstart for Flink-Hive compatibility
[ https://issues.apache.org/jira/browse/FLINK-13298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-13298: Fix Version/s: (was: 1.9.1) 1.9.2 > write Chinese documentation and quickstart for Flink-Hive compatibility > --- > > Key: FLINK-13298 > URL: https://issues.apache.org/jira/browse/FLINK-13298 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Connectors / Hive, Documentation >Reporter: Bowen Li >Assignee: Shengnan YU >Priority: Major > Fix For: 1.9.2 > > > its corresponding English one is FLINK-13276 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-13278) add Chinese documentation for Hive source/sink
[
https://issues.apache.org/jira/browse/FLINK-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940223#comment-16940223
]
Jark Wu commented on FLINK-13278:
-
FLINK-13277 has been merged. So I think we can start the translation.
[~alfredlu] I assigned this issue to you.
> add Chinese documentation for Hive source/sink
> --
>
> Key: FLINK-13278
> URL: https://issues.apache.org/jira/browse/FLINK-13278
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Hive, Documentation
>Reporter: Bowen Li
>Assignee: Zijie Lu
>Priority: Major
> Fix For: 1.9.2
>
>
> add Chinese documentation of Hive source/sink in {{batch/connector_zh.md}}
> it's corresponding English one is FLINK-13277
> cc [~xuefuz] [~lirui] [~Terry1897]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13278) add Chinese documentation for Hive source/sink
[
https://issues.apache.org/jira/browse/FLINK-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-13278:
Fix Version/s: (was: 1.9.1)
1.9.2
> add Chinese documentation for Hive source/sink
> --
>
> Key: FLINK-13278
> URL: https://issues.apache.org/jira/browse/FLINK-13278
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Hive, Documentation
>Reporter: Bowen Li
>Priority: Major
> Fix For: 1.9.2
>
>
> add Chinese documentation of Hive source/sink in {{batch/connector_zh.md}}
> it's corresponding English one is FLINK-13277
> cc [~xuefuz] [~lirui] [~Terry1897]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-13278) add Chinese documentation for Hive source/sink
[
https://issues.apache.org/jira/browse/FLINK-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu reassigned FLINK-13278:
---
Assignee: Zijie Lu
> add Chinese documentation for Hive source/sink
> --
>
> Key: FLINK-13278
> URL: https://issues.apache.org/jira/browse/FLINK-13278
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Hive, Documentation
>Reporter: Bowen Li
>Assignee: Zijie Lu
>Priority: Major
> Fix For: 1.9.2
>
>
> add Chinese documentation of Hive source/sink in {{batch/connector_zh.md}}
> it's corresponding English one is FLINK-13277
> cc [~xuefuz] [~lirui] [~Terry1897]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13271) Add documentation for all the new features of blink planner
[ https://issues.apache.org/jira/browse/FLINK-13271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-13271: Fix Version/s: (was: 1.9.1) 1.9.2 > Add documentation for all the new features of blink planner > --- > > Key: FLINK-13271 > URL: https://issues.apache.org/jira/browse/FLINK-13271 > Project: Flink > Issue Type: Task > Components: Documentation, Table SQL / Planner >Affects Versions: 1.9.0 >Reporter: Jark Wu >Assignee: Jark Wu >Priority: Major > Labels: pull-request-available > Fix For: 1.9.2 > > Time Spent: 10m > Remaining Estimate: 0h > > This is an umbrella issue to track documentations for blink planner. All new > features introduced by blink planner, or behavior different with flink > planner should be documented. > Structure and Tasks are proposed in the google doc: > https://docs.google.com/document/d/1xcI77x-15CbSPOdluRaFzx7jf2_V2SBOrlTyDPhIUHE/edit# > Subtasks will be added later. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13086) add Chinese documentation for catalogs
[ https://issues.apache.org/jira/browse/FLINK-13086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-13086: Fix Version/s: (was: 1.9.1) 1.9.2 > add Chinese documentation for catalogs > -- > > Key: FLINK-13086 > URL: https://issues.apache.org/jira/browse/FLINK-13086 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Connectors / Hive, Documentation, > Table SQL / API >Reporter: Bowen Li >Assignee: frank wang >Priority: Major > Labels: pull-request-available > Fix For: 1.9.2 > > Time Spent: 40m > Remaining Estimate: 0h > > the ticket for corresponding English documentation is FLINK-12277 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14118) Reduce the unnecessary flushing when there is no data available for flush
[ https://issues.apache.org/jira/browse/FLINK-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-14118: Fix Version/s: (was: 1.8.3) (was: 1.9.1) > Reduce the unnecessary flushing when there is no data available for flush > - > > Key: FLINK-14118 > URL: https://issues.apache.org/jira/browse/FLINK-14118 > Project: Flink > Issue Type: Improvement > Components: Runtime / Network >Reporter: Yingjie Cao >Priority: Critical > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > > The new flush implementation which works by triggering a netty user event may > cause performance regression compared to the old synchronization-based one. > More specifically, when there is exactly one BufferConsumer in the buffer > queue of subpartition and no new data will be added for a while in the future > (may because of just no input or the logic of the operator is to collect some > data for processing and will not emit records immediately), that is, there is > no data to send, the OutputFlusher will continuously notify data available > and wake up the netty thread, though no data will be returned by the > pollBuffer method. > For some of our production jobs, this will incur 20% to 40% CPU overhead > compared to the old implementation. We tried to fix the problem by checking > if there is new data available when flushing, if there is no new data, the > netty thread will not be notified. It works for our jobs and the cpu usage > falls to previous level. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-12894) add Chinese documentation of how to configure and use catalogs in SQL CLI
[ https://issues.apache.org/jira/browse/FLINK-12894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-12894: Fix Version/s: (was: 1.9.1) 1.9.2 > add Chinese documentation of how to configure and use catalogs in SQL CLI > - > > Key: FLINK-12894 > URL: https://issues.apache.org/jira/browse/FLINK-12894 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Connectors / Hive, Documentation, > Table SQL / Client >Reporter: Bowen Li >Assignee: frank wang >Priority: Major > Labels: pull-request-available > Fix For: 1.9.2 > > Time Spent: 1h > Remaining Estimate: 0h > > Ticket of its corresponding English version is FLINK-12627. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14118) Reduce the unnecessary flushing when there is no data available for flush
[ https://issues.apache.org/jira/browse/FLINK-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940221#comment-16940221 ] Jark Wu commented on FLINK-14118: - Thanks [~kevin.cyj] and [~pnowojski], as discussed in the issue, I will remove the fix version for 1.8 and 1.9 for now. > Reduce the unnecessary flushing when there is no data available for flush > - > > Key: FLINK-14118 > URL: https://issues.apache.org/jira/browse/FLINK-14118 > Project: Flink > Issue Type: Improvement > Components: Runtime / Network >Reporter: Yingjie Cao >Priority: Critical > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1, 1.8.3 > > Time Spent: 10m > Remaining Estimate: 0h > > The new flush implementation which works by triggering a netty user event may > cause performance regression compared to the old synchronization-based one. > More specifically, when there is exactly one BufferConsumer in the buffer > queue of subpartition and no new data will be added for a while in the future > (may because of just no input or the logic of the operator is to collect some > data for processing and will not emit records immediately), that is, there is > no data to send, the OutputFlusher will continuously notify data available > and wake up the netty thread, though no data will be returned by the > pollBuffer method. > For some of our production jobs, this will incur 20% to 40% CPU overhead > compared to the old implementation. We tried to fix the problem by checking > if there is new data available when flushing, if there is no new data, the > netty thread will not be notified. It works for our jobs and the cpu usage > falls to previous level. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (FLINK-14049) Update error message for failed partition updates to include task name
[ https://issues.apache.org/jira/browse/FLINK-14049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu resolved FLINK-14049. - Fix Version/s: (was: 1.8.3) Resolution: Fixed 1.10.0: fecd108d2b5164e65b46d61ffe5e69142f2c84d5 1.9.1: abaa901adc6a115756b2099339d80c992ceb14f0 > Update error message for failed partition updates to include task name > -- > > Key: FLINK-14049 > URL: https://issues.apache.org/jira/browse/FLINK-14049 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.9.0 >Reporter: Stephan Ewen >Assignee: Stephan Ewen >Priority: Critical > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1 > > Time Spent: 20m > Remaining Estimate: 0h > > The error message for failed partition updates does not include the task name. > That makes it useless during debugging. > Adding the task name is a simple addition that make this error message much > more helpful. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13910) Many serializable classes have no explicit 'serialVersionUID'
[
https://issues.apache.org/jira/browse/FLINK-13910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-13910:
Fix Version/s: (was: 1.9.1)
1.9.2
> Many serializable classes have no explicit 'serialVersionUID'
> -
>
> Key: FLINK-13910
> URL: https://issues.apache.org/jira/browse/FLINK-13910
> Project: Flink
> Issue Type: Bug
> Components: API / Type Serialization System
>Reporter: Yun Tang
>Priority: Critical
> Fix For: 1.10.0, 1.9.2
>
> Attachments: SerializableNoSerialVersionUIDField
>
>
> Currently, many serializable classes in Flink have no explicit
> 'serialVersionUID'. As [official
> doc|https://flink.apache.org/contributing/code-style-and-quality-java.html#java-serialization]
> said, {{Serializable classes must define a Serial Version UID}}.
> No 'serialVersionUID' would cause compatibility problem. Take
> {{TwoPhaseCommitSinkFunction}} for example, since no explicit
> 'serialVersionUID' defined, after
> [FLINK-10455|https://github.com/apache/flink/commit/489be82a6d93057ed4a3f9bf38ef50d01d11d96b]
> introduced, its default 'serialVersionUID' has changed from
> "4584405056408828651" to "4064406918549730832". In other words, if we submit
> a job from Flink-1.6.3 local home to remote Flink-1.6.2 cluster with the
> usage of {{TwoPhaseCommitSinkFunction}}, we would get exception like:
> {code:java}
> org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot
> instantiate user function.
> at
> org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(StreamConfig.java:239)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.(OperatorChain.java:104)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:267)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.InvalidClassException:
> org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction;
> local class incompatible: stream classdesc serialVersionUID =
> 4584405056408828651, local class serialVersionUID = 4064406918549730832
> at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)
> at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1885)
> at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751)
> at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1885)
> at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
> at
> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:537)
> at
> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:524)
> at
> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:512)
> at
> org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:473)
> at
> org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(StreamConfig.java:224)
> ... 4 more
> {code}
> Similar problems existed in
> {{org.apache.flink.streaming.api.operators.SimpleOperatorFactory}} which has
> different 'serialVersionUID' from release-1.9 and current master branch.
> IMO, we might have two options to fix this bug:
> # Add explicit serialVersionUID for those classes which is identical to
> latest Flink-1.9.0 release code.
> # Use similar mechanism like {{FailureTolerantObjectInputStream}} in
> {{InstantiationUtil}} to ignore serialVersionUID mismatch.
> I have collected all production classes without serialVersionUID from latest
> master branch in the attachment, which counts to 639 classes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13740) TableAggregateITCase.testNonkeyedFlatAggregate failed on Travis
[
https://issues.apache.org/jira/browse/FLINK-13740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-13740:
Fix Version/s: (was: 1.9.1)
1.9.2
> TableAggregateITCase.testNonkeyedFlatAggregate failed on Travis
> ---
>
> Key: FLINK-13740
> URL: https://issues.apache.org/jira/browse/FLINK-13740
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: test-stability
> Fix For: 1.10.0, 1.9.2
>
>
> The {{TableAggregateITCase.testNonkeyedFlatAggregate}} failed on Travis with
> {code}
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> at
> org.apache.flink.table.planner.runtime.stream.table.TableAggregateITCase.testNonkeyedFlatAggregate(TableAggregateITCase.scala:93)
> Caused by: java.lang.Exception: Artificial Failure
> {code}
> https://api.travis-ci.com/v3/job/225551182/log.txt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13733) FlinkKafkaInternalProducerITCase.testHappyPath fails on Travis
[
https://issues.apache.org/jira/browse/FLINK-13733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-13733:
Fix Version/s: (was: 1.9.1)
1.9.2
> FlinkKafkaInternalProducerITCase.testHappyPath fails on Travis
> --
>
> Key: FLINK-13733
> URL: https://issues.apache.org/jira/browse/FLINK-13733
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Tests
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Till Rohrmann
>Assignee: Jiangjie Qin
>Priority: Critical
> Labels: test-stability
> Fix For: 1.10.0, 1.9.2
>
>
> The {{FlinkKafkaInternalProducerITCase.testHappyPath}} fails on Travis with
> {code}
> Test
> testHappyPath(org.apache.flink.streaming.connectors.kafka.FlinkKafkaInternalProducerITCase)
> failed with:
> java.util.NoSuchElementException
> at
> org.apache.kafka.common.utils.AbstractIterator.next(AbstractIterator.java:52)
> at
> org.apache.flink.shaded.guava18.com.google.common.collect.Iterators.getOnlyElement(Iterators.java:302)
> at
> org.apache.flink.shaded.guava18.com.google.common.collect.Iterables.getOnlyElement(Iterables.java:289)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaInternalProducerITCase.assertRecord(FlinkKafkaInternalProducerITCase.java:169)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaInternalProducerITCase.testHappyPath(FlinkKafkaInternalProducerITCase.java:70)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> https://api.travis-ci.org/v3/job/571870358/log.txt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13708) transformations should be cleared because a table environment could execute multiple job
[ https://issues.apache.org/jira/browse/FLINK-13708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-13708: Fix Version/s: (was: 1.9.1) 1.9.2 > transformations should be cleared because a table environment could execute > multiple job > > > Key: FLINK-13708 > URL: https://issues.apache.org/jira/browse/FLINK-13708 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Reporter: godfrey he >Assignee: godfrey he >Priority: Critical > Labels: pull-request-available > Fix For: 1.10.0, 1.9.2 > > Time Spent: 10m > Remaining Estimate: 0h > > currently, if a table environment execute more than one sql jobs, the > following job contains transformations about the previous job. the reason is > the transformations is not cleared after execution -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9781: [FLINK-14098] Support multiple statements for TableEnvironment.
flinkbot edited a comment on issue #9781: [FLINK-14098] Support multiple statements for TableEnvironment. URL: https://github.com/apache/flink/pull/9781#issuecomment-535813967 ## CI report: * ed0e04d1cd481b096d68d0c9bf36a997dcca2550 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129400381) * ec8fccdeb53c5e2e0f089708753be33839706e2a : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129441309) * ad27f8e71d91534867d093f00cde3d5b81534282 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129526289) * 5247e6b03266d0d25d42a185e0a447643116d96f : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129545862) * bdad4f2c28154dd4d9d4ab05b9202c47644d4c9f : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13862) Remove or rewrite Execution Plan docs
[
https://issues.apache.org/jira/browse/FLINK-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940215#comment-16940215
]
Jark Wu commented on FLINK-13862:
-
As this is a documentation issue, I would like to not block 1.9.1. So I moved
it to 1.9.2.
> Remove or rewrite Execution Plan docs
> -
>
> Key: FLINK-13862
> URL: https://issues.apache.org/jira/browse/FLINK-13862
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.9.0
>Reporter: Stephan Ewen
>Priority: Blocker
> Fix For: 1.10.0, 1.9.2
>
>
> The *Execution Plans* section is totally outdated and refers to the old
> {{tools/planVisalizer.html}} file that has been removed for two years.
> https://ci.apache.org/projects/flink/flink-docs-master/dev/execution_plans.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13862) Remove or rewrite Execution Plan docs
[
https://issues.apache.org/jira/browse/FLINK-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-13862:
Fix Version/s: (was: 1.9.1)
1.9.2
> Remove or rewrite Execution Plan docs
> -
>
> Key: FLINK-13862
> URL: https://issues.apache.org/jira/browse/FLINK-13862
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.9.0
>Reporter: Stephan Ewen
>Priority: Blocker
> Fix For: 1.10.0, 1.9.2
>
>
> The *Execution Plans* section is totally outdated and refers to the old
> {{tools/planVisalizer.html}} file that has been removed for two years.
> https://ci.apache.org/projects/flink/flink-docs-master/dev/execution_plans.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9749: [FLINK-14115][docs-zh] Translate DataStream Code Walkthrough to Chinese
flinkbot edited a comment on issue #9749: [FLINK-14115][docs-zh] Translate DataStream Code Walkthrough to Chinese URL: https://github.com/apache/flink/pull/9749#issuecomment-534149758 ## CI report: * 141eb6db531af4b12a9506489f280f180c02a408 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/128770115) * 0b96f066d276e4a483785f01335d36876e0e971c : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/128887957) * 5d7e1626653293ea20f935829d553de3b3e421fe : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129282782) * 16fab72a9ee8ea4940082d9fd32f0331f6271540 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/129455836) * 087f60aef7141fc77a9dd632ca01bf6319bbf29d : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/129459090) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14059) Introduce option allSourcesInSamePipelinedRegion in ExecutionConfig
[
https://issues.apache.org/jira/browse/FLINK-14059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940214#comment-16940214
]
Zhu Zhu commented on FLINK-14059:
-
Thanks [~xintongsong] for the response!
Will change {{allSourcesInSamePipelinedRegion}} ->
{{allVerticesInSameSlotSharingGroupByDefault}} then.
> Introduce option allSourcesInSamePipelinedRegion in ExecutionConfig
> ---
>
> Key: FLINK-14059
> URL: https://issues.apache.org/jira/browse/FLINK-14059
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination, Table SQL / Runtime
>Affects Versions: 1.10.0
>Reporter: Xintong Song
>Assignee: Zhu Zhu
>Priority: Major
> Fix For: 1.10.0
>
>
> * Introduce option {{allSourcesInSamePipelinedRegion}} in {{ExecutionConfig}}
> * Set it to {{true}} by default
> * Set it to {{false}} for SQL/Table API bounded batch jobs by the Blink
> planner
> This step should not introduce any behavior changes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] Mrart edited a comment on issue #9542: [FLINK-13873][metrics] Change the column family as tags for influxdb …
Mrart edited a comment on issue #9542: [FLINK-13873][metrics] Change the column family as tags for influxdb … URL: https://github.com/apache/flink/pull/9542#issuecomment-536240909 @zentol , I add a config option like "state.backend.rocksdb.column-family-as-variable" for control the column_family This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] Mrart edited a comment on issue #9542: [FLINK-13873][metrics] Change the column family as tags for influxdb …
Mrart edited a comment on issue #9542: [FLINK-13873][metrics] Change the column family as tags for influxdb … URL: https://github.com/apache/flink/pull/9542#issuecomment-536240909 @zentol , I have add a config option like "state.backend.rocksdb.column-family-as-variable" for control the column_family This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (FLINK-13277) add documentation of Hive source/sink
[
https://issues.apache.org/jira/browse/FLINK-13277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu resolved FLINK-13277.
-
Resolution: Fixed
1.10.0: 3d3e6c11ba2a303dc9a19059406a597cf9348f3f
1.9.1: 2d4d8a2af5bc602dd181b9d644bee6b8d2f05c12
> add documentation of Hive source/sink
> -
>
> Key: FLINK-13277
> URL: https://issues.apache.org/jira/browse/FLINK-13277
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Hive, Documentation
>Reporter: Bowen Li
>Assignee: Rui Li
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.10.0, 1.9.1
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> add documentation of Hive source/sink in {{batch/connector.md}}
> its corresponding Chinese one is FLINK-13278
> cc [~xuefuz] [~lirui] [~Terry1897]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] gaofeilong198810 commented on issue #9749: [FLINK-14115][docs-zh] Translate DataStream Code Walkthrough to Chinese
gaofeilong198810 commented on issue #9749: [FLINK-14115][docs-zh] Translate DataStream Code Walkthrough to Chinese URL: https://github.com/apache/flink/pull/9749#issuecomment-536240774 @flinkbot run travis This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] hequn8128 commented on a change in pull request #9766: [FLINK-14018][python] Add Python building blocks to make sure the basic functionality of Python ScalarFunction could work
hequn8128 commented on a change in pull request #9766: [FLINK-14018][python] Add Python building blocks to make sure the basic functionality of Python ScalarFunction could work URL: https://github.com/apache/flink/pull/9766#discussion_r329309345 ## File path: flink-python/pyflink/table/udf.py ## @@ -0,0 +1,217 @@ + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import collections +import functools +import inspect +from abc import ABCMeta, abstractmethod + +from pyflink.java_gateway import get_gateway +from pyflink.table.types import DataType, _to_java_type +from pyflink.util import utils + +__all__ = ['FunctionContext', 'ScalarFunction', 'udf'] + + +class FunctionContext(object): +""" +Used to obtain global runtime information about the context in which the +user-defined function is executed. The information includes the metric group, +and global job parameters, etc. +""" +pass + + +class UserDefinedFunction(object): +""" +Base interface for user-defined function. +""" +__metaclass__ = ABCMeta + +def open(self, function_context): +""" +Initialization method for the function. It is called before the actual working methods +and thus suitable for one time setup work. + +:param function_context: the context of the function +:type function_context: FunctionContext +""" +pass + +def close(self): +""" +Tear-down method for the user code. It is called after the last call to the main +working methods. +""" +pass + +def is_deterministic(self): +""" +Returns information about the determinism of the function's results. +It returns true if and only if a call to this function is guaranteed to +always return the same result given the same parameters. true is assumed by default. +If the function is not pure functional like random(), date(), now(), +this method must return false. + +:return: the determinism of the function's results. +:rtype: bool +""" +return True + + +class ScalarFunction(UserDefinedFunction): +""" +Base interface for user-defined scalar function. A user-defined scalar functions maps zero, one, +or multiple scalar values to a new scalar value. +""" + +@abstractmethod +def eval(self, *args): +""" +Method which defines the logic of the scalar function. +""" +pass + + +class LambdaScalarFunction(ScalarFunction): +""" +Helper scalar function implementation for lambda function. It's for internal use only. +""" + +def __init__(self, func): +self.func = func + +def eval(self, *args): +return self.func(*args) + + +class UserDefinedFunctionWrapper(object): +""" +Wrapper for Python user-defined function. It handles things like converting lambda +functions to user-defined functions, creating the Java user-defined function representation, +etc. +""" + +def __init__(self, func, input_types, result_type, name=None, deterministic=True): +if inspect.isclass(func) or ( +not isinstance(func, UserDefinedFunction) and not callable(func)): +raise TypeError( Review comment: Add tests for these validations. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] hequn8128 commented on a change in pull request #9766: [FLINK-14018][python] Add Python building blocks to make sure the basic functionality of Python ScalarFunction could work
hequn8128 commented on a change in pull request #9766: [FLINK-14018][python]
Add Python building blocks to make sure the basic functionality of Python
ScalarFunction could work
URL: https://github.com/apache/flink/pull/9766#discussion_r329310180
##
File path: flink-python/pyflink/table/udf.py
##
@@ -0,0 +1,217 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import collections
+import functools
+import inspect
+from abc import ABCMeta, abstractmethod
+
+from pyflink.java_gateway import get_gateway
+from pyflink.table.types import DataType, _to_java_type
+from pyflink.util import utils
+
+__all__ = ['FunctionContext', 'ScalarFunction', 'udf']
+
+
+class FunctionContext(object):
+"""
+Used to obtain global runtime information about the context in which the
+user-defined function is executed. The information includes the metric
group,
+and global job parameters, etc.
+"""
+pass
+
+
+class UserDefinedFunction(object):
+"""
+Base interface for user-defined function.
+"""
+__metaclass__ = ABCMeta
+
+def open(self, function_context):
+"""
+Initialization method for the function. It is called before the actual
working methods
+and thus suitable for one time setup work.
+
+:param function_context: the context of the function
+:type function_context: FunctionContext
+"""
+pass
+
+def close(self):
+"""
+Tear-down method for the user code. It is called after the last call
to the main
+working methods.
+"""
+pass
+
+def is_deterministic(self):
+"""
+Returns information about the determinism of the function's results.
+It returns true if and only if a call to this function is guaranteed to
+always return the same result given the same parameters. true is
assumed by default.
+If the function is not pure functional like random(), date(), now(),
+this method must return false.
+
+:return: the determinism of the function's results.
+:rtype: bool
+"""
+return True
+
+
+class ScalarFunction(UserDefinedFunction):
+"""
+Base interface for user-defined scalar function. A user-defined scalar
functions maps zero, one,
+or multiple scalar values to a new scalar value.
+"""
+
+@abstractmethod
+def eval(self, *args):
+"""
+Method which defines the logic of the scalar function.
+"""
+pass
+
+
+class LambdaScalarFunction(ScalarFunction):
+"""
+Helper scalar function implementation for lambda function. It's for
internal use only.
+"""
+
+def __init__(self, func):
+self.func = func
+
+def eval(self, *args):
+return self.func(*args)
+
+
+class UserDefinedFunctionWrapper(object):
+"""
+Wrapper for Python user-defined function. It handles things like
converting lambda
+functions to user-defined functions, creating the Java user-defined
function representation,
+etc.
+"""
+
+def __init__(self, func, input_types, result_type, name=None,
deterministic=True):
+if inspect.isclass(func) or (
+not isinstance(func, UserDefinedFunction) and not
callable(func)):
+raise TypeError(
+"Invalid function: not a function or callable (__call__ is not
defined): "
+"{0}".format(type(func)))
+
+if not isinstance(input_types, collections.Iterable):
+input_types = [input_types]
+
+for input_type in input_types:
+if not isinstance(input_type, DataType):
+raise TypeError(
+"Invalid input_type: input_type should be DataType "
+"but contains {}".format(input_type))
+
+if not isinstance(result_type, DataType):
+raise TypeError(
+"Invalid returnType: returnType should be DataType "
+"but is {}".format(result_type))
+
+self._func = func
+self._input_types
[GitHub] [flink] hequn8128 commented on a change in pull request #9766: [FLINK-14018][python] Add Python building blocks to make sure the basic functionality of Python ScalarFunction could work
hequn8128 commented on a change in pull request #9766: [FLINK-14018][python] Add Python building blocks to make sure the basic functionality of Python ScalarFunction could work URL: https://github.com/apache/flink/pull/9766#discussion_r329304847 ## File path: flink-python/pyflink/table/udf.py ## @@ -0,0 +1,217 @@ + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import collections +import functools +import inspect +from abc import ABCMeta, abstractmethod + +from pyflink.java_gateway import get_gateway +from pyflink.table.types import DataType, _to_java_type +from pyflink.util import utils + +__all__ = ['FunctionContext', 'ScalarFunction', 'udf'] + + +class FunctionContext(object): +""" +Used to obtain global runtime information about the context in which the +user-defined function is executed. The information includes the metric group, +and global job parameters, etc. +""" +pass + + +class UserDefinedFunction(object): +""" +Base interface for user-defined function. +""" +__metaclass__ = ABCMeta Review comment: Can we remove this? The child class implements it through subclassing instead of register. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] hequn8128 commented on a change in pull request #9766: [FLINK-14018][python] Add Python building blocks to make sure the basic functionality of Python ScalarFunction could work
hequn8128 commented on a change in pull request #9766: [FLINK-14018][python] Add Python building blocks to make sure the basic functionality of Python ScalarFunction could work URL: https://github.com/apache/flink/pull/9766#discussion_r329310696 ## File path: flink-python/pyflink/table/udf.py ## @@ -0,0 +1,217 @@ + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import collections +import functools +import inspect +from abc import ABCMeta, abstractmethod + +from pyflink.java_gateway import get_gateway +from pyflink.table.types import DataType, _to_java_type +from pyflink.util import utils + +__all__ = ['FunctionContext', 'ScalarFunction', 'udf'] + + +class FunctionContext(object): +""" +Used to obtain global runtime information about the context in which the +user-defined function is executed. The information includes the metric group, +and global job parameters, etc. +""" +pass + + +class UserDefinedFunction(object): +""" +Base interface for user-defined function. +""" +__metaclass__ = ABCMeta + +def open(self, function_context): +""" +Initialization method for the function. It is called before the actual working methods +and thus suitable for one time setup work. + +:param function_context: the context of the function +:type function_context: FunctionContext +""" +pass + +def close(self): +""" +Tear-down method for the user code. It is called after the last call to the main +working methods. +""" +pass + +def is_deterministic(self): +""" +Returns information about the determinism of the function's results. +It returns true if and only if a call to this function is guaranteed to +always return the same result given the same parameters. true is assumed by default. +If the function is not pure functional like random(), date(), now(), +this method must return false. + +:return: the determinism of the function's results. +:rtype: bool +""" +return True + + +class ScalarFunction(UserDefinedFunction): +""" +Base interface for user-defined scalar function. A user-defined scalar functions maps zero, one, +or multiple scalar values to a new scalar value. +""" + +@abstractmethod +def eval(self, *args): +""" +Method which defines the logic of the scalar function. +""" +pass + + +class LambdaScalarFunction(ScalarFunction): Review comment: This class is used for both lambda and python functions. Maybe we can change the name to `DelegatingScalarFunction`? (Also change the comments below) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] hequn8128 commented on a change in pull request #9766: [FLINK-14018][python] Add Python building blocks to make sure the basic functionality of Python ScalarFunction could work
hequn8128 commented on a change in pull request #9766: [FLINK-14018][python] Add Python building blocks to make sure the basic functionality of Python ScalarFunction could work URL: https://github.com/apache/flink/pull/9766#discussion_r329295245 ## File path: flink-python/pyflink/table/table_environment.py ## @@ -542,6 +542,36 @@ def register_java_function(self, name, function_class_name): .loadClass(function_class_name).newInstance() self._j_tenv.registerFunction(name, java_function) +def register_function(self, name, function): Review comment: Previously, in `test_environment_completeness.py`, we ignore this method when check the Api alignment between python and Java. We should also check this method from now on. We can just remove the `registerFunction` in the `test_environment_completeness.py`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #9715: [FLINK-13993][runtime] Using FlinkUserCodeClassLoaders to load the user class in the perjob mode
zhuzhurk commented on a change in pull request #9715: [FLINK-13993][runtime]
Using FlinkUserCodeClassLoaders to load the user class in the perjob mode
URL: https://github.com/apache/flink/pull/9715#discussion_r329319129
##
File path:
flink-container/src/main/java/org/apache/flink/container/entrypoint/ClassPathJobGraphRetriever.java
##
@@ -110,12 +121,14 @@ public JobGraph retrieveJobGraph(Configuration
configuration) throws FlinkExcept
}
}
- private PackagedProgram createPackagedProgram() throws FlinkException {
+ private PackagedProgram createPackagedProgram() throws FlinkException,
IOException {
final String entryClass = getJobClassNameOrScanClassPath();
try {
- final Class mainClass =
getClass().getClassLoader().loadClass(entryClass);
- return new PackagedProgram(mainClass, programArguments);
- } catch (ClassNotFoundException | ProgramInvocationException e)
{
+ final String userClassPaths = getUserClassPaths().
+
stream().map(URL::toString).collect(Collectors.joining(":"));
Review comment:
According to the [code style
guide](https://flink.apache.org/contributing/code-style-and-quality-formatting.html#breaking-the-lines-of-too-long-statements),
each call should be in a new line when breaking long statements.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #9715: [FLINK-13993][runtime] Using FlinkUserCodeClassLoaders to load the user class in the perjob mode
zhuzhurk commented on a change in pull request #9715: [FLINK-13993][runtime]
Using FlinkUserCodeClassLoaders to load the user class in the perjob mode
URL: https://github.com/apache/flink/pull/9715#discussion_r329321484
##
File path:
flink-container/src/test/java/org/apache/flink/container/entrypoint/ClassPathJobGraphRetrieverTest.java
##
@@ -55,6 +66,62 @@
private static final String[] PROGRAM_ARGUMENTS = {"--arg", "suffix"};
+ private static URL testJobForUserClassLoaderJarURL = null;
+ private static URL testJobDependedJarURL = null;
+
+ private Path userDirHasEntryClass;
+ private Path userJarHasEntryClass;
+ private Path userEntryClassDependJarAtUserDirHasEntryClass;
+ private Path textAtUserDirHasEntryClass;
+
+ private Path userDirHasNotEntryClass;
+ private Path userJarHasNotEntryClass;
+ private Path userEntryClassDependJarAtUserDirHasNotEntryClass;
+ private Path textAtUserDirHasNotEntryClass;
+
+
Review comment:
These duplicated empty lines can be removed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #9715: [FLINK-13993][runtime] Using FlinkUserCodeClassLoaders to load the user class in the perjob mode
zhuzhurk commented on a change in pull request #9715: [FLINK-13993][runtime]
Using FlinkUserCodeClassLoaders to load the user class in the perjob mode
URL: https://github.com/apache/flink/pull/9715#discussion_r329320115
##
File path:
flink-container/src/main/java/org/apache/flink/container/entrypoint/StandaloneJobClusterEntryPoint.java
##
@@ -77,7 +81,9 @@ private StandaloneJobClusterEntryPoint(
protected DispatcherResourceManagerComponentFactory
createDispatcherResourceManagerComponentFactory(Configuration configuration) {
return new JobDispatcherResourceManagerComponentFactory(
StandaloneResourceManagerFactory.INSTANCE,
- new ClassPathJobGraphRetriever(jobId,
savepointRestoreSettings, programArguments, jobClassName));
+ new ClassPathJobGraphRetriever(jobId,
savepointRestoreSettings, programArguments, jobClassName,
+ Paths.get(System.getenv(ENV_FLINK_HOME_DIR) ==
null ? DEFAULT_FLINK_HOME : System.getenv(ENV_FLINK_HOME_DIR),
Review comment:
Hard coding "/opt/flink" looks a bit tricky.
Is it possible to we set "FLINK_HOME" in all cases? So that we can throw
exception here for unexpected cases.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9795: [hotfix] [variable name] Typo in Table Planner RuleSets: JOIN_REORDER_PERPARE_RULES
flinkbot edited a comment on issue #9795: [hotfix] [variable name] Typo in Table Planner RuleSets: JOIN_REORDER_PERPARE_RULES URL: https://github.com/apache/flink/pull/9795#issuecomment-536208693 ## CI report: * 1055410a5888dcada8045f57f5100dc09f095db9 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129557803) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9795: [hotfix] [variable name] Typo in Table Planner RuleSets: JOIN_REORDER_PERPARE_RULES
flinkbot edited a comment on issue #9795: [hotfix] [variable name] Typo in Table Planner RuleSets: JOIN_REORDER_PERPARE_RULES URL: https://github.com/apache/flink/pull/9795#issuecomment-536208693 ## CI report: * 1055410a5888dcada8045f57f5100dc09f095db9 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/129557803) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #9715: [FLINK-13993][runtime] Using FlinkUserCodeClassLoaders to load the user class in the perjob mode
zhuzhurk commented on a change in pull request #9715: [FLINK-13993][runtime]
Using FlinkUserCodeClassLoaders to load the user class in the perjob mode
URL: https://github.com/apache/flink/pull/9715#discussion_r329318669
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/entrypoint/component/AbstractUserClassPathJobGraphRetriever.java
##
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.entrypoint.component;
+
+import org.apache.flink.core.fs.Path;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URL;
+import java.nio.file.FileVisitOption;
+import java.nio.file.FileVisitResult;
+import java.nio.file.Files;
+import java.nio.file.SimpleFileVisitor;
+import java.nio.file.attribute.BasicFileAttributes;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.LinkedList;
+import java.util.List;
+
+/**
+ * Abstract class for the JobGraphRetriever, which wants to get classpath
user's code depends on.
+ */
+
+public abstract class AbstractUserClassPathJobGraphRetriever implements
JobGraphRetriever {
+
+ private final Logger log = LoggerFactory.getLogger(getClass());
Review comment:
Or maybe we can make it a private static logger for itself.
And then we can make `scanJarsInJobClassDir` static.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #9795: [hotfix] [variable name] Typo in Table Planner RuleSets: JOIN_REORDER_PERPARE_RULES
flinkbot commented on issue #9795: [hotfix] [variable name] Typo in Table Planner RuleSets: JOIN_REORDER_PERPARE_RULES URL: https://github.com/apache/flink/pull/9795#issuecomment-536208693 ## CI report: * 1055410a5888dcada8045f57f5100dc09f095db9 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #9795: [hotfix] [variable name] Typo in Table Planner RuleSets: JOIN_REORDER_PERPARE_RULES
flinkbot commented on issue #9795: [hotfix] [variable name] Typo in Table Planner RuleSets: JOIN_REORDER_PERPARE_RULES URL: https://github.com/apache/flink/pull/9795#issuecomment-536207542 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 1055410a5888dcada8045f57f5100dc09f095db9 (Sat Sep 28 17:05:18 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] benoitparis opened a new pull request #9795: [hotfix] [variable name] Typo in Table Planner RuleSets: JOIN_REORDER_PERPARE_RULES
benoitparis opened a new pull request #9795: [hotfix] [variable name] Typo in Table Planner RuleSets: JOIN_REORDER_PERPARE_RULES URL: https://github.com/apache/flink/pull/9795 ## What is the purpose of the change Fix a typo This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #9715: [FLINK-13993][runtime] Using FlinkUserCodeClassLoaders to load the user class in the perjob mode
zhuzhurk commented on a change in pull request #9715: [FLINK-13993][runtime]
Using FlinkUserCodeClassLoaders to load the user class in the perjob mode
URL: https://github.com/apache/flink/pull/9715#discussion_r329314051
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/entrypoint/component/AbstractUserClassPathJobGraphRetrieverTest.java
##
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.entrypoint.component;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URL;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Tests for the {@link AbstractUserClassPathJobGraphRetriever}.
+ */
+public class AbstractUserClassPathJobGraphRetrieverTest {
+
+ @Rule
+ public final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ /**
+* Test class.
+*/
+ public static class TestJobGraphRetrieverTest extends
AbstractUserClassPathJobGraphRetriever {
Review comment:
I guess the name should be `TestJobGraphRetriever`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #9715: [FLINK-13993][runtime] Using FlinkUserCodeClassLoaders to load the user class in the perjob mode
zhuzhurk commented on a change in pull request #9715: [FLINK-13993][runtime]
Using FlinkUserCodeClassLoaders to load the user class in the perjob mode
URL: https://github.com/apache/flink/pull/9715#discussion_r329317178
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/entrypoint/component/AbstractUserClassPathJobGraphRetriever.java
##
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.entrypoint.component;
+
+import org.apache.flink.core.fs.Path;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URL;
+import java.nio.file.FileVisitOption;
+import java.nio.file.FileVisitResult;
+import java.nio.file.Files;
+import java.nio.file.SimpleFileVisitor;
+import java.nio.file.attribute.BasicFileAttributes;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.LinkedList;
+import java.util.List;
+
+/**
+ * Abstract class for the JobGraphRetriever, which wants to get classpath
user's code depends on.
+ */
+
+public abstract class AbstractUserClassPathJobGraphRetriever implements
JobGraphRetriever {
+
+ private final Logger log = LoggerFactory.getLogger(getClass());
Review comment:
I think the logger should be in protected scope. And
`AbstractUserClassPathJobGraphRetriever` subclasses should use it rather than
creating loggers by themselves.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #9715: [FLINK-13993][runtime] Using FlinkUserCodeClassLoaders to load the user class in the perjob mode
zhuzhurk commented on a change in pull request #9715: [FLINK-13993][runtime]
Using FlinkUserCodeClassLoaders to load the user class in the perjob mode
URL: https://github.com/apache/flink/pull/9715#discussion_r329317918
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/entrypoint/component/AbstractUserClassPathJobGraphRetriever.java
##
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.entrypoint.component;
+
+import org.apache.flink.core.fs.Path;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URL;
+import java.nio.file.FileVisitOption;
+import java.nio.file.FileVisitResult;
+import java.nio.file.Files;
+import java.nio.file.SimpleFileVisitor;
+import java.nio.file.attribute.BasicFileAttributes;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.LinkedList;
+import java.util.List;
+
+/**
+ * Abstract class for the JobGraphRetriever, which wants to get classpath
user's code depends on.
+ */
+
+public abstract class AbstractUserClassPathJobGraphRetriever implements
JobGraphRetriever {
+
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ public static final String DEFAULT_JOB_DIR = "job";
+
+ /** The directory contains all the jars, which user code depends on. */
+ @Nullable
+ private final String jobDir;
+
+ private List userClassPaths;
+
+ public AbstractUserClassPathJobGraphRetriever(String jobDir) {
+ this.jobDir = jobDir;
+ }
+
+ public List getUserClassPaths() throws IOException {
+ if (userClassPaths == null) {
+ userClassPaths = scanJarsInJobClassDir(jobDir);
+ }
+ return userClassPaths;
+ }
+
+ private List scanJarsInJobClassDir(String dir) throws IOException {
+
+ if (dir == null) {
+ return Collections.emptyList();
+ }
+
+ final File dirFile = new File(new Path(dir).toString());
+ final List jarURLs = new LinkedList<>();
+
+ if (!dirFile.exists()) {
+ log.warn("the job dir " + dirFile + " dose not
exists.");
+ return Collections.emptyList();
+ }
+ if (!dirFile.isDirectory()) {
+ log.warn("the job dir " + dirFile + " is not a
directory.");
+ return Collections.emptyList();
+ }
+
+ Files.walkFileTree(dirFile.toPath(),
Review comment:
I'm not sure whether there can be cycles in the file tree in all our cases.
If there is a cycle, it will cause `FileSystemLoopException` and the
`SimpleFileVisitor` will throw it directly and the job may fail.
If we want to deal with it, we need to override
`FileVisitResult#visitFileFailed(..)`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #9715: [FLINK-13993][runtime] Using FlinkUserCodeClassLoaders to load the user class in the perjob mode
zhuzhurk commented on a change in pull request #9715: [FLINK-13993][runtime]
Using FlinkUserCodeClassLoaders to load the user class in the perjob mode
URL: https://github.com/apache/flink/pull/9715#discussion_r329313956
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/entrypoint/component/AbstractUserClassPathJobGraphRetrieverTest.java
##
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.entrypoint.component;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URL;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Tests for the {@link AbstractUserClassPathJobGraphRetriever}.
+ */
+public class AbstractUserClassPathJobGraphRetrieverTest {
Review comment:
The test class should `extends TestLogger`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] TisonKun commented on a change in pull request #9788: [FLINK-14259] Introduce JobManagerRunner interface
TisonKun commented on a change in pull request #9788: [FLINK-14259] Introduce
JobManagerRunner interface
URL: https://github.com/apache/flink/pull/9788#discussion_r329314554
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobManagerRunnerImpl.java
##
@@ -0,0 +1,445 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.client.JobExecutionException;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.execution.librarycache.LibraryCacheManager;
+import org.apache.flink.runtime.executiongraph.ArchivedExecutionGraph;
+import org.apache.flink.runtime.highavailability.HighAvailabilityServices;
+import org.apache.flink.runtime.highavailability.RunningJobsRegistry;
+import
org.apache.flink.runtime.highavailability.RunningJobsRegistry.JobSchedulingStatus;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobmanager.OnCompletionActions;
+import org.apache.flink.runtime.jobmaster.factories.JobMasterServiceFactory;
+import org.apache.flink.runtime.leaderelection.LeaderContender;
+import org.apache.flink.runtime.leaderelection.LeaderElectionService;
+import org.apache.flink.runtime.messages.Acknowledge;
+import org.apache.flink.runtime.rpc.FatalErrorHandler;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.function.FunctionUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnull;
+
+import java.io.IOException;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionStage;
+import java.util.concurrent.Executor;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * The runner for the job manager. It deals with job level leader election and
make underlying job manager
+ * properly reacted.
+ */
+public class JobManagerRunnerImpl implements LeaderContender,
OnCompletionActions, JobManagerRunner {
+
+ private static final Logger log =
LoggerFactory.getLogger(JobManagerRunnerImpl.class);
+
+ //
+
+ /** Lock to ensure that this runner can deal with leader election event
and job completion notifies simultaneously. */
+ private final Object lock = new Object();
+
+ /** The job graph needs to run. */
+ private final JobGraph jobGraph;
+
+ /** Used to check whether a job needs to be run. */
+ private final RunningJobsRegistry runningJobsRegistry;
+
+ /** Leader election for this job. */
+ private final LeaderElectionService leaderElectionService;
+
+ private final LibraryCacheManager libraryCacheManager;
+
+ private final Executor executor;
+
+ private final JobMasterService jobMasterService;
+
+ private final FatalErrorHandler fatalErrorHandler;
+
+ private final CompletableFuture resultFuture;
+
+ private final CompletableFuture terminationFuture;
+
+ private CompletableFuture leadershipOperation;
+
+ /** flag marking the runner as shut down. */
+ private volatile boolean shutdown;
+
+ private volatile CompletableFuture
leaderGatewayFuture;
+
+ //
+
+ /**
+* Exceptions that occur while creating the JobManager or
JobManagerRunner are directly
Review comment:
Since legacy `JobManager` has been removed, it can be consider how we deal
with naming issue of `JobMaster` vs `JobManager`. Many times we refer to
`JobManager`/JM which actually means `ClusterEntrypoint`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this
[GitHub] [flink] TisonKun commented on a change in pull request #9788: [FLINK-14259] Introduce JobManagerRunner interface
TisonKun commented on a change in pull request #9788: [FLINK-14259] Introduce
JobManagerRunner interface
URL: https://github.com/apache/flink/pull/9788#discussion_r329315033
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/dispatcher/DispatcherTest.java
##
@@ -629,7 +630,7 @@ public void testJobSubmissionErrorAfterJobRecovery()
throws Exception {
}
/**
-* Tests that a blocking {@link JobManagerRunner} creation, e.g. due to
blocking FileSystem access,
+* Tests that a blocking {@link JobManagerRunnerImpl} creation, e.g.
due to blocking FileSystem access,
Review comment:
I think it is modified by auto refactor tools. In document we can only
document interface instead of implementation. Thus +1 replace this-alike
`JobManagerRunnerImpl` in document with `JobManagerRunner`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] TisonKun commented on a change in pull request #9788: [FLINK-14259] Introduce JobManagerRunner interface
TisonKun commented on a change in pull request #9788: [FLINK-14259] Introduce
JobManagerRunner interface
URL: https://github.com/apache/flink/pull/9788#discussion_r329314925
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobManagerRunnerImpl.java
##
@@ -0,0 +1,445 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.client.JobExecutionException;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.execution.librarycache.LibraryCacheManager;
+import org.apache.flink.runtime.executiongraph.ArchivedExecutionGraph;
+import org.apache.flink.runtime.highavailability.HighAvailabilityServices;
+import org.apache.flink.runtime.highavailability.RunningJobsRegistry;
+import
org.apache.flink.runtime.highavailability.RunningJobsRegistry.JobSchedulingStatus;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobmanager.OnCompletionActions;
+import org.apache.flink.runtime.jobmaster.factories.JobMasterServiceFactory;
+import org.apache.flink.runtime.leaderelection.LeaderContender;
+import org.apache.flink.runtime.leaderelection.LeaderElectionService;
+import org.apache.flink.runtime.messages.Acknowledge;
+import org.apache.flink.runtime.rpc.FatalErrorHandler;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.function.FunctionUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnull;
+
+import java.io.IOException;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionStage;
+import java.util.concurrent.Executor;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * The runner for the job manager. It deals with job level leader election and
make underlying job manager
+ * properly reacted.
+ */
+public class JobManagerRunnerImpl implements LeaderContender,
OnCompletionActions, JobManagerRunner {
+
+ private static final Logger log =
LoggerFactory.getLogger(JobManagerRunnerImpl.class);
+
+ //
+
+ /** Lock to ensure that this runner can deal with leader election event
and job completion notifies simultaneously. */
+ private final Object lock = new Object();
+
+ /** The job graph needs to run. */
+ private final JobGraph jobGraph;
+
+ /** Used to check whether a job needs to be run. */
+ private final RunningJobsRegistry runningJobsRegistry;
+
+ /** Leader election for this job. */
+ private final LeaderElectionService leaderElectionService;
+
+ private final LibraryCacheManager libraryCacheManager;
+
+ private final Executor executor;
+
+ private final JobMasterService jobMasterService;
+
+ private final FatalErrorHandler fatalErrorHandler;
+
+ private final CompletableFuture resultFuture;
+
+ private final CompletableFuture terminationFuture;
+
+ private CompletableFuture leadershipOperation;
+
+ /** flag marking the runner as shut down. */
+ private volatile boolean shutdown;
+
+ private volatile CompletableFuture
leaderGatewayFuture;
+
+ //
+
+ /**
+* Exceptions that occur while creating the JobManager or
JobManagerRunner are directly
+* thrown and not reported to the given {@code FatalErrorHandler}.
+*
+* @throws Exception Thrown if the runner cannot be set up, because
either one of the
+* required services could not be started, or the Job
could not be initialized.
+*/
+ public JobManagerRunnerImpl(
+ final JobGraph jobGraph,
+ final JobMasterServiceFactory jobMasterFactory,
+ final
[GitHub] [flink] TisonKun commented on a change in pull request #9788: [FLINK-14259] Introduce JobManagerRunner interface
TisonKun commented on a change in pull request #9788: [FLINK-14259] Introduce
JobManagerRunner interface
URL: https://github.com/apache/flink/pull/9788#discussion_r329314681
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobManagerRunnerImpl.java
##
@@ -0,0 +1,445 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.client.JobExecutionException;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.execution.librarycache.LibraryCacheManager;
+import org.apache.flink.runtime.executiongraph.ArchivedExecutionGraph;
+import org.apache.flink.runtime.highavailability.HighAvailabilityServices;
+import org.apache.flink.runtime.highavailability.RunningJobsRegistry;
+import
org.apache.flink.runtime.highavailability.RunningJobsRegistry.JobSchedulingStatus;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobmanager.OnCompletionActions;
+import org.apache.flink.runtime.jobmaster.factories.JobMasterServiceFactory;
+import org.apache.flink.runtime.leaderelection.LeaderContender;
+import org.apache.flink.runtime.leaderelection.LeaderElectionService;
+import org.apache.flink.runtime.messages.Acknowledge;
+import org.apache.flink.runtime.rpc.FatalErrorHandler;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.function.FunctionUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnull;
+
+import java.io.IOException;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionStage;
+import java.util.concurrent.Executor;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * The runner for the job manager. It deals with job level leader election and
make underlying job manager
+ * properly reacted.
+ */
+public class JobManagerRunnerImpl implements LeaderContender,
OnCompletionActions, JobManagerRunner {
+
+ private static final Logger log =
LoggerFactory.getLogger(JobManagerRunnerImpl.class);
+
+ //
+
+ /** Lock to ensure that this runner can deal with leader election event
and job completion notifies simultaneously. */
+ private final Object lock = new Object();
+
+ /** The job graph needs to run. */
+ private final JobGraph jobGraph;
+
+ /** Used to check whether a job needs to be run. */
+ private final RunningJobsRegistry runningJobsRegistry;
+
+ /** Leader election for this job. */
+ private final LeaderElectionService leaderElectionService;
+
+ private final LibraryCacheManager libraryCacheManager;
+
+ private final Executor executor;
+
+ private final JobMasterService jobMasterService;
+
+ private final FatalErrorHandler fatalErrorHandler;
+
+ private final CompletableFuture resultFuture;
+
+ private final CompletableFuture terminationFuture;
+
+ private CompletableFuture leadershipOperation;
+
+ /** flag marking the runner as shut down. */
+ private volatile boolean shutdown;
+
+ private volatile CompletableFuture
leaderGatewayFuture;
+
+ //
+
+ /**
+* Exceptions that occur while creating the JobManager or
JobManagerRunner are directly
+* thrown and not reported to the given {@code FatalErrorHandler}.
+*
+* @throws Exception Thrown if the runner cannot be set up, because
either one of the
+* required services could not be started, or the Job
could not be initialized.
+*/
+ public JobManagerRunnerImpl(
+ final JobGraph jobGraph,
Review comment:
It could be a possible discussion that whether we can get rid of `final`
[jira] [Reopened] (FLINK-14179) Wrong description of SqlCommand.SHOW_FUNCTIONS
[
https://issues.apache.org/jira/browse/FLINK-14179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Canbin Zheng reopened FLINK-14179:
--
Not merged yet.
> Wrong description of SqlCommand.SHOW_FUNCTIONS
> --
>
> Key: FLINK-14179
> URL: https://issues.apache.org/jira/browse/FLINK-14179
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.9.0
>Reporter: Canbin Zheng
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Attachments: image-2019-09-24-10-59-26-286.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently '*SHOW FUNCTIONS*' lists not only user-defined functions, but also
> system-defined ones, the description {color:#172b4d}*'Shows all registered
> user-defined functions.'* not correctly depicts this functionality. I think
> we can change the description to '*Shows all system-defined and user-defined
> functions.*'{color}
>
> {color:#172b4d}!image-2019-09-24-10-59-26-286.png!{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zhuzhurk commented on a change in pull request #9788: [FLINK-14259] Introduce JobManagerRunner interface
zhuzhurk commented on a change in pull request #9788: [FLINK-14259] Introduce
JobManagerRunner interface
URL: https://github.com/apache/flink/pull/9788#discussion_r329311666
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/JobManagerRunnerTest.java
##
@@ -64,7 +64,7 @@
import static org.junit.Assert.fail;
/**
- * Tests for the {@link JobManagerRunner}.
+ * Tests for the {@link JobManagerRunnerImpl}.
*/
public class JobManagerRunnerTest extends TestLogger {
Review comment:
I think it can be renamed to `JobManagerRunnerImplTest`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #9788: [FLINK-14259] Introduce JobManagerRunner interface
zhuzhurk commented on a change in pull request #9788: [FLINK-14259] Introduce
JobManagerRunner interface
URL: https://github.com/apache/flink/pull/9788#discussion_r329311362
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/dispatcher/TestingJobManagerRunnerFactory.java
##
@@ -82,10 +83,10 @@ public JobManagerRunner createJobManagerRunner(
} else {
jobGraphFuture.complete(jobGraph);
- final JobManagerRunner mock =
mock(JobManagerRunner.class);
Review comment:
It can remain to be `JobManagerRunner` here.
But I think it does not matter since PR#9789 will refactor it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #9788: [FLINK-14259] Introduce JobManagerRunner interface
zhuzhurk commented on a change in pull request #9788: [FLINK-14259] Introduce
JobManagerRunner interface
URL: https://github.com/apache/flink/pull/9788#discussion_r329311153
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/JobManagerRunnerFactory.java
##
@@ -23,13 +23,14 @@
import org.apache.flink.runtime.highavailability.HighAvailabilityServices;
import org.apache.flink.runtime.jobgraph.JobGraph;
import org.apache.flink.runtime.jobmaster.JobManagerRunner;
+import org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl;
import org.apache.flink.runtime.jobmaster.JobManagerSharedServices;
import
org.apache.flink.runtime.jobmaster.factories.JobManagerJobMetricGroupFactory;
import org.apache.flink.runtime.rpc.FatalErrorHandler;
import org.apache.flink.runtime.rpc.RpcService;
/**
- * Factory for a {@link JobManagerRunner}.
+ * Factory for a {@link JobManagerRunnerImpl}.
Review comment:
It should remain `JobManagerRunner` here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #9788: [FLINK-14259] Introduce JobManagerRunner interface
zhuzhurk commented on a change in pull request #9788: [FLINK-14259] Introduce
JobManagerRunner interface
URL: https://github.com/apache/flink/pull/9788#discussion_r329311204
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/dispatcher/DispatcherTest.java
##
@@ -662,7 +663,7 @@ public void testBlockingJobManagerRunner() throws
Exception {
}
/**
-* Tests that a failing {@link JobManagerRunner} will be properly
cleaned up.
+* Tests that a failing {@link JobManagerRunnerImpl} will be properly
cleaned up.
Review comment:
`JobManagerRunner` maybe?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #9788: [FLINK-14259] Introduce JobManagerRunner interface
zhuzhurk commented on a change in pull request #9788: [FLINK-14259] Introduce
JobManagerRunner interface
URL: https://github.com/apache/flink/pull/9788#discussion_r329311666
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/JobManagerRunnerTest.java
##
@@ -64,7 +64,7 @@
import static org.junit.Assert.fail;
/**
- * Tests for the {@link JobManagerRunner}.
+ * Tests for the {@link JobManagerRunnerImpl}.
*/
public class JobManagerRunnerTest extends TestLogger {
Review comment:
I think it can be renamed it to `JobManagerRunnerImplTest`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #9788: [FLINK-14259] Introduce JobManagerRunner interface
zhuzhurk commented on a change in pull request #9788: [FLINK-14259] Introduce
JobManagerRunner interface
URL: https://github.com/apache/flink/pull/9788#discussion_r329311362
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/dispatcher/TestingJobManagerRunnerFactory.java
##
@@ -82,10 +83,10 @@ public JobManagerRunner createJobManagerRunner(
} else {
jobGraphFuture.complete(jobGraph);
- final JobManagerRunner mock =
mock(JobManagerRunner.class);
Review comment:
It can remain to be `JobManagerRunner` here.
But I think it's fine to since PR#9789 will refactor it later.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #9788: [FLINK-14259] Introduce JobManagerRunner interface
zhuzhurk commented on a change in pull request #9788: [FLINK-14259] Introduce
JobManagerRunner interface
URL: https://github.com/apache/flink/pull/9788#discussion_r329311040
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java
##
@@ -378,14 +379,14 @@ private boolean isPartialResourceConfigured(JobGraph
jobGraph) {
}
private JobManagerRunner startJobManagerRunner(JobManagerRunner
jobManagerRunner) throws Exception {
- final JobID jobId = jobManagerRunner.getJobGraph().getJobID();
+ final JobID jobId = jobManagerRunner.getJobID();
FutureUtils.assertNoException(
jobManagerRunner.getResultFuture().handleAsync(
(ArchivedExecutionGraph archivedExecutionGraph,
Throwable throwable) -> {
// check if we are still the active
JobManagerRunner by checking the identity
- final
CompletableFuture jobManagerRunnerFuture =
jobManagerRunnerFutures.get(jobId);
- final JobManagerRunner
currentJobManagerRunner = jobManagerRunnerFuture != null ?
jobManagerRunnerFuture.getNow(null) : null;
+ final
Optional> optionalJobManagerRunner =
Optional.ofNullable(jobManagerRunnerFutures.get(jobId));
+ final JobManagerRunner
currentJobManagerRunner = optionalJobManagerRunner.map(future ->
future.getNow(null)).orElse(null);
Review comment:
How about merging above 2 lines in one statement as they look on one chain
already?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
