[GitHub] [flink] flinkbot commented on pull request #12215: [FLINK-17565][k8s] Bump fabric8 version from 4.5.2 to 4.9.1
flinkbot commented on pull request #12215: URL: https://github.com/apache/flink/pull/12215#issuecomment-630034151 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 5f357acabcb13d64d8e9a042af14329415db0f87 (Mon May 18 08:39:12 UTC 2020) **Warnings:** * **1 pom.xml files were touched**: Check for build and licensing issues. * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-17788) scala shell in yarn mode is broken
[ https://issues.apache.org/jira/browse/FLINK-17788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-17788: --- Labels: pull-request-available (was: ) > scala shell in yarn mode is broken > -- > > Key: FLINK-17788 > URL: https://issues.apache.org/jira/browse/FLINK-17788 > Project: Flink > Issue Type: Bug > Components: Scala Shell >Affects Versions: 1.11.0 >Reporter: Jeff Zhang >Priority: Major > Labels: pull-request-available > > When I start scala shell in yarn mode, one yarn app will be launched, and > after I write some flink code and trigger a flink job, another yarn app will > be launched but would failed to launch due to some conflicts. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] zjffdu opened a new pull request #12216: [FLINK-17788][scala-shell] scala shell in yarn mode is broken
zjffdu opened a new pull request #12216:
URL: https://github.com/apache/flink/pull/12216
## What is the purpose of the change
The root cause of scala shell broken in yarn mode is that target is set as
`yarn-per-job` which is not correct. This PR fix this issue by removing
`target` after deploying yarn session cluster.
## Verifying this change
Verify it manually, start scala shell and run the following sample code.
```
val dataStream = senv.fromElements(1, 2, 3, 4)
dataStream.countWindowAll(2).sum(0).print()
senv.execute("My streaming program")
```
Before this PR, this piece of code will start another yarn app and fail to
run this flink job.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (no)
- The serializers: (no )
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable )
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] guoweiM commented on pull request #12132: [FLINK-17593][Connectors/FileSystem] Support arbitrary recovery mechanism for PartFileWriter
guoweiM commented on pull request #12132: URL: https://github.com/apache/flink/pull/12132#issuecomment-630035754 About the manual file clean up, we could move the template file to a temp folder. And use a special `RecovrableWriter` that could know the temp folder. WDYT? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] aljoscha commented on pull request #12132: [FLINK-17593][Connectors/FileSystem] Support arbitrary recovery mechanism for PartFileWriter
aljoscha commented on pull request #12132: URL: https://github.com/apache/flink/pull/12132#issuecomment-630037118 That sounds good, but it's also ok as is and could be changed later because it doesn't change the snapshot files. It's fine of course if you want to change it now before merging. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12216: [FLINK-17788][scala-shell] scala shell in yarn mode is broken
flinkbot commented on pull request #12216: URL: https://github.com/apache/flink/pull/12216#issuecomment-630037399 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 9b24ab2448e35acd0fabcf226409d37162136391 (Mon May 18 08:45:14 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-17788).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lirui-apache opened a new pull request #12217: [FLINK-17786][sql-client] Remove dialect from ExecutionEntry
lirui-apache opened a new pull request #12217: URL: https://github.com/apache/flink/pull/12217 ## What is the purpose of the change Since SQL dialect is already a table config option, we don't need to support it in `ExecutionEntry`. ## Brief change log - Remove the dialect config from `ExecutionEntry`. - Refactor test to cover switching dialect in a session. ## Verifying this change Updated test case ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? NA This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-17786) Cannot switch dialect in SQL CLI
[ https://issues.apache.org/jira/browse/FLINK-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-17786: --- Labels: pull-request-available (was: ) > Cannot switch dialect in SQL CLI > > > Key: FLINK-17786 > URL: https://issues.apache.org/jira/browse/FLINK-17786 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Reporter: Rui Li >Assignee: Rui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] lirui-apache commented on pull request #12217: [FLINK-17786][sql-client] Remove dialect from ExecutionEntry
lirui-apache commented on pull request #12217: URL: https://github.com/apache/flink/pull/12217#issuecomment-630038276 @JingsongLi Please take a look. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-17794) Tear down installed software in reverse order in Jepsen Tests
Gary Yao created FLINK-17794:
Summary: Tear down installed software in reverse order in Jepsen
Tests
Key: FLINK-17794
URL: https://issues.apache.org/jira/browse/FLINK-17794
Project: Flink
Issue Type: Bug
Components: Tests
Affects Versions: 1.10.1, 1.11.0
Reporter: Gary Yao
Assignee: Gary Yao
Fix For: 1.11.0
Tear down installed software in reverse order in Jepsen Tests. This mitigates
the issue that sometimes hadoop's node manager directories cannot be removed
using {{rm -rf}} because Flink processes keep running and generate files after
the YARN NodeManager is shut down. {{rm -r}} removes files recursively but if
files are created in the background concurrently, the command can still fail
with a non-zero exit code.
{noformat}
sh -c \"cd /; rm -rf /opt/hadoop\"", :exit 1, :out "", :err "rm: cannot remove
'/opt/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1587567275082_0001/flink-io-3587fdbb-15be-4482-94f2-338bfe6b1acc/job_77be6dd9f1b2aa218348e8b8a2512660_op_StreamMap_5271c210329e73bd743f3227edfb3b71__27_30__uuid_02dbbf1e-d2d5-43e8-ab34-040345f96476/db':
Directory not empty\nrm: cannot remove
'/opt/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1587567275082_0001/flink-io-d14f2078-74ee-4b8b-aafe-4299577f214f/job_77be6dd9f1b2aa218348e8b8a2512660_op_StreamMap_7d23c6ceabda05a587f0217e44f21301__17_30__uuid_2de2b67d-0767-4e32-99f0-ddd291460947/db':
Directory not empty
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] docete opened a new pull request #12218: [FLINK-16160][table] Fix proctime()/rowtime() doesn't work for TableE…
docete opened a new pull request #12218: URL: https://github.com/apache/flink/pull/12218 …nvironment.connect().createTemporaryTable() ## What is the purpose of the change Since FLINK-14490 , proctime()/rowtime() doesn't work for TableEnvironment.connect().createTemporaryTable(), The root cause is described on jira ticket. ## Brief change log - instantiate the TableSource in CatalogSchemaTable and check if it's a DefinedRowtimeAttributes/DefinedProctimeAttribute instance. If so, rewrite the TableSchema to patch the time indicator(as it is in ConnectorCatalogTable#calculateSourceSchema) - avoid erasing time indicator in CatalogSourceTable if the TableSource is a DefinedRowtimeAttributes/DefinedProctimeAttribute instance ## Verifying this change This change added tests. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (**yes** / no) - If yes, how is the feature documented? (not applicable / docs / **JavaDocs** / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] docete commented on pull request #12218: [FLINK-16160][table] Fix proctime()/rowtime() doesn't work for TableE…
docete commented on pull request #12218: URL: https://github.com/apache/flink/pull/12218#issuecomment-630039520 @wuchong This PR fix FLINK-16160 on release-1.10, please have a look and merge, Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-17794) Tear down installed software in reverse order in Jepsen Tests
[
https://issues.apache.org/jira/browse/FLINK-17794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Yao updated FLINK-17794:
-
Description:
Tear down installed software in reverse order in Jepsen Tests. This mitigates
the issue that sometimes YARN's NodeManager directories cannot be removed using
{{rm -rf}} because Flink processes keep running and generate files after the
YARN NodeManager is shut down. {{rm -r}} removes files recursively but if files
are created in the background concurrently, the command can still fail with a
non-zero exit code.
{noformat}
sh -c \"cd /; rm -rf /opt/hadoop\"", :exit 1, :out "", :err "rm: cannot remove
'/opt/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1587567275082_0001/flink-io-3587fdbb-15be-4482-94f2-338bfe6b1acc/job_77be6dd9f1b2aa218348e8b8a2512660_op_StreamMap_5271c210329e73bd743f3227edfb3b71__27_30__uuid_02dbbf1e-d2d5-43e8-ab34-040345f96476/db':
Directory not empty\nrm: cannot remove
'/opt/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1587567275082_0001/flink-io-d14f2078-74ee-4b8b-aafe-4299577f214f/job_77be6dd9f1b2aa218348e8b8a2512660_op_StreamMap_7d23c6ceabda05a587f0217e44f21301__17_30__uuid_2de2b67d-0767-4e32-99f0-ddd291460947/db':
Directory not empty
{noformat}
was:
Tear down installed software in reverse order in Jepsen Tests. This mitigates
the issue that sometimes hadoop's node manager directories cannot be removed
using {{rm -rf}} because Flink processes keep running and generate files after
the YARN NodeManager is shut down. {{rm -r}} removes files recursively but if
files are created in the background concurrently, the command can still fail
with a non-zero exit code.
{noformat}
sh -c \"cd /; rm -rf /opt/hadoop\"", :exit 1, :out "", :err "rm: cannot remove
'/opt/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1587567275082_0001/flink-io-3587fdbb-15be-4482-94f2-338bfe6b1acc/job_77be6dd9f1b2aa218348e8b8a2512660_op_StreamMap_5271c210329e73bd743f3227edfb3b71__27_30__uuid_02dbbf1e-d2d5-43e8-ab34-040345f96476/db':
Directory not empty\nrm: cannot remove
'/opt/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1587567275082_0001/flink-io-d14f2078-74ee-4b8b-aafe-4299577f214f/job_77be6dd9f1b2aa218348e8b8a2512660_op_StreamMap_7d23c6ceabda05a587f0217e44f21301__17_30__uuid_2de2b67d-0767-4e32-99f0-ddd291460947/db':
Directory not empty
{noformat}
> Tear down installed software in reverse order in Jepsen Tests
> -
>
> Key: FLINK-17794
> URL: https://issues.apache.org/jira/browse/FLINK-17794
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.10.1, 1.11.0
>Reporter: Gary Yao
>Assignee: Gary Yao
>Priority: Major
> Fix For: 1.11.0
>
>
> Tear down installed software in reverse order in Jepsen Tests. This mitigates
> the issue that sometimes YARN's NodeManager directories cannot be removed
> using {{rm -rf}} because Flink processes keep running and generate files
> after the YARN NodeManager is shut down. {{rm -r}} removes files recursively
> but if files are created in the background concurrently, the command can
> still fail with a non-zero exit code.
> {noformat}
> sh -c \"cd /; rm -rf /opt/hadoop\"", :exit 1, :out "", :err "rm: cannot
> remove
> '/opt/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1587567275082_0001/flink-io-3587fdbb-15be-4482-94f2-338bfe6b1acc/job_77be6dd9f1b2aa218348e8b8a2512660_op_StreamMap_5271c210329e73bd743f3227edfb3b71__27_30__uuid_02dbbf1e-d2d5-43e8-ab34-040345f96476/db':
> Directory not empty\nrm: cannot remove
> '/opt/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1587567275082_0001/flink-io-d14f2078-74ee-4b8b-aafe-4299577f214f/job_77be6dd9f1b2aa218348e8b8a2512660_op_StreamMap_7d23c6ceabda05a587f0217e44f21301__17_30__uuid_2de2b67d-0767-4e32-99f0-ddd291460947/db':
> Directory not empty
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17789) DelegatingConfiguration should remove prefix instead of add prefix in toMap
[
https://issues.apache.org/jira/browse/FLINK-17789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110051#comment-17110051
]
Chesnay Schepler commented on FLINK-17789:
--
Alright hold on, this is getting really confusing.
Let's clarify the behaviors a bit.
For a DelegatingConfiguration with prefix P:
1) get(X) reads P.X from the backing configuration
1) set(X) writes P.X to the backing configuration
2) addAllToProperties() writes all options starting with P into the properties,
such that properties.get(X) works.
- this behavior is required by the metric system, where reporters use
suffixes for specifying options, since the prefix is dynamic (because includes
the reporter name)
3) toMap() writes all options of the backing configuration, with an added
prefix P. map.get(X) will not work, but map.get(P.X) will.
It is not a problem that toMap() behaves differently to addAllToProperties() in
general, as they actually serve different purposes. I can see that the
behavior could be, but currently it is necessary and we would break code if we
were to change anything. Change addAllToProperties() and the metric system is
broken, change toMap() and the configuration of formats is broken.
However, one problem I do have with 3) is the following:
{code}
Configuration conf = new Configuration();
conf.setString("k0", "v0");
conf.setString("prefix.k1", "v1");
DelegatingConfiguration dc = new DelegatingConfiguration(conf, "prefix.");
System.out.println(dc.getString("k1", null)); //v1
System.out.println(dc.getString("k0", null)); //null
Map map = dc.toMap();
System.out.println(map.get("prefix.k1")); //v1
System.out.println(map.get("prefix.k0")); //v0
{code}
Basically, toMap() conjures settings out of thin air, where it should be an
accurate representation of the configuration.
> DelegatingConfiguration should remove prefix instead of add prefix in toMap
> ---
>
> Key: FLINK-17789
> URL: https://issues.apache.org/jira/browse/FLINK-17789
> Project: Flink
> Issue Type: Bug
> Components: API / Core
>Reporter: Jingsong Lee
>Priority: Major
> Fix For: 1.11.0
>
>
> {code:java}
> Configuration conf = new Configuration();
> conf.setString("k0", "v0");
> conf.setString("prefix.k1", "v1");
> DelegatingConfiguration dc = new DelegatingConfiguration(conf, "prefix.");
> System.out.println(dc.getString("k0", "empty")); // empty
> System.out.println(dc.getString("k1", "empty")); // v1
> System.out.println(dc.toMap().get("k1")); // should be v1, but null
> System.out.println(dc.toMap().get("prefix.prefix.k1")); // should be null,
> but v1
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-17794) Tear down installed software in reverse order in Jepsen Tests
[
https://issues.apache.org/jira/browse/FLINK-17794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Yao updated FLINK-17794:
-
Description:
Tear down installed software in reverse order in Jepsen tests. This mitigates
the issue that sometimes YARN's NodeManager directories cannot be removed using
{{rm -rf}} because Flink processes keep running and generate files after the
YARN NodeManager is shut down. {{rm -r}} removes files recursively but if files
are created in the background concurrently, the command can still fail with a
non-zero exit code.
{noformat}
sh -c \"cd /; rm -rf /opt/hadoop\"", :exit 1, :out "", :err "rm: cannot remove
'/opt/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1587567275082_0001/flink-io-3587fdbb-15be-4482-94f2-338bfe6b1acc/job_77be6dd9f1b2aa218348e8b8a2512660_op_StreamMap_5271c210329e73bd743f3227edfb3b71__27_30__uuid_02dbbf1e-d2d5-43e8-ab34-040345f96476/db':
Directory not empty\nrm: cannot remove
'/opt/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1587567275082_0001/flink-io-d14f2078-74ee-4b8b-aafe-4299577f214f/job_77be6dd9f1b2aa218348e8b8a2512660_op_StreamMap_7d23c6ceabda05a587f0217e44f21301__17_30__uuid_2de2b67d-0767-4e32-99f0-ddd291460947/db':
Directory not empty
{noformat}
was:
Tear down installed software in reverse order in Jepsen Tests. This mitigates
the issue that sometimes YARN's NodeManager directories cannot be removed using
{{rm -rf}} because Flink processes keep running and generate files after the
YARN NodeManager is shut down. {{rm -r}} removes files recursively but if files
are created in the background concurrently, the command can still fail with a
non-zero exit code.
{noformat}
sh -c \"cd /; rm -rf /opt/hadoop\"", :exit 1, :out "", :err "rm: cannot remove
'/opt/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1587567275082_0001/flink-io-3587fdbb-15be-4482-94f2-338bfe6b1acc/job_77be6dd9f1b2aa218348e8b8a2512660_op_StreamMap_5271c210329e73bd743f3227edfb3b71__27_30__uuid_02dbbf1e-d2d5-43e8-ab34-040345f96476/db':
Directory not empty\nrm: cannot remove
'/opt/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1587567275082_0001/flink-io-d14f2078-74ee-4b8b-aafe-4299577f214f/job_77be6dd9f1b2aa218348e8b8a2512660_op_StreamMap_7d23c6ceabda05a587f0217e44f21301__17_30__uuid_2de2b67d-0767-4e32-99f0-ddd291460947/db':
Directory not empty
{noformat}
> Tear down installed software in reverse order in Jepsen Tests
> -
>
> Key: FLINK-17794
> URL: https://issues.apache.org/jira/browse/FLINK-17794
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.10.1, 1.11.0
>Reporter: Gary Yao
>Assignee: Gary Yao
>Priority: Major
> Fix For: 1.11.0
>
>
> Tear down installed software in reverse order in Jepsen tests. This mitigates
> the issue that sometimes YARN's NodeManager directories cannot be removed
> using {{rm -rf}} because Flink processes keep running and generate files
> after the YARN NodeManager is shut down. {{rm -r}} removes files recursively
> but if files are created in the background concurrently, the command can
> still fail with a non-zero exit code.
> {noformat}
> sh -c \"cd /; rm -rf /opt/hadoop\"", :exit 1, :out "", :err "rm: cannot
> remove
> '/opt/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1587567275082_0001/flink-io-3587fdbb-15be-4482-94f2-338bfe6b1acc/job_77be6dd9f1b2aa218348e8b8a2512660_op_StreamMap_5271c210329e73bd743f3227edfb3b71__27_30__uuid_02dbbf1e-d2d5-43e8-ab34-040345f96476/db':
> Directory not empty\nrm: cannot remove
> '/opt/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1587567275082_0001/flink-io-d14f2078-74ee-4b8b-aafe-4299577f214f/job_77be6dd9f1b2aa218348e8b8a2512660_op_StreamMap_7d23c6ceabda05a587f0217e44f21301__17_30__uuid_2de2b67d-0767-4e32-99f0-ddd291460947/db':
> Directory not empty
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-17789) DelegatingConfiguration should remove prefix instead of add prefix in toMap
[
https://issues.apache.org/jira/browse/FLINK-17789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110051#comment-17110051
]
Chesnay Schepler edited comment on FLINK-17789 at 5/18/20, 8:51 AM:
Alright hold on, this is getting really confusing.
Let's clarify the behaviors a bit.
For a DelegatingConfiguration with prefix P:
1) get(X) reads P.X from the backing configuration
1) set(X) writes P.X to the backing configuration
2) addAllToProperties() writes all options starting with P into the properties,
such that properties.get(X) works.
3) toMap() writes all options of the backing configuration, with an added
prefix P. map.get(X) will not work, but map.get(P.X) will.
It is not a problem that toMap() behaves differently to addAllToProperties() in
general, as they actually serve different purposes.
I can see that the behavior could be, but currently it is necessary and we
would break code if we were to change anything.
Change addAllToProperties() and the metric system breaks (reporters look up
options by suffix, since the prefix contains the reporter name and is hence
dynamic), change toMap() and the configuration of formats is broken (because
suddenly the options are no longer fully qualified.
However, one problem I do have with 3) is the following:
{code}
Configuration conf = new Configuration();
conf.setString("k0", "v0");
conf.setString("prefix.k1", "v1");
DelegatingConfiguration dc = new DelegatingConfiguration(conf, "prefix.");
System.out.println(dc.getString("k1", null)); //v1
System.out.println(dc.getString("k0", null)); //null
Map map = dc.toMap();
System.out.println(map.get("prefix.k1")); //v1
System.out.println(map.get("prefix.k0")); //v0
{code}
Basically, toMap() conjures settings out of thin air, where it should be an
accurate representation of the configuration.
was (Author: zentol):
Alright hold on, this is getting really confusing.
Let's clarify the behaviors a bit.
For a DelegatingConfiguration with prefix P:
1) get(X) reads P.X from the backing configuration
1) set(X) writes P.X to the backing configuration
2) addAllToProperties() writes all options starting with P into the properties,
such that properties.get(X) works.
- this behavior is required by the metric system, where reporters use
suffixes for specifying options, since the prefix is dynamic (because includes
the reporter name)
3) toMap() writes all options of the backing configuration, with an added
prefix P. map.get(X) will not work, but map.get(P.X) will.
It is not a problem that toMap() behaves differently to addAllToProperties() in
general, as they actually serve different purposes. I can see that the
behavior could be, but currently it is necessary and we would break code if we
were to change anything. Change addAllToProperties() and the metric system is
broken, change toMap() and the configuration of formats is broken.
However, one problem I do have with 3) is the following:
{code}
Configuration conf = new Configuration();
conf.setString("k0", "v0");
conf.setString("prefix.k1", "v1");
DelegatingConfiguration dc = new DelegatingConfiguration(conf, "prefix.");
System.out.println(dc.getString("k1", null)); //v1
System.out.println(dc.getString("k0", null)); //null
Map map = dc.toMap();
System.out.println(map.get("prefix.k1")); //v1
System.out.println(map.get("prefix.k0")); //v0
{code}
Basically, toMap() conjures settings out of thin air, where it should be an
accurate representation of the configuration.
> DelegatingConfiguration should remove prefix instead of add prefix in toMap
> ---
>
> Key: FLINK-17789
> URL: https://issues.apache.org/jira/browse/FLINK-17789
> Project: Flink
> Issue Type: Bug
> Components: API / Core
>Reporter: Jingsong Lee
>Priority: Major
> Fix For: 1.11.0
>
>
> {code:java}
> Configuration conf = new Configuration();
> conf.setString("k0", "v0");
> conf.setString("prefix.k1", "v1");
> DelegatingConfiguration dc = new DelegatingConfiguration(conf, "prefix.");
> System.out.println(dc.getString("k0", "empty")); // empty
> System.out.println(dc.getString("k1", "empty")); // v1
> System.out.println(dc.toMap().get("k1")); // should be v1, but null
> System.out.println(dc.toMap().get("prefix.prefix.k1")); // should be null,
> but v1
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] twalthr commented on a change in pull request #12201: [hotfix] Remove raw class usages in Configuration.
twalthr commented on a change in pull request #12201:
URL: https://github.com/apache/flink/pull/12201#discussion_r426467949
##
File path:
flink-core/src/main/java/org/apache/flink/configuration/Configuration.java
##
@@ -909,12 +909,10 @@ private void loggingFallback(FallbackKey fallbackKey,
ConfigOption configOpti
List listOfRawProperties =
StructuredOptionsSplitter.splitEscaped(o.toString(), ',');
return listOfRawProperties.stream()
.map(s ->
StructuredOptionsSplitter.splitEscaped(s, ':'))
- .map(pair -> {
+ .peek(pair -> {
Review comment:
Thanks for the explanation. +1
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-15792) Make Flink logs accessible via kubectl logs per default
[
https://issues.apache.org/jira/browse/FLINK-15792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110052#comment-17110052
]
Till Rohrmann commented on FLINK-15792:
---
[~fly_in_gis] why do we still need
https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/native_kubernetes.html#log-files?
I think with FLINK-17166 we log by default to stdout and to a file which is
accessible from the web ui.
> Make Flink logs accessible via kubectl logs per default
> ---
>
> Key: FLINK-15792
> URL: https://issues.apache.org/jira/browse/FLINK-15792
> Project: Flink
> Issue Type: Sub-task
> Components: Deployment / Kubernetes
>Affects Versions: 1.10.0
>Reporter: Till Rohrmann
>Priority: Major
> Fix For: 1.11.0, 1.10.2
>
>
> I think we should make Flink's logs accessible via {{kubectl logs}} per
> default. Firstly, this is the idiomatic way to obtain the logs from a
> container on Kubernetes. Secondly, especially if something does not work and
> the container cannot start/stops abruptly, there is no way to log into the
> container and look for the log.file. This makes debugging the setup quite
> hard.
> I think the best way would be to create the Flink Docker image in such a way
> that it logs to stdout. In order to allow access to the log file from the web
> ui, it should also create a log file. One way to achieve this is to add a
> ConsoleAppender to the respective logging configuration. Another way could be
> to start the process in the console mode and then to teeing the stdout output
> into the log file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-17789) DelegatingConfiguration should remove prefix instead of add prefix in toMap
[
https://issues.apache.org/jira/browse/FLINK-17789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110051#comment-17110051
]
Chesnay Schepler edited comment on FLINK-17789 at 5/18/20, 8:52 AM:
Alright hold on, this is getting really confusing.
Let's clarify the behaviors a bit.
For a DelegatingConfiguration with prefix P:
1) get(X) reads P.X from the backing configuration
1) set(X) writes P.X to the backing configuration
2) addAllToProperties() writes all options starting with P into the properties,
such that properties.get(X) works.
3) toMap() writes all options of the backing configuration, with an added
prefix P. map.get(X) will not work, but map.get(P.X) will.
It is not a problem that toMap() behaves differently to addAllToProperties() in
general, as they actually serve very different purposes.
I can see that the behavior can be surprising, but currently it is necessary
and we would break code if we were to change anything.
Change addAllToProperties() and the metric system breaks (reporters look up
options by suffix, since the prefix contains the reporter name and is hence
dynamic), change toMap() and the configuration of formats is broken (because
suddenly the options are no longer fully qualified.
However, one problem I do have with 3) is the following:
{code}
Configuration conf = new Configuration();
conf.setString("k0", "v0");
conf.setString("prefix.k1", "v1");
DelegatingConfiguration dc = new DelegatingConfiguration(conf, "prefix.");
System.out.println(dc.getString("k1", null)); //v1
System.out.println(dc.getString("k0", null)); //null
Map map = dc.toMap();
System.out.println(map.get("prefix.k1")); //v1
System.out.println(map.get("prefix.k0")); //v0
{code}
Basically, toMap() conjures settings out of thin air, where it should be an
accurate representation of the configuration.
was (Author: zentol):
Alright hold on, this is getting really confusing.
Let's clarify the behaviors a bit.
For a DelegatingConfiguration with prefix P:
1) get(X) reads P.X from the backing configuration
1) set(X) writes P.X to the backing configuration
2) addAllToProperties() writes all options starting with P into the properties,
such that properties.get(X) works.
3) toMap() writes all options of the backing configuration, with an added
prefix P. map.get(X) will not work, but map.get(P.X) will.
It is not a problem that toMap() behaves differently to addAllToProperties() in
general, as they actually serve different purposes.
I can see that the behavior could be, but currently it is necessary and we
would break code if we were to change anything.
Change addAllToProperties() and the metric system breaks (reporters look up
options by suffix, since the prefix contains the reporter name and is hence
dynamic), change toMap() and the configuration of formats is broken (because
suddenly the options are no longer fully qualified.
However, one problem I do have with 3) is the following:
{code}
Configuration conf = new Configuration();
conf.setString("k0", "v0");
conf.setString("prefix.k1", "v1");
DelegatingConfiguration dc = new DelegatingConfiguration(conf, "prefix.");
System.out.println(dc.getString("k1", null)); //v1
System.out.println(dc.getString("k0", null)); //null
Map map = dc.toMap();
System.out.println(map.get("prefix.k1")); //v1
System.out.println(map.get("prefix.k0")); //v0
{code}
Basically, toMap() conjures settings out of thin air, where it should be an
accurate representation of the configuration.
> DelegatingConfiguration should remove prefix instead of add prefix in toMap
> ---
>
> Key: FLINK-17789
> URL: https://issues.apache.org/jira/browse/FLINK-17789
> Project: Flink
> Issue Type: Bug
> Components: API / Core
>Reporter: Jingsong Lee
>Priority: Major
> Fix For: 1.11.0
>
>
> {code:java}
> Configuration conf = new Configuration();
> conf.setString("k0", "v0");
> conf.setString("prefix.k1", "v1");
> DelegatingConfiguration dc = new DelegatingConfiguration(conf, "prefix.");
> System.out.println(dc.getString("k0", "empty")); // empty
> System.out.println(dc.getString("k1", "empty")); // v1
> System.out.println(dc.toMap().get("k1")); // should be v1, but null
> System.out.println(dc.toMap().get("prefix.prefix.k1")); // should be null,
> but v1
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #12217: [FLINK-17786][sql-client] Remove dialect from ExecutionEntry
flinkbot commented on pull request #12217: URL: https://github.com/apache/flink/pull/12217#issuecomment-630041279 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit a9407fa236d3cb710f131c04ced0ae449bbc15a9 (Mon May 18 08:52:21 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12218: [FLINK-16160][table] Fix proctime()/rowtime() doesn't work for TableE…
flinkbot commented on pull request #12218: URL: https://github.com/apache/flink/pull/12218#issuecomment-630041237 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit ced394a88ef14eb97b61391aa2fb8b571c6b5977 (Mon May 18 08:52:16 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] KurtYoung merged pull request #12202: [FLINK-17791][table][streaming] Support collecting query results under all execution and network environments
KurtYoung merged pull request #12202: URL: https://github.com/apache/flink/pull/12202 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-17791) Combine collecting sink and iterator to support collecting query results under all execution and network environments
[ https://issues.apache.org/jira/browse/FLINK-17791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young closed FLINK-17791. -- Assignee: Caizhi Weng Resolution: Fixed master: cfeaedd9df677b90c8e31214ca7704eead4e6655 > Combine collecting sink and iterator to support collecting query results > under all execution and network environments > - > > Key: FLINK-17791 > URL: https://issues.apache.org/jira/browse/FLINK-17791 > Project: Flink > Issue Type: Sub-task > Components: API / DataStream, Table SQL / Planner >Reporter: Caizhi Weng >Assignee: Caizhi Weng >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > > After introducing specialized collecting sink and iterator, the last thing we > need to do is to combine them together in Table / DataSteram API so that the > whole collecting mechanism works for the users. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] dawidwys merged pull request #12201: [hotfix] Remove raw class usages in Configuration.
dawidwys merged pull request #12201: URL: https://github.com/apache/flink/pull/12201 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12139: [FLINK-16076] Translate "Queryable State" page into Chinese
flinkbot edited a comment on pull request #12139: URL: https://github.com/apache/flink/pull/12139#issuecomment-628360660 ## CI report: * b02e502fa3bb3ae57d4678d23868cad20d51caca Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1230) * 158aea29d67643ce7c7e140f32c32e4c8fc177be Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1702) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12132: [FLINK-17593][Connectors/FileSystem] Support arbitrary recovery mechanism for PartFileWriter
flinkbot edited a comment on pull request #12132: URL: https://github.com/apache/flink/pull/12132#issuecomment-628151415 ## CI report: * 10ef0c696350fcd84866fde27f19ed2a0312ee4b Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1683) * 79f3bb064a15bfde312932e603ae2a65e67545fd UNKNOWN * ec7e775f83706fe690c3d4214f30c7e81a5167f5 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12103: [FLINK-16998][core] Add a changeflag to Row
flinkbot edited a comment on pull request #12103: URL: https://github.com/apache/flink/pull/12103#issuecomment-627456433 ## CI report: * 460711a0fe014e079ea2eb9c6e98da11e1946b48 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1693) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12204: [FLINK-17777][tests] Set HADOOP_CLASSPATH for Mesos TaskManagers
flinkbot edited a comment on pull request #12204: URL: https://github.com/apache/flink/pull/12204#issuecomment-629847515 ## CI report: * 097ac2b06f10900c435115c24e699c5328ee3227 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1649) * 1b0f95eb45256569484cff22599d080d968841f0 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1704) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12181: [FLINK-17645][runtime] Fix SafetyNetCloseableRegistry constructor bug.
flinkbot edited a comment on pull request #12181: URL: https://github.com/apache/flink/pull/12181#issuecomment-629344595 ## CI report: * b954ba073cba912b98c5992b05caec91e7657871 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1687) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12209: [FLINK-17520] [core] Extend CompositeTypeSerializerSnapshot to support migration
flinkbot edited a comment on pull request #12209: URL: https://github.com/apache/flink/pull/12209#issuecomment-629971018 ## CI report: * c91cdd32db0464d8b60d0efc0763078388a15daf Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1684) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12214: [FLINK-17618][flink-dist] Update the outdated comments in log4j2 configuration files
flinkbot commented on pull request #12214: URL: https://github.com/apache/flink/pull/12214#issuecomment-630047206 ## CI report: * 78d9cd9f70ac69a78f6d045d998a004492ce4cfe UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12197: [FLINK-17357][table-planner-blink] add 'DROP catalog' DDL to blink pl…
flinkbot edited a comment on pull request #12197: URL: https://github.com/apache/flink/pull/12197#issuecomment-629768433 ## CI report: * 15cb31fea013766318c482cd266aa294b9df225b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1603) * 1dd4c872d42e9b26907e350aeacb5f278b2b74c1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1703) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12215: [FLINK-17565][k8s] Bump fabric8 version from 4.5.2 to 4.9.1
flinkbot commented on pull request #12215: URL: https://github.com/apache/flink/pull/12215#issuecomment-630047332 ## CI report: * 5f357acabcb13d64d8e9a042af14329415db0f87 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12217: [FLINK-17786][sql-client] Remove dialect from ExecutionEntry
flinkbot commented on pull request #12217: URL: https://github.com/apache/flink/pull/12217#issuecomment-630047579 ## CI report: * a9407fa236d3cb710f131c04ced0ae449bbc15a9 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12216: [FLINK-17788][scala-shell] scala shell in yarn mode is broken
flinkbot commented on pull request #12216: URL: https://github.com/apache/flink/pull/12216#issuecomment-630047470 ## CI report: * 9b24ab2448e35acd0fabcf226409d37162136391 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] kl0u commented on pull request #12132: [FLINK-17593][Connectors/FileSystem] Support arbitrary recovery mechanism for PartFileWriter
kl0u commented on pull request #12132: URL: https://github.com/apache/flink/pull/12132#issuecomment-630047870 Hi, these are some comments with renamings and moving classes for consistency https://github.com/kl0u/flink/tree/FLINK-17593-comments. Please check them out. Probably we can also merge them later, but if it is possible I think it makes sense to do it before cutting the branch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-16863) Sorting descendingly on the last modified date of LogInfo
[ https://issues.apache.org/jira/browse/FLINK-16863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Yao updated FLINK-16863: - Affects Version/s: 1.11.0 > Sorting descendingly on the last modified date of LogInfo > - > > Key: FLINK-16863 > URL: https://issues.apache.org/jira/browse/FLINK-16863 > Project: Flink > Issue Type: Sub-task > Components: Runtime / REST >Affects Versions: 1.11.0 >Reporter: lining >Priority: Major > > Sorting descendingly on the last modified date could a user be able to see > the most recent files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] TsReaper commented on a change in pull request #12199: [FLINK-17774] [table] supports all kinds of changes for select result
TsReaper commented on a change in pull request #12199:
URL: https://github.com/apache/flink/pull/12199#discussion_r426454791
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/sinks/BatchSelectTableSink.java
##
@@ -43,71 +43,68 @@
import java.util.concurrent.ExecutionException;
/**
- * A {@link SelectTableSink} for batch select job.
+ * A {@link StreamTableSink} for batch select job to collect the result to
local.
*
* NOTES: This is a temporary solution,
* once FLINK-14807 is finished, the implementation should be changed.
*/
-public class BatchSelectTableSink implements StreamTableSink,
SelectTableSink {
- private final TableSchema tableSchema;
+public class BatchSelectTableSink extends SelectTableSinkBase
implements StreamTableSink {
private final String accumulatorName;
- private final TypeSerializer typeSerializer;
- private JobClient jobClient;
+ private final TypeSerializer typeSerializer;
@SuppressWarnings("unchecked")
public BatchSelectTableSink(TableSchema tableSchema) {
- this.tableSchema =
SelectTableSinkSchemaConverter.convertTimeAttributeToRegularTimestamp(
-
SelectTableSinkSchemaConverter.changeDefaultConversionClass(tableSchema));
+ super(tableSchema);
this.accumulatorName = new AbstractID().toString();
- this.typeSerializer = (TypeSerializer)
TypeInfoDataTypeConverter
-
.fromDataTypeToTypeInfo(this.tableSchema.toRowDataType())
- .createSerializer(new ExecutionConfig());
+ this.typeSerializer = rowDataTypeInfo.createSerializer(new
ExecutionConfig());
}
@Override
- public DataType getConsumedDataType() {
- return tableSchema.toRowDataType();
+ public TypeInformation getOutputType() {
+ return rowDataTypeInfo;
}
@Override
- public TableSchema getTableSchema() {
- return tableSchema;
- }
-
- @Override
- public DataStreamSink consumeDataStream(DataStream dataStream) {
+ public DataStreamSink consumeDataStream(DataStream
dataStream) {
return dataStream.writeUsingOutputFormat(
new Utils.CollectHelper<>(accumulatorName,
typeSerializer))
.name("Batch select table sink")
.setParallelism(1);
}
@Override
- public void setJobClient(JobClient jobClient) {
- this.jobClient = Preconditions.checkNotNull(jobClient,
"jobClient should not be null");
- }
+ public SelectResultProvider getSelectResultProvider() {
+ return new SelectResultProvider() {
Review comment:
Move into an inner class. The implementation is too long for an
anonymous class.
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/sinks/StreamSelectTableSink.java
##
@@ -66,30 +66,44 @@ public StreamSelectTableSink(TableSchema tableSchema) {
}
@Override
- public DataType getConsumedDataType() {
- return tableSchema.toRowDataType();
+ public TypeInformation getRecordType() {
+ return rowDataTypeInfo;
}
@Override
- public TableSchema getTableSchema() {
- return tableSchema;
- }
-
- @Override
- public DataStreamSink consumeDataStream(DataStream dataStream) {
+ public DataStreamSink consumeDataStream(DataStream> dataStream) {
return dataStream
.addSink(new
CollectSink<>(iterator.getBindAddress(), iterator.getPort(), typeSerializer))
.name("Streaming select table sink")
.setParallelism(1);
}
@Override
- public void setJobClient(JobClient jobClient) {
- }
+ public SelectResultProvider getSelectResultProvider() {
+ return new SelectResultProvider() {
Review comment:
Also move this to an inner class.
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/sinks/StreamSelectTableSink.java
##
@@ -19,44 +19,44 @@
package org.apache.flink.table.planner.sinks;
import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.typeutils.TupleTypeInfo;
import org.apache.flink.core.execution.JobClient;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.datastream.DataStr
[jira] [Closed] (FLINK-17715) Supports function DDLs in SQL-CLI
[ https://issues.apache.org/jira/browse/FLINK-17715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young closed FLINK-17715. -- Assignee: Danny Chen Resolution: Fixed master: 20c28ac77ecd6b8d11e38ed84c9a5c36317721f3 > Supports function DDLs in SQL-CLI > - > > Key: FLINK-17715 > URL: https://issues.apache.org/jira/browse/FLINK-17715 > Project: Flink > Issue Type: New Feature > Components: Table SQL / Client >Affects Versions: 1.10.0 >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] KurtYoung merged pull request #12171: [FLINK-17715][sql client] Supports function DDLs in SQL-CLI
KurtYoung merged pull request #12171: URL: https://github.com/apache/flink/pull/12171 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-16863) Sorting descendingly on the last modified date of LogInfo
[ https://issues.apache.org/jira/browse/FLINK-16863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Yao updated FLINK-16863: - Parent: (was: FLINK-13987) Issue Type: Improvement (was: Sub-task) > Sorting descendingly on the last modified date of LogInfo > - > > Key: FLINK-16863 > URL: https://issues.apache.org/jira/browse/FLINK-16863 > Project: Flink > Issue Type: Improvement > Components: Runtime / REST >Affects Versions: 1.11.0 >Reporter: lining >Priority: Major > > Sorting descendingly on the last modified date could a user be able to see > the most recent files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] TsReaper commented on a change in pull request #12199: [FLINK-17774] [table] supports all kinds of changes for select result
TsReaper commented on a change in pull request #12199:
URL: https://github.com/apache/flink/pull/12199#discussion_r426477145
##
File path:
flink-table/flink-table-common/src/main/java/org/apache/flink/table/utils/PrintUtils.java
##
@@ -124,17 +134,22 @@ public static void printAsTableauForm(
}
public static String[] rowToString(Row row) {
- return rowToString(row, NULL_COLUMN);
+ return rowToString(row, NULL_COLUMN, false);
}
- public static String[] rowToString(Row row, String nullColumn) {
- final String[] fields = new String[row.getArity()];
+ public static String[] rowToString(Row row, String nullColumn, boolean
printChangeMode) {
+ final int len = printChangeMode ? row.getArity() + 1 :
row.getArity();
+ final String[] fields = new String[len];
+ int index = 0;
+ if (printChangeMode) {
+ fields[index++] = row.getKind().shortString();
+ }
for (int i = 0; i < row.getArity(); i++) {
final Object field = row.getField(i);
if (field == null) {
- fields[i] = nullColumn;
+ fields[index++] = nullColumn;
} else {
- fields[i] =
StringUtils.arrayAwareToString(field);
+ fields[index++] =
StringUtils.arrayAwareToString(field);
Review comment:
Change this to `List fields` and `add` to it. Directly
maintaining the `index` is sort of hard to read.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Closed] (FLINK-13987) add log list and read log by name
[
https://issues.apache.org/jira/browse/FLINK-13987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Yao closed FLINK-13987.
Resolution: Fixed
> add log list and read log by name
> -
>
> Key: FLINK-13987
> URL: https://issues.apache.org/jira/browse/FLINK-13987
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / REST
>Reporter: lining
>Assignee: lining
>Priority: Major
> Fix For: 1.11.0
>
>
> As the job running, the log files are becoming large.
> As the application runs on JVM, sometimes the user needs to see the log of
> GC, but there isn't this content.
> Above all, we need new apis:
> * list taskmanager all log file
> ** /taskmanagers/taskmanagerid/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "taskmanager.log",
> "size": 12529
> }
> ]
> } {code}
> * read taskmanager log file
> ** /taskmanagers/logs/[filename]
> ** response: same as taskmanager’s log
> * list jobmanager all log file
> ** /jobmanager/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "jobmanager.log",
> "size": 12529
> }
> ]
> }{code}
> * read jobmanager log file
> ** /jobmanager/logs/[filename]
> ** response: same as jobmanager's log
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13987) add log list and read log by name
[
https://issues.apache.org/jira/browse/FLINK-13987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Yao updated FLINK-13987:
-
Fix Version/s: 1.11.0
> add log list and read log by name
> -
>
> Key: FLINK-13987
> URL: https://issues.apache.org/jira/browse/FLINK-13987
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / REST
>Reporter: lining
>Assignee: lining
>Priority: Major
> Fix For: 1.11.0
>
>
> As the job running, the log files are becoming large.
> As the application runs on JVM, sometimes the user needs to see the log of
> GC, but there isn't this content.
> Above all, we need new apis:
> * list taskmanager all log file
> ** /taskmanagers/taskmanagerid/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "taskmanager.log",
> "size": 12529
> }
> ]
> } {code}
> * read taskmanager log file
> ** /taskmanagers/logs/[filename]
> ** response: same as taskmanager’s log
> * list jobmanager all log file
> ** /jobmanager/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "jobmanager.log",
> "size": 12529
> }
> ]
> }{code}
> * read jobmanager log file
> ** /jobmanager/logs/[filename]
> ** response: same as jobmanager's log
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] leonardBang commented on a change in pull request #12212: [FLINK-17626][fs-connector] Fs connector should use FLIP-122 format options style
leonardBang commented on a change in pull request #12212:
URL: https://github.com/apache/flink/pull/12212#discussion_r426454025
##
File path:
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/filesystem/FileSystemOptions.java
##
@@ -29,6 +29,35 @@
*/
public class FileSystemOptions {
+ public static final ConfigOption PATH = key("path")
+ .stringType()
+ .noDefaultValue()
+ .withDescription("The path of a directory");
+
+ public static final ConfigOption PARTITION_DEFAULT_NAME =
key("partition.default-name")
+ .stringType()
+ .defaultValue("__DEFAULT_PARTITION__")
+ .withDescription("The default partition name in case
the dynamic partition" +
+ " column value is null/empty string");
+
+ public static final ConfigOption SINK_ROLLING_POLICY_FILE_SIZE =
key("sink.rolling-policy.file-size")
+ .longType()
+ .defaultValue(1024L * 1024L * 128L)
+ .withDescription("The maximum part file size before
rolling (by default 128MB).");
+
+ public static final ConfigOption
SINK_ROLLING_POLICY_TIME_INTERVAL = key("sink.rolling-policy.time.interval")
Review comment:
```suggestion
public static final ConfigOption
SINK_ROLLING_POLICY_TIME_INTERVAL = key("sink.rolling-policy.time-interval")
```
how about rename to `sink.rolling-policy.time-interval` which is closer to
FLIP-122's style ?
##
File path:
flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/ParquetFileSystemFormatFactory.java
##
@@ -23,137 +23,104 @@
import org.apache.flink.api.common.serialization.BulkWriter;
import org.apache.flink.api.common.serialization.Encoder;
import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
import org.apache.flink.core.fs.FileInputSplit;
import org.apache.flink.core.fs.Path;
import org.apache.flink.formats.parquet.row.ParquetRowDataBuilder;
import org.apache.flink.formats.parquet.utils.SerializableConfiguration;
import org.apache.flink.formats.parquet.vector.ParquetColumnarRowSplitReader;
import org.apache.flink.formats.parquet.vector.ParquetSplitReaderUtil;
import org.apache.flink.table.data.RowData;
-import org.apache.flink.table.descriptors.DescriptorProperties;
import org.apache.flink.table.factories.FileSystemFormatFactory;
import org.apache.flink.table.types.DataType;
import org.apache.flink.table.types.logical.LogicalType;
import org.apache.flink.table.types.logical.RowType;
import org.apache.flink.table.utils.PartitionPathUtils;
import org.apache.hadoop.conf.Configuration;
-import org.apache.parquet.hadoop.ParquetOutputFormat;
import java.io.IOException;
import java.util.Arrays;
-import java.util.HashMap;
+import java.util.HashSet;
import java.util.LinkedHashMap;
import java.util.List;
-import java.util.Map;
import java.util.Optional;
+import java.util.Properties;
+import java.util.Set;
import static org.apache.flink.configuration.ConfigOptions.key;
import static
org.apache.flink.table.data.vector.VectorizedColumnBatch.DEFAULT_SIZE;
-import static
org.apache.flink.table.descriptors.FormatDescriptorValidator.FORMAT;
import static
org.apache.flink.table.filesystem.RowPartitionComputer.restorePartValueFromType;
/**
* Parquet {@link FileSystemFormatFactory} for file system.
*/
public class ParquetFileSystemFormatFactory implements FileSystemFormatFactory
{
- public static final ConfigOption UTC_TIMEZONE =
key("format.utc-timezone")
+ public static final String IDENTIFIER = "parquet";
+
+ public static final ConfigOption UTC_TIMEZONE =
key("utc-timezone")
.booleanType()
.defaultValue(false)
.withDescription("Use UTC timezone or local timezone to
the conversion between epoch" +
" time and LocalDateTime. Hive
0.x/1.x/2.x use local timezone. But Hive 3.x" +
" use UTC timezone");
- /**
-* Prefix for parquet-related properties, besides format, start with
"parquet".
-* See more in {@link ParquetOutputFormat}.
-* - parquet.compression
-* - parquet.block.size
-* - parquet.page.size
-* - parquet.dictionary.page.size
-* - parquet.writer.max-padding
-* - parquet.enable.dictionary
-* - parquet.validation
-* - parquet.writer.version
-* ...
-*/
- public static final String PARQUET_PROPERTIES = "format.parquet";
-
@Override
- public Map requiredContext() {
- Map context = new HashMap<>();
- context.put(FORMAT, "parquet");
- return context;
+
[jira] [Resolved] (FLINK-14807) Add Table#collect api for fetching data to client
[ https://issues.apache.org/jira/browse/FLINK-14807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Caizhi Weng resolved FLINK-14807. - Resolution: Implemented > Add Table#collect api for fetching data to client > - > > Key: FLINK-14807 > URL: https://issues.apache.org/jira/browse/FLINK-14807 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API >Affects Versions: 1.9.1 >Reporter: Jeff Zhang >Assignee: Caizhi Weng >Priority: Major > Labels: pull-request-available, usability > Fix For: 1.11.0 > > Attachments: table-collect-draft.patch, table-collect.png > > > Currently, it is very unconvinient for user to fetch data of flink job unless > specify sink expclitly and then fetch data from this sink via its api (e.g. > write to hdfs sink, then read data from hdfs). However, most of time user > just want to get the data and do whatever processing he want. So it is very > necessary for flink to provide api Table#collect for this purpose. > > Other apis such as Table#head, Table#print is also helpful. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14807) Add TableResult#collect api for fetching data to client
[ https://issues.apache.org/jira/browse/FLINK-14807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Caizhi Weng updated FLINK-14807: Summary: Add TableResult#collect api for fetching data to client (was: Add Table#collect api for fetching data to client) > Add TableResult#collect api for fetching data to client > --- > > Key: FLINK-14807 > URL: https://issues.apache.org/jira/browse/FLINK-14807 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API >Affects Versions: 1.9.1 >Reporter: Jeff Zhang >Assignee: Caizhi Weng >Priority: Major > Labels: pull-request-available, usability > Fix For: 1.11.0 > > Attachments: table-collect-draft.patch, table-collect.png > > > Currently, it is very unconvinient for user to fetch data of flink job unless > specify sink expclitly and then fetch data from this sink via its api (e.g. > write to hdfs sink, then read data from hdfs). However, most of time user > just want to get the data and do whatever processing he want. So it is very > necessary for flink to provide api Table#collect for this purpose. > > Other apis such as Table#head, Table#print is also helpful. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13987) Better TM/JM Log Display
[
https://issues.apache.org/jira/browse/FLINK-13987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Yao updated FLINK-13987:
-
Summary: Better TM/JM Log Display (was: add log list and read log by name)
> Better TM/JM Log Display
>
>
> Key: FLINK-13987
> URL: https://issues.apache.org/jira/browse/FLINK-13987
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / REST
>Reporter: lining
>Assignee: lining
>Priority: Major
> Fix For: 1.11.0
>
>
> As the job running, the log files are becoming large.
> As the application runs on JVM, sometimes the user needs to see the log of
> GC, but there isn't this content.
> Above all, we need new apis:
> * list taskmanager all log file
> ** /taskmanagers/taskmanagerid/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "taskmanager.log",
> "size": 12529
> }
> ]
> } {code}
> * read taskmanager log file
> ** /taskmanagers/logs/[filename]
> ** response: same as taskmanager’s log
> * list jobmanager all log file
> ** /jobmanager/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "jobmanager.log",
> "size": 12529
> }
> ]
> }{code}
> * read jobmanager log file
> ** /jobmanager/logs/[filename]
> ** response: same as jobmanager's log
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] curcur opened a new pull request #12219: [FLINK-15670][API] Provide a Kafka Source/Sink pair as KafkaShuffle
curcur opened a new pull request #12219: URL: https://github.com/apache/flink/pull/12219 ## What is the purpose of the change This pull request is to provide a Kafka Source/Sink pair to simulate a KafkaShuffle that can: 1. Read topics that are already partitioned by key and process them without partitioning them again (avoid shuffles); and 2. Use this to decompose the job into smaller jobs and independent pipelined regions that failover independently. ## Brief Change Log - Extend Datastream API to allow user-defined SinkFunction to manipulate watermark - Kafka Shuffle Producer change - Write timestamp and watermark information together with a record in Kafka - Use `KeyGroupRangeAssignment` to assign records to different Kafka partitions - Kafka Shuffle Consumer change, Kafka Fechter Change - Each consumer read partitions equal to the key group indices that it got assigned. For now, the number of partitions is enforced to equal to the number of consumer parallelism. - KafkaShuffleFechter encapsulates the logic of record emitting and watermark emitting. ## Verifying this change This change added tests: 1. End-2-end tests 2. partition assignment tests Some other tests are missing: 1. Specific watermark tests 2. failover tests ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes) - The serializers: (don't know) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (JavaDocs) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-17793) Replace TableSchema with dedicated CatalogSchema
[
https://issues.apache.org/jira/browse/FLINK-17793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110068#comment-17110068
]
Kurt Young commented on FLINK-17793:
+1, how about calling it `CatalogTableSchema` or something else.
`CatalogSchema` seems like describing schema of the catalog, which sounds like
has nothing to do with a table.
> Replace TableSchema with dedicated CatalogSchema
>
>
> Key: FLINK-17793
> URL: https://issues.apache.org/jira/browse/FLINK-17793
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Timo Walther
>Assignee: Timo Walther
>Priority: Major
>
> The {{TableSchema}} is used for representing the schema of catalog table and
> the schema of a {{Table}} object and operation. We should split those
> responsibilities both for a cleaner API and long-term separation of concerns.
> Connectors should work on a CatalogSchema instead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16303) Add JobMananger Log List and enable reading Logs by Filename
[
https://issues.apache.org/jira/browse/FLINK-16303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Yao updated FLINK-16303:
-
Summary: Add JobMananger Log List and enable reading Logs by Filename
(was: add log list and read log by name for jobmanager)
> Add JobMananger Log List and enable reading Logs by Filename
>
>
> Key: FLINK-16303
> URL: https://issues.apache.org/jira/browse/FLINK-16303
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / REST
>Reporter: lining
>Assignee: lining
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> * list jobmanager all log file
> ** /jobmanager/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "jobmanager.log",
> "size": 12529
> }
> ]
> }{code}
> * read jobmanager log file
> ** /jobmanager/log/[filename]
> ** response: same as jobmanager's log
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-17781) OperatorCoordinator Context must support calls from thread other than JobMaster Main Thread
[ https://issues.apache.org/jira/browse/FLINK-17781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-17781: --- Component/s: Runtime / Coordination > OperatorCoordinator Context must support calls from thread other than > JobMaster Main Thread > --- > > Key: FLINK-17781 > URL: https://issues.apache.org/jira/browse/FLINK-17781 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Stephan Ewen >Assignee: Stephan Ewen >Priority: Blocker > Fix For: 1.11.0 > > > Currently, calls on the Context in the OperatorCoordinator go directly > synchronously to the ExcutionGraph. > There are two critical problems are: > - It is common that the code in the OperatorCoordinator runs in a separate > thread (for example, because it executes blocking operations). Calling the > scheduler from another thread causes the Scheduler to crash (Assertion Error, > violation of single threaded property) > - Calls on the ExecutionGraph are removed as part of removing the legacy > scheduler. Certain calls do not work any more. > +Problem Level 1:+ > The solution would be to pass in the scheduler and a main thread executor to > interact with it. > However, to do that the scheduler needs to be created before the > OperatorCoordinators are created. One could do that by creating the > Coordinators lazily after the Scheduler. > +Problem Level 2:+ > The Scheduler restores the savepoints as part of the scheduler creation, when > the ExecutionGraph and the CheckpointCoordinator are created early in the > constructor. > (Side note: That design is tricky in itself, because it means state is > restored before the scheduler is even properly constructed.) > That means the OperatorCoordinator needs to exist (or an in placeholder > component needs to exist) to accept the restored state. > That brings us to a cyclic dependency: > - OperatorCoordinator (context) needs Scheduler and MainThreadExecutor > - Scheduler and MainThreadExecutor need constructed ExecutionGraph > - ExecutionGraph needs CheckpointCoordinator > - CheckpointCoordinator needs OperatorCoordinator > +Breaking the Cycle+ > The only way we can do this is with a form of lazy initialization: > - We eagerly create the OperatorCoordinators so they exist for state restore > - We provide an uninitialized context to them > - When the Scheduler is started (after leadership is granted) we initialize > the context with the (then readily constructed) Scheduler and > MainThreadExecutor > +Longer-term Solution+ > The longer term solution would require a major change in the Scheduler and > CheckpointCoordinator setup. Something like this: > - Scheduler (and ExecutionGraph) are constructed first > - JobMaster waits for leadership > - Upon leader grant, Operator Coordinators are constructed and can > reference the Scheduler and FencedMainThreadExecutor > - CheckpointCoordinator is constructed and references ExecutionGraph and > OperatorCoordinators > - Savepoint or latest checkpoint is restored > The implementation of the current should try to couple parts as loosely as > possible to make it easy to implement the above approach later. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] liuyongvs removed a comment on pull request #12144: [FLINK-17384][flink-dist] support read hbase conf dir from flink.conf.
liuyongvs removed a comment on pull request #12144: URL: https://github.com/apache/flink/pull/12144#issuecomment-629091637 > ## CI report: > * [5addfcf](https://github.com/apache/flink/commit/5addfcff9eb59b5153e76627ed74e7255fe8b69b) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1292) > > Bot commands > The @flinkbot bot supports the following commands: > * `@flinkbot run travis` re-run the last Travis build > * `@flinkbot run azure` re-run the last Azure build @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JingsongLi opened a new pull request #12220: [FLINK-14255][hive] Integrate mapred record writer to hive streaming sink
JingsongLi opened a new pull request #12220: URL: https://github.com/apache/flink/pull/12220 ## What is the purpose of the change Integrate mapred record writer to hive streaming sink. Hadoop Record writer can support all hive/hadoop formats. ## Brief change log - Introduce `table.exec.hive.fallback-mapred-writer` config, flink native writer better performance, but mapred writer better compatible. - Introduce HiveBulkWriterFactory - Integrate record writer ## Verifying this change `HiveTableSinkTest` ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? JavaDocs This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-17049) Support implicit conversions of rows
[
https://issues.apache.org/jira/browse/FLINK-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-17049:
---
Component/s: Table SQL / API
> Support implicit conversions of rows
>
>
> Key: FLINK-17049
> URL: https://issues.apache.org/jira/browse/FLINK-17049
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Dawid Wysakowicz
>Priority: Major
>
> {code}
> @Test
> public void testCastRowTableFunction() throws Exception {
> final List sourceData = Arrays.asList(
> Row.of(Row.of(1))
> );
> final List sinkData = Arrays.asList(
> Row.of(Row.of(new BigDecimal("1")))
> );
> TestCollectionTableFactory.reset();
> TestCollectionTableFactory.initData(sourceData);
> tEnv().sqlUpdate("CREATE TABLE SourceTable(s Row(a INT)) WITH
> ('connector' = 'COLLECTION')");
> tEnv().sqlUpdate("CREATE TABLE SinkTable(s ROW(a DECIMAL(10,
> 2))) WITH ('connector' = 'COLLECTION')");
> tEnv().createTemporarySystemFunction("func",
> RowCastScalarFunction.class);
> tEnv().sqlUpdate("INSERT INTO SinkTable SELECT func(s) FROM
> SourceTable");
> tEnv().execute("Test Job");
> assertThat(TestCollectionTableFactory.getResult(),
> equalTo(sinkData));
> }
> public static class RowCastScalarFunction extends ScalarFunction {
> public @DataTypeHint("ROW") Row eval(
> @DataTypeHint("ROW") Row row) {
> return Row.of(1);
> }
> }
> {code}
> fails with:
> {code}
> java.lang.AssertionError: use createStructType() instead
> at
> org.apache.calcite.sql.type.SqlTypeFactoryImpl.assertBasic(SqlTypeFactoryImpl.java:225)
> at
> org.apache.calcite.sql.type.SqlTypeFactoryImpl.createSqlType(SqlTypeFactoryImpl.java:48)
> at
> org.apache.flink.table.planner.calcite.FlinkTypeFactory.createSqlType(FlinkTypeFactory.scala:275)
> at
> org.apache.calcite.sql.SqlBasicTypeNameSpec.deriveType(SqlBasicTypeNameSpec.java:205)
> at
> org.apache.calcite.sql.SqlDataTypeSpec.deriveType(SqlDataTypeSpec.java:222)
> at
> org.apache.calcite.sql.SqlDataTypeSpec.deriveType(SqlDataTypeSpec.java:209)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:5960)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:5845)
> at
> org.apache.calcite.sql.SqlDataTypeSpec.accept(SqlDataTypeSpec.java:186)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1800)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1785)
> at org.apache.calcite.sql.SqlNode.validateExpr(SqlNode.java:260)
> at org.apache.calcite.sql.SqlOperator.validateCall(SqlOperator.java:423)
> at org.apache.calcite.sql.SqlFunction.validateCall(SqlFunction.java:199)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateCall(SqlValidatorImpl.java:5552)
> at org.apache.calcite.sql.SqlCall.validate(SqlCall.java:116)
> at org.apache.calcite.sql.SqlNode.validateExpr(SqlNode.java:259)
> at org.apache.calcite.sql.SqlOperator.validateCall(SqlOperator.java:423)
> at org.apache.calcite.sql.SqlFunction.validateCall(SqlFunction.java:199)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateCall(SqlValidatorImpl.java:5552)
> at org.apache.calcite.sql.SqlCall.validate(SqlCall.java:116)
> at org.apache.calcite.sql.SqlNode.validateExpr(SqlNode.java:259)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateExpr(SqlValidatorImpl.java:4306)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelectList(SqlValidatorImpl.java:4284)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3523)
> at
> org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:1110)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:1084)
> at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:232)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:1059)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:766)
>
[jira] [Updated] (FLINK-16303) Add REST Log List and enable reading Logs by Filename
[
https://issues.apache.org/jira/browse/FLINK-16303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Yao updated FLINK-16303:
-
Summary: Add REST Log List and enable reading Logs by Filename (was: Add
Rest Handler to list JM Log Files and enable reading Logs by Filename)
> Add REST Log List and enable reading Logs by Filename
> -
>
> Key: FLINK-16303
> URL: https://issues.apache.org/jira/browse/FLINK-16303
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / REST
>Reporter: lining
>Assignee: lining
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> * list jobmanager all log file
> ** /jobmanager/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "jobmanager.log",
> "size": 12529
> }
> ]
> }{code}
> * read jobmanager log file
> ** /jobmanager/log/[filename]
> ** response: same as jobmanager's log
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16303) Add Rest Handler to list JM Log Files and enable reading Logs by Filename
[
https://issues.apache.org/jira/browse/FLINK-16303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Yao updated FLINK-16303:
-
Summary: Add Rest Handler to list JM Log Files and enable reading Logs by
Filename (was: Add JobMananger Log List and enable reading Logs by Filename)
> Add Rest Handler to list JM Log Files and enable reading Logs by Filename
> -
>
> Key: FLINK-16303
> URL: https://issues.apache.org/jira/browse/FLINK-16303
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / REST
>Reporter: lining
>Assignee: lining
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> * list jobmanager all log file
> ** /jobmanager/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "jobmanager.log",
> "size": 12529
> }
> ]
> }{code}
> * read jobmanager log file
> ** /jobmanager/log/[filename]
> ** response: same as jobmanager's log
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16303) Add Rest Handler to list JM Logfiles and enable reading Logs by Filename
[
https://issues.apache.org/jira/browse/FLINK-16303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Yao updated FLINK-16303:
-
Summary: Add Rest Handler to list JM Logfiles and enable reading Logs by
Filename (was: Add REST Log List and enable reading Logs by Filename)
> Add Rest Handler to list JM Logfiles and enable reading Logs by Filename
>
>
> Key: FLINK-16303
> URL: https://issues.apache.org/jira/browse/FLINK-16303
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / REST
>Reporter: lining
>Assignee: lining
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> * list jobmanager all log file
> ** /jobmanager/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "jobmanager.log",
> "size": 12529
> }
> ]
> }{code}
> * read jobmanager log file
> ** /jobmanager/log/[filename]
> ** response: same as jobmanager's log
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] curcur commented on pull request #12219: [FLINK-15670][API] Provide a Kafka Source/Sink pair as KafkaShuffle
curcur commented on pull request #12219: URL: https://github.com/apache/flink/pull/12219#issuecomment-630056221 Please refer to PR https://github.com/apache/flink/pull/11725 for details. The PR is reverted because of conflicts with [FLINK-17659] Rework WatermarkStrategy, add Suppliers for TimestampAssigner/WatermarkGenerator committed last night. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-17641) How to secure flink applications on yarn on multi-tenant environment
[ https://issues.apache.org/jira/browse/FLINK-17641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-17641: --- Component/s: Deployment / YARN > How to secure flink applications on yarn on multi-tenant environment > > > Key: FLINK-17641 > URL: https://issues.apache.org/jira/browse/FLINK-17641 > Project: Flink > Issue Type: Wish > Components: Deployment / YARN >Reporter: Ethan Li >Priority: Major > > This is a question I wish to get some insights on. > We are trying to support and secure flink on shared yarn cluster. Besides the > security provided by yarn side (queueACL, kerberos), what I noticed is that > flink CLI can still interact with the flink job as long as it knows the > jobmanager rpc port/hostname and rest.port, which can be obtained easily with > yarn command. > Also on the UI side, on yarn cluster, users can visit flink job UI via yarn > proxy using browser. As long as the user can authenticate and view yarn > resourcemanager webpage, he/she can visit the flink UI without any problem. > This basically means Flink UI is wide-open to corp internal users. > On the internal connection side, I am aware of the support added in 1.10 to > limit the mTLS connection by configuring > security.ssl.internal.cert.fingerprint > (https://ci.apache.org/projects/flink/flink-docs-stable/ops/security-ssl.html) > This works but it is not very flexible. Users need to update the config if > the cert changes before they submit a new job. > I asked the similar question on the mailing list before. I am really > interested in how other folks deal with this issue. Thanks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-17556) FATAL: Thread 'flink-metrics-akka.remote.default-remote-dispatcher-3' produced an uncaught exception. Stopping the process... java.lang.OutOfMemoryError: Direct buffer m
[ https://issues.apache.org/jira/browse/FLINK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-17556: --- Component/s: Runtime / Coordination > FATAL: Thread 'flink-metrics-akka.remote.default-remote-dispatcher-3' > produced an uncaught exception. Stopping the process... > java.lang.OutOfMemoryError: Direct buffer memory > -- > > Key: FLINK-17556 > URL: https://issues.apache.org/jira/browse/FLINK-17556 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Reporter: Tammy zhang >Priority: Major > > My job consumes the data in kafka and then processes the data. After the job > lasts for a while, the following error appears: > ERROR org.apache.flink.runtime.util.FatalExitExceptionHandler - FATAL: Thread > 'flink-metrics-akka.remote.default-remote-dispatcher-3' produced an uncaught > exception. Stopping the process... > java.lang.OutOfMemoryError: Direct buffer memory > i have set the "max.poll.records" propertity is "250", and it does not work. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] curcur edited a comment on pull request #12219: [FLINK-15670][API] Provide a Kafka Source/Sink pair as KafkaShuffle
curcur edited a comment on pull request #12219: URL: https://github.com/apache/flink/pull/12219#issuecomment-630056221 Please refer to PR https://github.com/apache/flink/pull/11725 for revewing details. The PR is reverted because of conflicts with [FLINK-17659] Rework WatermarkStrategy, add Suppliers for TimestampAssigner/WatermarkGenerator committed last night. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-17557) Add configuration to disallow ambiguous file schemes
[ https://issues.apache.org/jira/browse/FLINK-17557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-17557: --- Component/s: FileSystems > Add configuration to disallow ambiguous file schemes > > > Key: FLINK-17557 > URL: https://issues.apache.org/jira/browse/FLINK-17557 > Project: Flink > Issue Type: Improvement > Components: FileSystems >Reporter: Seth Wiesman >Priority: Major > > Flink supports usage of both S3 presto and S3 hadoop within the same session > via the plugin system. When this happens the scheme 's3://' is and references > whichever filesystem happened to have been loaded last. Instead, users should > use the schemes 's3p' and 's3a' respectively which are unique. > There should be a configuration option that disallows the use of ambiguous > file schemes to prevent strange behavior in produciton. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-17194) TPC-DS end-to-end test fails due to missing execution attempt
[
https://issues.apache.org/jira/browse/FLINK-17194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110071#comment-17110071
]
Till Rohrmann commented on FLINK-17194:
---
Another instance reported by [~rmetzger]. The problem also occurs with the
increased akka ask timeout: Hadoop 3.1.3 nightly profile:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=1579&view=logs&j=91bf6583-3fb2-592f-e4d4-d79d79c3230a&t=94459a52-42b6-5bfc-5d74-690b5d3c6de8.
{code}
2020-05-17T06:39:29.8582972Z [INFO]Run TPC-DS query 62 success.
2020-05-17T06:39:29.8584484Z [INFO]Run TPC-DS query 63 ...
2020-05-17T06:39:30.1995132Z Job has been submitted with JobID
92560393bbba78447aa14558d0b08014
2020-05-17T06:40:12.9847942Z
2020-05-17T06:40:12.9853097Z
2020-05-17T06:40:12.9856909Z The program finished with the following exception:
2020-05-17T06:40:12.9859275Z
2020-05-17T06:40:12.9885450Z
org.apache.flink.client.program.ProgramInvocationException: The main method
caused an error: org.apache.flink.client.program.ProgramInvocationException:
Job failed (JobID: 92560393bbba78447aa14558d0b08014)
2020-05-17T06:40:12.9889713Zat
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:302)
2020-05-17T06:40:12.9890513Zat
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198)
2020-05-17T06:40:12.9891153Zat
org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:148)
2020-05-17T06:40:12.9892141Zat
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:689)
2020-05-17T06:40:12.9893860Zat
org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:227)
2020-05-17T06:40:12.9894484Zat
org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:906)
2020-05-17T06:40:12.9899582Zat
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:982)
2020-05-17T06:40:12.9900278Zat
org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
2020-05-17T06:40:12.9901521Zat
org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:982)
2020-05-17T06:40:12.9902541Z Caused by:
java.util.concurrent.ExecutionException:
org.apache.flink.client.program.ProgramInvocationException: Job failed (JobID:
92560393bbba78447aa14558d0b08014)
2020-05-17T06:40:12.9903292Zat
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
2020-05-17T06:40:12.9905140Zat
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
2020-05-17T06:40:12.9905834Zat
org.apache.flink.table.tpcds.TpcdsTestProgram.main(TpcdsTestProgram.java:109)
2020-05-17T06:40:12.9906355Zat
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2020-05-17T06:40:12.9906880Zat
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2020-05-17T06:40:12.9907468Zat
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2020-05-17T06:40:12.9908022Zat
java.lang.reflect.Method.invoke(Method.java:498)
2020-05-17T06:40:12.9908569Zat
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:288)
2020-05-17T06:40:12.9909098Z... 8 more
2020-05-17T06:40:12.9909584Z Caused by:
org.apache.flink.client.program.ProgramInvocationException: Job failed (JobID:
92560393bbba78447aa14558d0b08014)
2020-05-17T06:40:12.9910277Zat
org.apache.flink.client.deployment.ClusterClientJobClientAdapter.lambda$null$6(ClusterClientJobClientAdapter.java:116)
2020-05-17T06:40:12.9910886Zat
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
2020-05-17T06:40:12.9912098Zat
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
2020-05-17T06:40:12.9912767Zat
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
2020-05-17T06:40:12.9913363Zat
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
2020-05-17T06:40:12.9914013Zat
org.apache.flink.client.program.rest.RestClusterClient.lambda$pollResourceAsync$22(RestClusterClient.java:602)
2020-05-17T06:40:12.9915121Zat
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
2020-05-17T06:40:12.9915848Zat
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
2020-05-17T06:40:12.9916515Zat
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
2020-05-17T06:40:12.9917245Zat
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
2020-05-17T06:40:12.9917955Zat
org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$8(FutureUtils.java:307)
2020-05-17T06:40:12.9918660Z
[jira] [Closed] (FLINK-17770) TPC-DS end-to-end test (Blink planner) fails with AskTimeoutException
[
https://issues.apache.org/jira/browse/FLINK-17770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann closed FLINK-17770.
-
Fix Version/s: (was: 1.11.0)
Resolution: Duplicate
> TPC-DS end-to-end test (Blink planner) fails with AskTimeoutException
> -
>
> Key: FLINK-17770
> URL: https://issues.apache.org/jira/browse/FLINK-17770
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination, Table SQL / Planner, Tests
>Affects Versions: 1.11.0
>Reporter: Robert Metzger
>Priority: Critical
> Labels: test-stability
>
> Hadoop 3.1.3 nightly profile:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=1579&view=logs&j=91bf6583-3fb2-592f-e4d4-d79d79c3230a&t=94459a52-42b6-5bfc-5d74-690b5d3c6de8
> {code}
> 2020-05-17T06:39:29.8582972Z [INFO]Run TPC-DS query 62 success.
> 2020-05-17T06:39:29.8584484Z [INFO]Run TPC-DS query 63 ...
> 2020-05-17T06:39:30.1995132Z Job has been submitted with JobID
> 92560393bbba78447aa14558d0b08014
> 2020-05-17T06:40:12.9847942Z
> 2020-05-17T06:40:12.9853097Z
>
> 2020-05-17T06:40:12.9856909Z The program finished with the following
> exception:
> 2020-05-17T06:40:12.9859275Z
> 2020-05-17T06:40:12.9885450Z
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: org.apache.flink.client.program.ProgramInvocationException:
> Job failed (JobID: 92560393bbba78447aa14558d0b08014)
> 2020-05-17T06:40:12.9889713Z at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:302)
> 2020-05-17T06:40:12.9890513Z at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198)
> 2020-05-17T06:40:12.9891153Z at
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:148)
> 2020-05-17T06:40:12.9892141Z at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:689)
> 2020-05-17T06:40:12.9893860Z at
> org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:227)
> 2020-05-17T06:40:12.9894484Z at
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:906)
> 2020-05-17T06:40:12.9899582Z at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:982)
> 2020-05-17T06:40:12.9900278Z at
> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
> 2020-05-17T06:40:12.9901521Z at
> org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:982)
> 2020-05-17T06:40:12.9902541Z Caused by:
> java.util.concurrent.ExecutionException:
> org.apache.flink.client.program.ProgramInvocationException: Job failed
> (JobID: 92560393bbba78447aa14558d0b08014)
> 2020-05-17T06:40:12.9903292Z at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> 2020-05-17T06:40:12.9905140Z at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> 2020-05-17T06:40:12.9905834Z at
> org.apache.flink.table.tpcds.TpcdsTestProgram.main(TpcdsTestProgram.java:109)
> 2020-05-17T06:40:12.9906355Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-05-17T06:40:12.9906880Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-05-17T06:40:12.9907468Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-05-17T06:40:12.9908022Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-05-17T06:40:12.9908569Z at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:288)
> 2020-05-17T06:40:12.9909098Z ... 8 more
> 2020-05-17T06:40:12.9909584Z Caused by:
> org.apache.flink.client.program.ProgramInvocationException: Job failed
> (JobID: 92560393bbba78447aa14558d0b08014)
> 2020-05-17T06:40:12.9910277Z at
> org.apache.flink.client.deployment.ClusterClientJobClientAdapter.lambda$null$6(ClusterClientJobClientAdapter.java:116)
> 2020-05-17T06:40:12.9910886Z at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
> 2020-05-17T06:40:12.9912098Z at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
> 2020-05-17T06:40:12.9912767Z at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> 2020-05-17T06:40:12.9913363Z at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
> 2020-05-17T06:40:12.9914013Z at
> org.apache.flink.client.program.rest.RestClusterClient.lambda$pollResourceAsync$22(RestClusterClient.java:602)
> 2020-05-17T06:40:12.9915121Z at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> 202
[GitHub] [flink] liuyongvs commented on pull request #12144: [FLINK-17384][flink-dist] support read hbase conf dir from flink.conf.
liuyongvs commented on pull request #12144:
URL: https://github.com/apache/flink/pull/12144#issuecomment-630057647
hi @rmetzger
I have committed my code, but the log is below, which is not relevant with
my code
2020-05-14T13:46:32.9352627Z
[ERROR]
Failures:
2020-05-14T13:46:32.9361371Z
[ERROR]
KafkaProducerExactlyOnceITCase>KafkaProducerTestBase.testExactlyOnceRegularSink:309->KafkaProducerTestBase.testExactlyOnce:370
Test failed: Job execution failed
ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:2.22.1:test (end-to-end-tests)
on project flink-metrics-availability-test: Unable to generate classpath:
org.apache.maven.artifact.resolver.ArtifactResolutionException: Could not
transfer artifact org.apache.maven.surefire:surefire-grouper:jar:2.22.1 from/to
alicloud-mvn-mirror
(http://mavenmirror.alicloud.dak8s.net:/repository/maven-central/): Entry
[id:18]
[route:{}->http://mavenmirror.alicloud.dak8s.net:]
[state:null] has not been leased from this pool.
How to solve it , and why that happends. how to make the flinkbot rerun
azure.
I am looking forward your response. Thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-17795) Add an example to show how to leverage GPU resources
Yangze Guo created FLINK-17795: -- Summary: Add an example to show how to leverage GPU resources Key: FLINK-17795 URL: https://issues.apache.org/jira/browse/FLINK-17795 Project: Flink Issue Type: Task Components: Examples Reporter: Yangze Guo Fix For: 1.11.0 Add an example to show how to leverage GPU resources. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15670) Provide a Kafka Source/Sink pair that aligns Kafka's Partitions and Flink's KeyGroups
[ https://issues.apache.org/jira/browse/FLINK-15670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110072#comment-17110072 ] Yuan Mei commented on FLINK-15670: -- The code has conflicts with [FLINK-17659] Rework WatermarkStrategy, add Suppliers for TimestampAssigner/WatermarkGenerator committed last night. Fix the problem in the new PR: [https://github.com/apache/flink/pull/12219] > Provide a Kafka Source/Sink pair that aligns Kafka's Partitions and Flink's > KeyGroups > - > > Key: FLINK-15670 > URL: https://issues.apache.org/jira/browse/FLINK-15670 > Project: Flink > Issue Type: New Feature > Components: API / DataStream, Connectors / Kafka >Reporter: Stephan Ewen >Assignee: Yuan Mei >Priority: Major > Labels: pull-request-available, usability > Fix For: 1.11.0 > > Time Spent: 10m > Remaining Estimate: 0h > > This Source/Sink pair would serve two purposes: > 1. You can read topics that are already partitioned by key and process them > without partitioning them again (avoid shuffles) > 2. You can use this to shuffle through Kafka, thereby decomposing the job > into smaller jobs and independent pipelined regions that fail over > independently. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-17675) Resolve CVE-2019-11358 from jquery
[ https://issues.apache.org/jira/browse/FLINK-17675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-17675: --- Component/s: Documentation > Resolve CVE-2019-11358 from jquery > -- > > Key: FLINK-17675 > URL: https://issues.apache.org/jira/browse/FLINK-17675 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Koala Lam >Priority: Major > > https://nvd.nist.gov/vuln/detail/CVE-2019-11358 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-15666) GPU scheduling support in Kubernetes mode
[ https://issues.apache.org/jira/browse/FLINK-15666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yangze Guo closed FLINK-15666. -- Fix Version/s: 1.11.0 Resolution: Duplicate > GPU scheduling support in Kubernetes mode > - > > Key: FLINK-15666 > URL: https://issues.apache.org/jira/browse/FLINK-15666 > Project: Flink > Issue Type: New Feature > Components: Deployment / Kubernetes >Reporter: Canbin Zheng >Priority: Major > Fix For: 1.11.0 > > > This is an umbrella ticket for work on GPU scheduling in Kubernetes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-17750) YARNHighAvailabilityITCase.testKillYarnSessionClusterEntrypoint failed on azure
[
https://issues.apache.org/jira/browse/FLINK-17750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110074#comment-17110074
]
Till Rohrmann commented on FLINK-17750:
---
The problem might be that a previous Yarn test did not properly stop:
{code}
Suppressed: java.lang.AssertionError: There is at least one application on the
cluster that is not finished.[App application_1 589586092047_0002 is in state
RUNNING.]
2020-05-15T23:42:29.5363571Zat
org.junit.Assert.fail(Assert.java:88)
2020-05-15T23:42:29.5364060Zat
org.apache.flink.yarn.YarnTestBase$CleanupYarnApplication.close(YarnTestBase.java:286)
2020-05-15T23:42:29.5364448Zat
org.apache.flink.yarn.YarnTestBase.runTest(YarnTestBase.java:259)
2020-05-15T23:42:29.5364745Z... 36 more
{code}
> YARNHighAvailabilityITCase.testKillYarnSessionClusterEntrypoint failed on
> azure
> ---
>
> Key: FLINK-17750
> URL: https://issues.apache.org/jira/browse/FLINK-17750
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.11.0
>Reporter: Roman Khachatryan
>Priority: Critical
> Labels: test-stability
>
> [https://dev.azure.com/khachatryanroman/810e80cc-0656-4d3c-9d8c-186764456a01/_apis/build/builds/6/logs/156]
>
> {code:java}
> 2020-05-15T23:42:29.5307581Z [ERROR]
> testKillYarnSessionClusterEntrypoint(org.apache.flink.yarn.YARNHighAvailabilityITCase)
> Time elapsed: 21.68 s <<< ERROR!
> 2020-05-15T23:42:29.5308406Z java.util.concurrent.ExecutionException:
> 2020-05-15T23:42:29.5308864Z
> org.apache.flink.runtime.rest.util.RestClientException: [Internal server
> error., 2020-05-15T23:42:29.5309678Z java.util.concurrent.TimeoutException:
> Invocation of public abstract java.util.concurrent.CompletableFuture
> org.apache.flink.runt
> ime.dispatcher.DispatcherGateway.requestJob(org.apache.flink.api.common.JobID,org.apache.flink.api.common.time.Time)
> timed out.
> 2020-05-15T23:42:29.5310322Zat com.sun.proxy.$Proxy33.requestJob(Unknown
> Source)
> 2020-05-15T23:42:29.5311018Zat
> org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCache.getExecutionGraphInternal(DefaultExecutionGraphCach
> e.java:103)
> 2020-05-15T23:42:29.5311704Zat
> org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCache.getExecutionGraph(DefaultExecutionGraphCache.java:7
> 1)
> 2020-05-15T23:42:29.5312355Zat
> org.apache.flink.runtime.rest.handler.job.AbstractExecutionGraphHandler.handleRequest(AbstractExecutionGraphHandler.java:75
> )
> 2020-05-15T23:42:29.5312924Zat
> org.apache.flink.runtime.rest.handler.AbstractRestHandler.respondToRequest(AbstractRestHandler.java:73)
> 2020-05-15T23:42:29.5313423Zat
> org.apache.flink.runtime.rest.handler.AbstractHandler.respondAsLeader(AbstractHandler.java:172)
> 2020-05-15T23:42:29.5314497Zat
> org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.lambda$channelRead0$0(LeaderRetrievalHandler.java:81)
> 2020-05-15T23:42:29.5315083Zat
> java.util.Optional.ifPresent(Optional.java:159)
> 2020-05-15T23:42:29.5315474Zat
> org.apache.flink.util.OptionalConsumer.ifPresent(OptionalConsumer.java:46)
> 2020-05-15T23:42:29.5315979Zat
> org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:78)
> 2020-05-15T23:42:29.5316520Zat
> org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:49)
> 2020-05-15T23:42:29.5317092Zat
> org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:10
> 5)
> 2020-05-15T23:42:29.5317705Zat
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerConte
> xt.java:374)
> 2020-05-15T23:42:29.5318586Zat
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerConte
> xt.java:360)
> 2020-05-15T23:42:29.5319249Zat
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext
> .java:352)
> 2020-05-15T23:42:29.5319729Zat
> org.apache.flink.runtime.rest.handler.router.RouterHandler.routed(RouterHandler.java:110)
> 2020-05-15T23:42:29.5320136Zat
> org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:89)
> 2020-05-15T23:42:29.5320742Zat
> org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:54)
> 2020-05-15T23:42:29.5321195Zat
> org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(Si
[jira] [Updated] (FLINK-17726) Scheduler should take care of tasks directly canceled by TaskManager
[ https://issues.apache.org/jira/browse/FLINK-17726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-17726: -- Affects Version/s: 1.11.0 > Scheduler should take care of tasks directly canceled by TaskManager > > > Key: FLINK-17726 > URL: https://issues.apache.org/jira/browse/FLINK-17726 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.11.0, 1.12.0 >Reporter: Zhu Zhu >Priority: Critical > Fix For: 1.12.0 > > > JobManager will not trigger failure handling when receiving CANCELED task > update. > This is because CANCELED tasks are usually caused by another FAILED task. > These CANCELED tasks will be restarted by the failover process triggered > FAILED task. > However, if a task is directly CANCELED by TaskManager due to its own runtime > issue, the task will not be recovered by JM and thus the job would hang. > This is a potential issue and we should avoid it. > A possible solution is to let JobManager treat tasks transitioning to > CANCELED from all states except from CANCELING as failed tasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] kl0u edited a comment on pull request #12132: [FLINK-17593][Connectors/FileSystem] Support arbitrary recovery mechanism for PartFileWriter
kl0u edited a comment on pull request #12132: URL: https://github.com/apache/flink/pull/12132#issuecomment-630047870 Hi, these are some comments with renamings and moving classes for consistency https://github.com/kl0u/flink/tree/FLINK-17593-comments. Please check them out. Probably we can also merge them later, but if it is possible it makes sense to do it before cutting the branch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-17725) FileUploadHandlerTest.testUploadCleanupOnFailure fails with "SocketTimeout timeout"
[
https://issues.apache.org/jira/browse/FLINK-17725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-17725:
--
Priority: Critical (was: Major)
> FileUploadHandlerTest.testUploadCleanupOnFailure fails with "SocketTimeout
> timeout"
> ---
>
> Key: FLINK-17725
> URL: https://issues.apache.org/jira/browse/FLINK-17725
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Reporter: Robert Metzger
>Priority: Critical
> Labels: test-stability
>
> CI:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=1392&view=logs&j=0da23115-68bb-5dcd-192c-bd4c8adebde1&t=4ed44b66-cdd6-5dcf-5f6a-88b07dda665d
> ( select 1st attempt)
> {code}
> 2020-05-15T09:01:40.0547958Z [ERROR]
> testUploadCleanupOnFailure(org.apache.flink.runtime.rest.FileUploadHandlerTest)
> Time elapsed: 10.415 s <<< ERROR!
> 2020-05-15T09:01:40.0548716Z java.net.SocketTimeoutException: timeout
> 2020-05-15T09:01:40.0549048Z at
> okio.Okio$4.newTimeoutException(Okio.java:227)
> 2020-05-15T09:01:40.0549361Z at okio.AsyncTimeout.exit(AsyncTimeout.java:284)
> 2020-05-15T09:01:40.0549688Z at
> okio.AsyncTimeout$2.read(AsyncTimeout.java:240)
> 2020-05-15T09:01:40.0552454Z at
> okio.RealBufferedSource.indexOf(RealBufferedSource.java:344)
> 2020-05-15T09:01:40.0554987Z at
> okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:216)
> 2020-05-15T09:01:40.0555636Z at
> okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:210)
> 2020-05-15T09:01:40.0556307Z at
> okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189)
> 2020-05-15T09:01:40.0556856Z at
> okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75)
> 2020-05-15T09:01:40.0557505Z at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
> 2020-05-15T09:01:40.0558021Z at
> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
> 2020-05-15T09:01:40.0558498Z at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
> 2020-05-15T09:01:40.0558932Z at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
> 2020-05-15T09:01:40.0559381Z at
> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
> 2020-05-15T09:01:40.0559803Z at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
> 2020-05-15T09:01:40.0560262Z at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
> 2020-05-15T09:01:40.0561022Z at
> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
> 2020-05-15T09:01:40.0561701Z at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
> 2020-05-15T09:01:40.0562439Z at
> okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
> 2020-05-15T09:01:40.0563170Z at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
> 2020-05-15T09:01:40.0565934Z at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
> 2020-05-15T09:01:40.0566781Z at
> okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185)
> 2020-05-15T09:01:40.0575046Z at okhttp3.RealCall.execute(RealCall.java:69)
> 2020-05-15T09:01:40.0575858Z at
> org.apache.flink.runtime.rest.FileUploadHandlerTest.testUploadCleanupOnFailure(FileUploadHandlerTest.java:250)
> 2020-05-15T09:01:40.0576567Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-05-15T09:01:40.0577242Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-05-15T09:01:40.0577979Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-05-15T09:01:40.0578594Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-05-15T09:01:40.0579234Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-05-15T09:01:40.0580279Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-05-15T09:01:40.0581129Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-05-15T09:01:40.0581862Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-05-15T09:01:40.0582538Z at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2020-05-15T09:01:40.0583174Z at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2020-05-15T09:01:40.0583934Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2020-05-15T09:01:40.0584501Z
[GitHub] [flink] curcur commented on pull request #11725: [FLINK-15670][API] Provide a Kafka Source/Sink pair as KafkaShuffle
curcur commented on pull request #11725: URL: https://github.com/apache/flink/pull/11725#issuecomment-630059574 The PR is reverted because of conflicts with [FLINK-17659] Rework WatermarkStrategy, add Suppliers for TimestampAssigner/WatermarkGenerator committed last night. File the new PR: https://github.com/apache/flink/pull/12219 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-16303) Add Rest Handler to list JM Logfiles and enable reading Logs by Filename
[
https://issues.apache.org/jira/browse/FLINK-16303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Yao updated FLINK-16303:
-
Release Note: Requesting an unavailable log or stdout file from the
JobManager's HTTP server returns status code 404 now. In previous releases, the
HTTP server would return a file with '(file unavailable)' as its content.
> Add Rest Handler to list JM Logfiles and enable reading Logs by Filename
>
>
> Key: FLINK-16303
> URL: https://issues.apache.org/jira/browse/FLINK-16303
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / REST
>Reporter: lining
>Assignee: lining
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> * list jobmanager all log file
> ** /jobmanager/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "jobmanager.log",
> "size": 12529
> }
> ]
> }{code}
> * read jobmanager log file
> ** /jobmanager/log/[filename]
> ** response: same as jobmanager's log
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-17726) Scheduler should take care of tasks directly canceled by TaskManager
[ https://issues.apache.org/jira/browse/FLINK-17726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-17726: -- Fix Version/s: 1.11.0 > Scheduler should take care of tasks directly canceled by TaskManager > > > Key: FLINK-17726 > URL: https://issues.apache.org/jira/browse/FLINK-17726 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.11.0, 1.12.0 >Reporter: Zhu Zhu >Priority: Critical > Fix For: 1.11.0, 1.12.0 > > > JobManager will not trigger failure handling when receiving CANCELED task > update. > This is because CANCELED tasks are usually caused by another FAILED task. > These CANCELED tasks will be restarted by the failover process triggered > FAILED task. > However, if a task is directly CANCELED by TaskManager due to its own runtime > issue, the task will not be recovered by JM and thus the job would hang. > This is a potential issue and we should avoid it. > A possible solution is to let JobManager treat tasks transitioning to > CANCELED from all states except from CANCELING as failed tasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-17675) Resolve CVE-2019-11358 from jquery
[
https://issues.apache.org/jira/browse/FLINK-17675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110077#comment-17110077
]
Robert Metzger commented on FLINK-17675:
Thanks for opening this ticket. We are using jQuery 1.11.2 in
{{docs/page/js/jquery.min.js}}, the CVE affects all versions before 3.4.0.
> Resolve CVE-2019-11358 from jquery
> --
>
> Key: FLINK-17675
> URL: https://issues.apache.org/jira/browse/FLINK-17675
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Koala Lam
>Priority: Major
>
> https://nvd.nist.gov/vuln/detail/CVE-2019-11358
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-17725) FileUploadHandlerTest.testUploadCleanupOnFailure fails with "SocketTimeout timeout"
[
https://issues.apache.org/jira/browse/FLINK-17725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-17725:
--
Affects Version/s: 1.11.0
> FileUploadHandlerTest.testUploadCleanupOnFailure fails with "SocketTimeout
> timeout"
> ---
>
> Key: FLINK-17725
> URL: https://issues.apache.org/jira/browse/FLINK-17725
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.11.0
>Reporter: Robert Metzger
>Priority: Critical
> Labels: test-stability
>
> CI:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=1392&view=logs&j=0da23115-68bb-5dcd-192c-bd4c8adebde1&t=4ed44b66-cdd6-5dcf-5f6a-88b07dda665d
> ( select 1st attempt)
> {code}
> 2020-05-15T09:01:40.0547958Z [ERROR]
> testUploadCleanupOnFailure(org.apache.flink.runtime.rest.FileUploadHandlerTest)
> Time elapsed: 10.415 s <<< ERROR!
> 2020-05-15T09:01:40.0548716Z java.net.SocketTimeoutException: timeout
> 2020-05-15T09:01:40.0549048Z at
> okio.Okio$4.newTimeoutException(Okio.java:227)
> 2020-05-15T09:01:40.0549361Z at okio.AsyncTimeout.exit(AsyncTimeout.java:284)
> 2020-05-15T09:01:40.0549688Z at
> okio.AsyncTimeout$2.read(AsyncTimeout.java:240)
> 2020-05-15T09:01:40.0552454Z at
> okio.RealBufferedSource.indexOf(RealBufferedSource.java:344)
> 2020-05-15T09:01:40.0554987Z at
> okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:216)
> 2020-05-15T09:01:40.0555636Z at
> okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:210)
> 2020-05-15T09:01:40.0556307Z at
> okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189)
> 2020-05-15T09:01:40.0556856Z at
> okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75)
> 2020-05-15T09:01:40.0557505Z at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
> 2020-05-15T09:01:40.0558021Z at
> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
> 2020-05-15T09:01:40.0558498Z at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
> 2020-05-15T09:01:40.0558932Z at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
> 2020-05-15T09:01:40.0559381Z at
> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
> 2020-05-15T09:01:40.0559803Z at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
> 2020-05-15T09:01:40.0560262Z at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
> 2020-05-15T09:01:40.0561022Z at
> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
> 2020-05-15T09:01:40.0561701Z at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
> 2020-05-15T09:01:40.0562439Z at
> okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
> 2020-05-15T09:01:40.0563170Z at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
> 2020-05-15T09:01:40.0565934Z at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
> 2020-05-15T09:01:40.0566781Z at
> okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185)
> 2020-05-15T09:01:40.0575046Z at okhttp3.RealCall.execute(RealCall.java:69)
> 2020-05-15T09:01:40.0575858Z at
> org.apache.flink.runtime.rest.FileUploadHandlerTest.testUploadCleanupOnFailure(FileUploadHandlerTest.java:250)
> 2020-05-15T09:01:40.0576567Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-05-15T09:01:40.0577242Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-05-15T09:01:40.0577979Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-05-15T09:01:40.0578594Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-05-15T09:01:40.0579234Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-05-15T09:01:40.0580279Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-05-15T09:01:40.0581129Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-05-15T09:01:40.0581862Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-05-15T09:01:40.0582538Z at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2020-05-15T09:01:40.0583174Z at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2020-05-15T09:01:40.0583934Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2020-0
[jira] [Updated] (FLINK-17725) FileUploadHandlerTest.testUploadCleanupOnFailure fails with "SocketTimeout timeout"
[
https://issues.apache.org/jira/browse/FLINK-17725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-17725:
--
Fix Version/s: 1.11.0
> FileUploadHandlerTest.testUploadCleanupOnFailure fails with "SocketTimeout
> timeout"
> ---
>
> Key: FLINK-17725
> URL: https://issues.apache.org/jira/browse/FLINK-17725
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.11.0
>Reporter: Robert Metzger
>Priority: Critical
> Labels: test-stability
> Fix For: 1.11.0
>
>
> CI:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=1392&view=logs&j=0da23115-68bb-5dcd-192c-bd4c8adebde1&t=4ed44b66-cdd6-5dcf-5f6a-88b07dda665d
> ( select 1st attempt)
> {code}
> 2020-05-15T09:01:40.0547958Z [ERROR]
> testUploadCleanupOnFailure(org.apache.flink.runtime.rest.FileUploadHandlerTest)
> Time elapsed: 10.415 s <<< ERROR!
> 2020-05-15T09:01:40.0548716Z java.net.SocketTimeoutException: timeout
> 2020-05-15T09:01:40.0549048Z at
> okio.Okio$4.newTimeoutException(Okio.java:227)
> 2020-05-15T09:01:40.0549361Z at okio.AsyncTimeout.exit(AsyncTimeout.java:284)
> 2020-05-15T09:01:40.0549688Z at
> okio.AsyncTimeout$2.read(AsyncTimeout.java:240)
> 2020-05-15T09:01:40.0552454Z at
> okio.RealBufferedSource.indexOf(RealBufferedSource.java:344)
> 2020-05-15T09:01:40.0554987Z at
> okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:216)
> 2020-05-15T09:01:40.0555636Z at
> okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:210)
> 2020-05-15T09:01:40.0556307Z at
> okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189)
> 2020-05-15T09:01:40.0556856Z at
> okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75)
> 2020-05-15T09:01:40.0557505Z at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
> 2020-05-15T09:01:40.0558021Z at
> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
> 2020-05-15T09:01:40.0558498Z at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
> 2020-05-15T09:01:40.0558932Z at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
> 2020-05-15T09:01:40.0559381Z at
> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
> 2020-05-15T09:01:40.0559803Z at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
> 2020-05-15T09:01:40.0560262Z at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
> 2020-05-15T09:01:40.0561022Z at
> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
> 2020-05-15T09:01:40.0561701Z at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
> 2020-05-15T09:01:40.0562439Z at
> okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
> 2020-05-15T09:01:40.0563170Z at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
> 2020-05-15T09:01:40.0565934Z at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
> 2020-05-15T09:01:40.0566781Z at
> okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185)
> 2020-05-15T09:01:40.0575046Z at okhttp3.RealCall.execute(RealCall.java:69)
> 2020-05-15T09:01:40.0575858Z at
> org.apache.flink.runtime.rest.FileUploadHandlerTest.testUploadCleanupOnFailure(FileUploadHandlerTest.java:250)
> 2020-05-15T09:01:40.0576567Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-05-15T09:01:40.0577242Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-05-15T09:01:40.0577979Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-05-15T09:01:40.0578594Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-05-15T09:01:40.0579234Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-05-15T09:01:40.0580279Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-05-15T09:01:40.0581129Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-05-15T09:01:40.0581862Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-05-15T09:01:40.0582538Z at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2020-05-15T09:01:40.0583174Z at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2020-05-15T09:01:40.0583934Z at
> org.junit.rules.RunRules.evaluat
[GitHub] [flink] flinkbot commented on pull request #12219: [FLINK-15670][API] Provide a Kafka Source/Sink pair as KafkaShuffle
flinkbot commented on pull request #12219: URL: https://github.com/apache/flink/pull/12219#issuecomment-630060886 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit c849ac42145725091a3ab166095a672eaf33f350 (Mon May 18 09:26:52 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12220: [FLINK-14255][hive] Integrate mapred record writer to hive streaming sink
flinkbot commented on pull request #12220: URL: https://github.com/apache/flink/pull/12220#issuecomment-630060789 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 7b2231fb942a944c54a29c2bcc57bfaaff2fe797 (Mon May 18 09:26:40 UTC 2020) **Warnings:** * **4 pom.xml files were touched**: Check for build and licensing issues. * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-17676) Is there some way to rollback the .out file of TaskManager
[
https://issues.apache.org/jira/browse/FLINK-17676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-17676:
---
Component/s: Command Line Client
> Is there some way to rollback the .out file of TaskManager
> --
>
> Key: FLINK-17676
> URL: https://issues.apache.org/jira/browse/FLINK-17676
> Project: Flink
> Issue Type: Improvement
> Components: Command Line Client
>Reporter: JieFang.He
>Priority: Major
>
> When use .print() API, the result all write to the out file, But there is no
> way to rollback the out file.
>
> out in flink-daemon.sh
> {code:java}
> // $JAVA_RUN $JVM_ARGS ${FLINK_ENV_JAVA_OPTS} "${log_setting[@]}" -classpath
> "`manglePathList "$FLINK_TM_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS"`"
> ${CLASS_TO_RUN} "${ARGS[@]}" > "$out" 200<&- 2>&1 < /dev/null &
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-17618) Update the outdated comments in the log4j properties files
[
https://issues.apache.org/jira/browse/FLINK-17618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-17618:
---
Component/s: Command Line Client
> Update the outdated comments in the log4j properties files
> --
>
> Key: FLINK-17618
> URL: https://issues.apache.org/jira/browse/FLINK-17618
> Project: Flink
> Issue Type: Bug
> Components: Command Line Client
>Reporter: Yang Wang
>Priority: Major
> Labels: pull-request-available
>
> When we upgrade the log4j to log4j2, there are some residual log4j logger
> configuration in the comments. Just like following,
> log4j.properties and log4j-console.properties
> {code:java}
> # Uncomment this if you want to _only_ change Flink's logging
> #log4j.logger.org.apache.flink=INFO
> {code}
> We should update them to the log4j2 format.
>
> {code:java}
> # Uncomment this if you want to _only_ change Flink's logging
> logger.flink.name = org.apache.flink
> logger.flink.level = INFO
> {code}
>
> cc [~chesnay]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-17102) Add -Dkubernetes.container.image= for the start-flink-session section
[
https://issues.apache.org/jira/browse/FLINK-17102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-17102:
---
Component/s: Deployment / Kubernetes
> Add -Dkubernetes.container.image= for the start-flink-session
> section
>
>
> Key: FLINK-17102
> URL: https://issues.apache.org/jira/browse/FLINK-17102
> Project: Flink
> Issue Type: Sub-task
> Components: Deployment / Kubernetes
>Reporter: Canbin Zheng
>Priority: Minor
>
> Add {{-Dkubernetes.container.image=}} as a guide for new users in
> the existing command:
> {quote}{{}}
>
> {{./bin/kubernetes-session.sh \}}
> {{-Dkubernetes.cluster-id=
> \-Dtaskmanager.memory.process.size=4096m \-Dkubernetes.taskmanager.cpu=2
> \-Dtaskmanager.numberOfTaskSlots=4
> \-Dresourcemanager.taskmanager-timeout=360}}{{}}
> {quote}
> Details could refer to
> [https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/native_kubernetes.html#start-flink-session]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-17762) Postgres Catalog should pass table's primary key to catalogTable
[ https://issues.apache.org/jira/browse/FLINK-17762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-17762: --- Component/s: Table SQL / Ecosystem > Postgres Catalog should pass table's primary key to catalogTable > > > Key: FLINK-17762 > URL: https://issues.apache.org/jira/browse/FLINK-17762 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Ecosystem >Reporter: Leonard Xu >Priority: Major > > for upsert query, if the table comes from a catalog rather than create in > FLINK, Postgres Catalog should pass table's primary key to catalogTable so > that JdbcDynamicTableSink can determine to work on upsert mode or append only > mode. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-17783) Add array,map,row types support for orc row writer
[ https://issues.apache.org/jira/browse/FLINK-17783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-17783: --- Component/s: Table SQL / Ecosystem > Add array,map,row types support for orc row writer > -- > > Key: FLINK-17783 > URL: https://issues.apache.org/jira/browse/FLINK-17783 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Ecosystem >Reporter: Jingsong Lee >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-17331) Add NettyMessageContent interface for all the class which could be write to NettyMessage
[
https://issues.apache.org/jira/browse/FLINK-17331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-17331:
---
Component/s: Runtime / Network
> Add NettyMessageContent interface for all the class which could be write to
> NettyMessage
>
>
> Key: FLINK-17331
> URL: https://issues.apache.org/jira/browse/FLINK-17331
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
>Reporter: Yangze Guo
>Priority: Minor
>
> Currently, there are some classes, e.g. {{JobVertexID}},
> {{ExecutionAttemptID}} need to write to {{NettyMessage}}. However, the size
> of these classes in {{ByteBuf}} are directly written in {{NettyMessage}}
> class, which is error-prone. If someone edits those classes, there would be
> no warning or error during the compile phase. I think it would be better to
> add a {{NettyMessageContent}}(the name could be discussed) interface:
> {code:java}
> public interface NettyMessageContent {
> void writeTo(ByteBuf bug)
> int getContentLen();
> }
> {code}
> Regarding the {{fromByteBuf}}, since it is a static method, we could not add
> it to the interface. We might explain it in the javaDoc of
> {{NettyMessageContent}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15792) Make Flink logs accessible via kubectl logs per default
[
https://issues.apache.org/jira/browse/FLINK-15792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110083#comment-17110083
]
Yang Wang commented on FLINK-15792:
---
[~trohrmann] No, even after FLINK-17166, we still could not achieve this for
native k8s. Because we are using {{log4j.properties}} for the JobManager and
TaskManager start command.
Of cause, we could simply change it to {{flink-console.properties}}. And then
the STDOUT/STDERR could not be accessed via web dashboard. I think this is a
burden.
For the solution in current doc[1], the STDOUT/STDERR could be accessed via
dashboard, but not {{kubectl logs}}.
[1].
[https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/native_kubernetes.html#log-files]
> Make Flink logs accessible via kubectl logs per default
> ---
>
> Key: FLINK-15792
> URL: https://issues.apache.org/jira/browse/FLINK-15792
> Project: Flink
> Issue Type: Sub-task
> Components: Deployment / Kubernetes
>Affects Versions: 1.10.0
>Reporter: Till Rohrmann
>Priority: Major
> Fix For: 1.11.0, 1.10.2
>
>
> I think we should make Flink's logs accessible via {{kubectl logs}} per
> default. Firstly, this is the idiomatic way to obtain the logs from a
> container on Kubernetes. Secondly, especially if something does not work and
> the container cannot start/stops abruptly, there is no way to log into the
> container and look for the log.file. This makes debugging the setup quite
> hard.
> I think the best way would be to create the Flink Docker image in such a way
> that it logs to stdout. In order to allow access to the log file from the web
> ui, it should also create a log file. One way to achieve this is to add a
> ConsoleAppender to the respective logging configuration. Another way could be
> to start the process in the console mode and then to teeing the stdout output
> into the log file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15813) Set default value of jobmanager.execution.failover-strategy to region
[
https://issues.apache.org/jira/browse/FLINK-15813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110085#comment-17110085
]
Till Rohrmann commented on FLINK-15813:
---
[~zhuzh] can we resolve this issue now? I can help with the review of it.
> Set default value of jobmanager.execution.failover-strategy to region
> -
>
> Key: FLINK-15813
> URL: https://issues.apache.org/jira/browse/FLINK-15813
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Till Rohrmann
>Assignee: Zhu Zhu
>Priority: Blocker
> Labels: usability
> Fix For: 1.11.0
>
>
> We should set the default value of {{jobmanager.execution.failover-strategy}}
> to {{region}}. This might require to adapt existing tests to make them pass.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] JingsongLi commented on a change in pull request #12212: [FLINK-17626][fs-connector] Fs connector should use FLIP-122 format options style
JingsongLi commented on a change in pull request #12212:
URL: https://github.com/apache/flink/pull/12212#discussion_r426494943
##
File path:
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/filesystem/FileSystemOptions.java
##
@@ -29,6 +29,35 @@
*/
public class FileSystemOptions {
+ public static final ConfigOption PATH = key("path")
+ .stringType()
+ .noDefaultValue()
+ .withDescription("The path of a directory");
+
+ public static final ConfigOption PARTITION_DEFAULT_NAME =
key("partition.default-name")
+ .stringType()
+ .defaultValue("__DEFAULT_PARTITION__")
+ .withDescription("The default partition name in case
the dynamic partition" +
+ " column value is null/empty string");
+
+ public static final ConfigOption SINK_ROLLING_POLICY_FILE_SIZE =
key("sink.rolling-policy.file-size")
+ .longType()
+ .defaultValue(1024L * 1024L * 128L)
+ .withDescription("The maximum part file size before
rolling (by default 128MB).");
+
+ public static final ConfigOption
SINK_ROLLING_POLICY_TIME_INTERVAL = key("sink.rolling-policy.time.interval")
Review comment:
Good catch, I think yes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Assigned] (FLINK-17558) Partitions are released in TaskExecutor Main Thread
[
https://issues.apache.org/jira/browse/FLINK-17558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler reassigned FLINK-17558:
Assignee: Chesnay Schepler
> Partitions are released in TaskExecutor Main Thread
> ---
>
> Key: FLINK-17558
> URL: https://issues.apache.org/jira/browse/FLINK-17558
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.11.0
>Reporter: Gary Yao
>Assignee: Chesnay Schepler
>Priority: Critical
> Fix For: 1.11.0
>
>
> Partitions are released in the main thread of the TaskExecutor (see the
> stacktrace below). This can lead to missed heartbeats, timeouts of RPCs, etc.
> because deleting files is blocking I/O. The partitions should be released in
> a devoted I/O thread pool ({{TaskExecutor#ioExecutor}} is a candidate but
> requires a higher default thread count).
> {noformat}
> 2020-05-06T19:13:12.4383402Z "flink-akka.actor.default-dispatcher-35" #3555
> prio=5 os_prio=0 tid=0x7f7fcc071000 nid=0x1f3f9 runnable
> [0x7f7fd302c000]
> 2020-05-06T19:13:12.4383983Zjava.lang.Thread.State: RUNNABLE
> 2020-05-06T19:13:12.4384519Zat
> sun.nio.fs.UnixNativeDispatcher.unlink0(Native Method)
> 2020-05-06T19:13:12.4384971Zat
> sun.nio.fs.UnixNativeDispatcher.unlink(UnixNativeDispatcher.java:146)
> 2020-05-06T19:13:12.4385465Zat
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:231)
> 2020-05-06T19:13:12.4386000Zat
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> 2020-05-06T19:13:12.4386458Zat java.nio.file.Files.delete(Files.java:1126)
> 2020-05-06T19:13:12.4386968Zat
> org.apache.flink.runtime.io.network.partition.FileChannelBoundedData.close(FileChannelBoundedData.java:93)
> 2020-05-06T19:13:12.4388088Zat
> org.apache.flink.runtime.io.network.partition.BoundedBlockingSubpartition.checkReaderReferencesAndDispose(BoundedBlockingSubpartition.java:247)
> 2020-05-06T19:13:12.4388765Zat
> org.apache.flink.runtime.io.network.partition.BoundedBlockingSubpartition.release(BoundedBlockingSubpartition.java:208)
> 2020-05-06T19:13:12.4389444Z- locked <0xff836d78> (a
> java.lang.Object)
> 2020-05-06T19:13:12.4389905Zat
> org.apache.flink.runtime.io.network.partition.ResultPartition.release(ResultPartition.java:290)
> 2020-05-06T19:13:12.4390481Zat
> org.apache.flink.runtime.io.network.partition.ResultPartitionManager.releasePartition(ResultPartitionManager.java:80)
> 2020-05-06T19:13:12.4391118Z- locked <0x9d452b90> (a
> java.util.HashMap)
> 2020-05-06T19:13:12.4391597Zat
> org.apache.flink.runtime.io.network.NettyShuffleEnvironment.releasePartitionsLocally(NettyShuffleEnvironment.java:153)
> 2020-05-06T19:13:12.4392267Zat
> org.apache.flink.runtime.io.network.partition.TaskExecutorPartitionTrackerImpl.stopTrackingAndReleaseJobPartitions(TaskExecutorPartitionTrackerImpl.java:62)
> 2020-05-06T19:13:12.4392914Zat
> org.apache.flink.runtime.taskexecutor.TaskExecutor.releaseOrPromotePartitions(TaskExecutor.java:776)
> 2020-05-06T19:13:12.4393366Zat
> sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> 2020-05-06T19:13:12.4393813Zat
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-05-06T19:13:12.4394257Zat
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-05-06T19:13:12.4394693Zat
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:279)
> 2020-05-06T19:13:12.4395202Zat
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:199)
> 2020-05-06T19:13:12.4395686Zat
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
> 2020-05-06T19:13:12.4396165Zat
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor$$Lambda$72/775020844.apply(Unknown
> Source)
> 2020-05-06T19:13:12.4396606Zat
> akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
> 2020-05-06T19:13:12.4397015Zat
> akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
> 2020-05-06T19:13:12.4397447Zat
> scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
> 2020-05-06T19:13:12.4397874Zat
> akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
> 2020-05-06T19:13:12.4398414Zat
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
> 2020-05-06T19:13:12.4398879Zat
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> 2020-05-06T19:13:12.4399321Zat
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> 2020-05-06T19:13:12.4399737Zat
> akka.actor.Actor$class.aroundReceive(Actor.scala:517)
> 2020-05-0
[jira] [Created] (FLINK-17796) Application mode should respect user specified classpath configuration
Yang Wang created FLINK-17796:
-
Summary: Application mode should respect user specified classpath
configuration
Key: FLINK-17796
URL: https://issues.apache.org/jira/browse/FLINK-17796
Project: Flink
Issue Type: Sub-task
Reporter: Yang Wang
Currently, when we deploy a Flink application cluster(e.g. Yarn, K8s),
{{-C/--classpath}} could not work properly. Since we override the
{{PipelineOptions.CLASSPATHS}} in {{YarnApplicationClusterEntrypoint}} and
{{KubernetesApplicationClusterEntrypoint}} . Then client updated configuration
will not take effect.
{code:java}
// code placeholder
./bin/flink run-application -p 5 -t yarn-application\
--classpath file:///flink/shared/dependency.jar ... \
-Dtaskmanager.memory.process.size=4096m \
examples/streaming/StateMachineExample.jar
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] GJL commented on pull request #12204: [FLINK-17777][tests] Set HADOOP_CLASSPATH for Mesos TaskManagers
GJL commented on pull request #12204: URL: https://github.com/apache/flink/pull/12204#issuecomment-630066941 I have to revert the `hotfix` commit. In standalone this happens ``` Trying to start actor system, external address 127.0.0.1:0, bind address 0.0.0.0:0. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] GJL edited a comment on pull request #12204: [FLINK-17777][tests] Set HADOOP_CLASSPATH for Mesos TaskManagers
GJL edited a comment on pull request #12204: URL: https://github.com/apache/flink/pull/12204#issuecomment-630066941 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-15813) Set default value of jobmanager.execution.failover-strategy to region
[
https://issues.apache.org/jira/browse/FLINK-15813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110088#comment-17110088
]
Zhu Zhu commented on FLINK-15813:
-
Sure. I will open a PR for it.
> Set default value of jobmanager.execution.failover-strategy to region
> -
>
> Key: FLINK-15813
> URL: https://issues.apache.org/jira/browse/FLINK-15813
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Till Rohrmann
>Assignee: Zhu Zhu
>Priority: Blocker
> Labels: usability
> Fix For: 1.11.0
>
>
> We should set the default value of {{jobmanager.execution.failover-strategy}}
> to {{region}}. This might require to adapt existing tests to make them pass.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] JingsongLi commented on pull request #12220: [FLINK-14255][hive] Integrate mapred record writer to hive streaming sink
JingsongLi commented on pull request #12220: URL: https://github.com/apache/flink/pull/12220#issuecomment-630068280 Previous abandoned PR: https://github.com/apache/flink/pull/12168 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] curcur commented on pull request #12219: [FLINK-15670][API] Provide a Kafka Source/Sink pair as KafkaShuffle
curcur commented on pull request #12219: URL: https://github.com/apache/flink/pull/12219#issuecomment-630068235 local azure is running here: https://dev.azure.com/mymeiyuan/Flink/_build/results?buildId=15&view=logs&j=1f3ed471-1849-5d3c-a34c-19792af4ad16 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-16383) KafkaProducerExactlyOnceITCase.testExactlyOnceRegularSink fails with "The producer has already been closed"
[
https://issues.apache.org/jira/browse/FLINK-16383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110089#comment-17110089
]
Jiangjie Qin commented on FLINK-16383:
--
[~AHeise] It seems the bug fix in [https://github.com/apache/flink/pull/12186]
was incomplete. We need to make sure all the methods called on the wrapped
{{StreamOperator}} are going through the {{StreamOperatorWrapper}}.
In this case, the test failure was because `notifyCheckpointComplete()` was
invoked directly on the wrapper operator instead of the wrapper. See:
https://github.com/apache/flink/blob/20c28ac77ecd6b8d11e38ed84c9a5c36317721f3/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/SubtaskCheckpointCoordinatorImpl.java#L178
> KafkaProducerExactlyOnceITCase.testExactlyOnceRegularSink fails with "The
> producer has already been closed"
> ---
>
> Key: FLINK-16383
> URL: https://issues.apache.org/jira/browse/FLINK-16383
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task, Tests
>Reporter: Robert Metzger
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.11.0
>
>
> Logs:
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=5779&view=logs&j=a54de925-e958-5e24-790a-3a6150eb72d8&t=24e561e9-4c8d-598d-a290-e6acce191345
> {code}
> 2020-03-01T01:06:57.4738418Z 01:06:57,473 [Source: Custom Source -> Map ->
> Sink: Unnamed (1/1)] INFO
> org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaInternalProducer
> [] - Flushing new partitions
> 2020-03-01T01:06:57.4739960Z 01:06:57,473 [FailingIdentityMapper Status
> Printer] INFO
> org.apache.flink.streaming.connectors.kafka.testutils.FailingIdentityMapper
> [] - > Failing mapper 0: count=680,
> totalCount=1000
> 2020-03-01T01:06:57.4909074Z
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> 2020-03-01T01:06:57.4910001Z at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:147)
> 2020-03-01T01:06:57.4911000Z at
> org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:648)
> 2020-03-01T01:06:57.4912078Z at
> org.apache.flink.streaming.util.TestStreamEnvironment.execute(TestStreamEnvironment.java:77)
> 2020-03-01T01:06:57.4913039Z at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1619)
> 2020-03-01T01:06:57.4914421Z at
> org.apache.flink.test.util.TestUtils.tryExecute(TestUtils.java:35)
> 2020-03-01T01:06:57.4915423Z at
> org.apache.flink.streaming.connectors.kafka.KafkaProducerTestBase.testExactlyOnce(KafkaProducerTestBase.java:370)
> 2020-03-01T01:06:57.4916483Z at
> org.apache.flink.streaming.connectors.kafka.KafkaProducerTestBase.testExactlyOnceRegularSink(KafkaProducerTestBase.java:309)
> 2020-03-01T01:06:57.4917305Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-03-01T01:06:57.4917982Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-03-01T01:06:57.4918769Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-03-01T01:06:57.4919477Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-03-01T01:06:57.4920156Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-03-01T01:06:57.4920995Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-03-01T01:06:57.4921927Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-03-01T01:06:57.4922728Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-03-01T01:06:57.4923428Z at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2020-03-01T01:06:57.4924048Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2020-03-01T01:06:57.4924779Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2020-03-01T01:06:57.4925528Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2020-03-01T01:06:57.4926318Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2020-03-01T01:06:57.4927214Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-03-01T01:06:57.4927872Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-03-01T01:06:57.4928587Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-03-01T01:06:57.4929289Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-
[jira] [Created] (FLINK-17797) Align the behavior between the new and legacy HBase table source
Jark Wu created FLINK-17797:
---
Summary: Align the behavior between the new and legacy HBase table
source
Key: FLINK-17797
URL: https://issues.apache.org/jira/browse/FLINK-17797
Project: Flink
Issue Type: Sub-task
Reporter: Jark Wu
The legacy HBase table source, i.e. {{HBaseTableSource}}, supports projection
push down. In order to make the user experience consistent. We should align the
behavior and add tests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
