[jira] [Commented] (FLINK-15533) Writing DataStream as text file fails due to output path already exists
[
https://issues.apache.org/jira/browse/FLINK-15533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013392#comment-17013392
]
Rui Li commented on FLINK-15533:
[~kkl0u] I just tried the latest branch and it fixes the issue, no matter
whether {{parallelism.default}} is set or not. Thanks.
> Writing DataStream as text file fails due to output path already exists
> ---
>
> Key: FLINK-15533
> URL: https://issues.apache.org/jira/browse/FLINK-15533
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission
>Affects Versions: 1.10.0
>Reporter: Rui Li
>Assignee: Kostas Kloudas
>Priority: Blocker
> Fix For: 1.10.0
>
>
> The following program reproduces the issue.
> {code}
> Configuration configuration = GlobalConfiguration.loadConfiguration();
> configuration.set(DeploymentOptions.TARGET, RemoteExecutor.NAME);
> StreamExecutionEnvironment streamEnv = new
> StreamExecutionEnvironment(configuration);
> DataStream dataStream = streamEnv.fromCollection(Arrays.asList(1,2,3));
> dataStream.writeAsText("hdfs://localhost:8020/tmp/output");
> streamEnv.execute();
> {code}
> The job will fail with the follow error, even though the output path doesn't
> exist before job submission:
> {noformat}
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.FileAlreadyExistsException):
> /tmp/output already exists as a directory
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729 ## CI report: * 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10833: [FLINK-15535][documentation] Add usage of ProcessFunctionTestHarnesses for testing documentation
flinkbot edited a comment on issue #10833: [FLINK-15535][documentation] Add usage of ProcessFunctionTestHarnesses for testing documentation URL: https://github.com/apache/flink/pull/10833#issuecomment-573278988 ## CI report: * f4701fe75def6307564a3b533c0bc165ae6b8bab Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143985226) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4265) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10833: [FLINK-15535][documentation] Add usage of ProcessFunctionTestHarnesses for testing documentation
flinkbot edited a comment on issue #10833: [FLINK-15535][documentation] Add usage of ProcessFunctionTestHarnesses for testing documentation URL: https://github.com/apache/flink/pull/10833#issuecomment-573278988 ## CI report: * f4701fe75def6307564a3b533c0bc165ae6b8bab Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/143985226) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4265) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729 ## CI report: * 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10833: [FLINK-15535][documentation] Add usage of ProcessFunctionTestHarnesses for testing documentation
flinkbot commented on issue #10833: [FLINK-15535][documentation] Add usage of ProcessFunctionTestHarnesses for testing documentation URL: https://github.com/apache/flink/pull/10833#issuecomment-573278988 ## CI report: * f4701fe75def6307564a3b533c0bc165ae6b8bab UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729 ## CI report: * 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15514) Bump Elasticsearch version from 6.3.1 to 6.8.6 for es6 connector
[ https://issues.apache.org/jira/browse/FLINK-15514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013357#comment-17013357 ] vinoyang commented on FLINK-15514: -- [~aljoscha] thanks for your opinion. I have created another for ES 7.x, the Jira ID is FLINK-15558. > Bump Elasticsearch version from 6.3.1 to 6.8.6 for es6 connector > > > Key: FLINK-15514 > URL: https://issues.apache.org/jira/browse/FLINK-15514 > Project: Flink > Issue Type: Wish > Components: Connectors / ElasticSearch >Reporter: vinoyang >Assignee: Aljoscha Krettek >Priority: Major > > Currently, Elasticsearch 6.x has released to > 6.8.6(https://www.elastic.co/guide/en/elasticsearch/reference/6.8/release-notes-6.8.6.html). > While es6 connector still tracks 6.3.1. It seems that the current version is > a little old. > I propose that we can upgrade to 6.8.6 for es6 connector. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15558) Bump Elasticsearch version from 7.3.2 to 7.5.1 for es7 connector
vinoyang created FLINK-15558: Summary: Bump Elasticsearch version from 7.3.2 to 7.5.1 for es7 connector Key: FLINK-15558 URL: https://issues.apache.org/jira/browse/FLINK-15558 Project: Flink Issue Type: Wish Components: Connectors / ElasticSearch Reporter: vinoyang It would be better to track the newest ES 7.x client version just like we have done for Kafka universal connector. Currently, the ES7 connector track version 7.3.2 and the latest ES 7.x version is 7.5.1. We can upgrade it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on issue #10833: [FLINK-15535][documentation] Add usage of ProcessFunctionTestHarnesses for testing documentation
flinkbot commented on issue #10833: [FLINK-15535][documentation] Add usage of ProcessFunctionTestHarnesses for testing documentation URL: https://github.com/apache/flink/pull/10833#issuecomment-573276966 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit f4701fe75def6307564a3b533c0bc165ae6b8bab (Sat Jan 11 03:41:31 UTC 2020) **Warnings:** * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-15535).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-15535) Add usage of ProcessFunctionTestHarnesses for testing documentation
[
https://issues.apache.org/jira/browse/FLINK-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-15535:
---
Labels: pull-request-available (was: )
> Add usage of ProcessFunctionTestHarnesses for testing documentation
> ---
>
> Key: FLINK-15535
> URL: https://issues.apache.org/jira/browse/FLINK-15535
> Project: Flink
> Issue Type: Wish
> Components: Documentation
>Reporter: vinoyang
>Priority: Major
> Labels: pull-request-available
>
> Recently, we added {{ProcessFunctionTestHarness}} for testing
> {{ProcessFunction}}. However, except {{ProcessFunctionTestHarnessesTest}} I
> can not find anything about this test harness in the master codebase.
> Considering {{ProcessFunction}} is the very important and frenquency-used UDF.
> I suggest that we could add a test example in the [testing
> documentation|https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/testing.html#integration-testing].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] yanghua opened a new pull request #10833: [FLINK-15535][documentation] Add usage of ProcessFunctionTestHarnesses for testing documentation
yanghua opened a new pull request #10833: [FLINK-15535][documentation] Add usage of ProcessFunctionTestHarnesses for testing documentation URL: https://github.com/apache/flink/pull/10833 ## What is the purpose of the change *This pull request adds usage of ProcessFunctionTestHarnesses for testing documentation* ## Brief change log - *Add usage of ProcessFunctionTestHarnesses for testing documentation* ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / **not documented**) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
flinkbot commented on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729 ## CI report: * 5eb1599945f1bee35c342f762cc25684013d2d83 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
flinkbot commented on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions URL: https://github.com/apache/flink/pull/10832#issuecomment-573275694 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit f327a0f2cf227e32c6c46fd0a698117d972a35eb (Sat Jan 11 03:22:23 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14163) Execution#producedPartitions is possibly not assigned when used
[
https://issues.apache.org/jira/browse/FLINK-14163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013353#comment-17013353
]
Yuan Mei commented on FLINK-14163:
--
PR: pull request #10832
> Execution#producedPartitions is possibly not assigned when used
> ---
>
> Key: FLINK-14163
> URL: https://issues.apache.org/jira/browse/FLINK-14163
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Zhu Zhu
>Assignee: Yuan Mei
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently {{Execution#producedPartitions}} is assigned after the partitions
> have completed the registration to shuffle master in
> {{Execution#registerProducedPartitions(...)}}.
> The partition registration is an async interface
> ({{ShuffleMaster#registerPartitionWithProducer(...)}}), so
> {{Execution#producedPartitions}} is possible[1] not set when used.
> Usages includes:
> 1. deploying this task, so that the task may be deployed without its result
> partitions assigned, and the job would hang. (DefaultScheduler issue only,
> since legacy scheduler handled this case)
> 2. generating input descriptors for downstream tasks:
> 3. retrieve {{ResultPartitionID}} for partition releasing:
> [1] If a user uses Flink default shuffle master {{NettyShuffleMaster}}, it is
> not problematic at the moment since it returns a completed future on
> registration, so that it would be a synchronized process. However, if users
> implement their own shuffle service in which the
> {{ShuffleMaster#registerPartitionWithProducer}} returns an pending future, it
> can be a problem. This is possible since customizable shuffle service is open
> to users since 1.9 (via config "shuffle-service-factory.class").
> To avoid issues to happen, we may either
> 1. fix all the usages of {{Execution#producedPartitions}} regarding the async
> assigning, or
> 2. change {{ShuffleMaster#registerPartitionWithProducer(...)}} to a sync
> interface
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14163) Execution#producedPartitions is possibly not assigned when used
[
https://issues.apache.org/jira/browse/FLINK-14163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-14163:
---
Labels: pull-request-available (was: )
> Execution#producedPartitions is possibly not assigned when used
> ---
>
> Key: FLINK-14163
> URL: https://issues.apache.org/jira/browse/FLINK-14163
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Zhu Zhu
>Assignee: Yuan Mei
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
>
> Currently {{Execution#producedPartitions}} is assigned after the partitions
> have completed the registration to shuffle master in
> {{Execution#registerProducedPartitions(...)}}.
> The partition registration is an async interface
> ({{ShuffleMaster#registerPartitionWithProducer(...)}}), so
> {{Execution#producedPartitions}} is possible[1] not set when used.
> Usages includes:
> 1. deploying this task, so that the task may be deployed without its result
> partitions assigned, and the job would hang. (DefaultScheduler issue only,
> since legacy scheduler handled this case)
> 2. generating input descriptors for downstream tasks:
> 3. retrieve {{ResultPartitionID}} for partition releasing:
> [1] If a user uses Flink default shuffle master {{NettyShuffleMaster}}, it is
> not problematic at the moment since it returns a completed future on
> registration, so that it would be a synchronized process. However, if users
> implement their own shuffle service in which the
> {{ShuffleMaster#registerPartitionWithProducer}} returns an pending future, it
> can be a problem. This is possible since customizable shuffle service is open
> to users since 1.9 (via config "shuffle-service-factory.class").
> To avoid issues to happen, we may either
> 1. fix all the usages of {{Execution#producedPartitions}} regarding the async
> assigning, or
> 2. change {{ShuffleMaster#registerPartitionWithProducer(...)}} to a sync
> interface
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] curcur opened a new pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
curcur opened a new pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions URL: https://github.com/apache/flink/pull/10832 ## What is the purpose of the change `Execution#producedPartitions` are registered through an asynchronous interface `ShuffleMaster#registerPartitionWithProducer` to `ShuffleMaster`, however they are not always accessed through callbacks. So, it is possible that `Execution#producedPartitions` have not been available yet when accessed (in `Execution#deploy` for example). Since the only implementation of `ShuffleMaster` is `NettyShuffleMaster`, which indeed registers producedPartition in a synchronous way, hence this pull request enforces synchronous registrations under an asynchronous interface as a temporary fix to prevent future misuses. ## Brief change log Make sure the future of registration is done immediately after registration. Throws IllegalStateException otherwise. ## Verifying this change unit test: ExecutionTest#testSynchronousRegisterProducedPartitions ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? JavaDocs This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15549) integer overflow in SpillingResettableMutableObjectIterator
[ https://issues.apache.org/jira/browse/FLINK-15549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013348#comment-17013348 ] caojian0613 commented on FLINK-15549: - We have fixed this problem and could someone assign this ticket to me? > integer overflow in SpillingResettableMutableObjectIterator > --- > > Key: FLINK-15549 > URL: https://issues.apache.org/jira/browse/FLINK-15549 > Project: Flink > Issue Type: Bug > Components: Runtime / Task >Affects Versions: 1.10.0 >Reporter: caojian0613 >Priority: Major > Labels: overflow > > The SpillingResettableMutableObjectIterator has a data overflow problem if > the number of elements in a single input exceeds Integer.MAX_VALUE. > The reason is inside the SpillingResettableMutableObjectIterator, it track > the total number of elements and the number of elements currently read with > two int type fileds (elementCount and currentElementNum), and if the number > of elements exceeds Integer.MAX_VALUE, it will overflow. > If there is an overflow, then in the next iteration, after reset the input , > the data will not be read or only part of the data will be read. > Therefore, we should changing the type of these two fields of > SpillingResettableIterator* from int to long, and we also need a pre-check > mechanism before such numerical. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] xintongsong commented on issue #10829: [FLINK-14853][docs] Use higher granularity units in generated docs for Duration & MemorySize if possible
xintongsong commented on issue #10829: [FLINK-14853][docs] Use higher granularity units in generated docs for Duration & MemorySize if possible URL: https://github.com/apache/flink/pull/10829#issuecomment-573274699 Thanks for pointing me to this, @zentol. Hi @dawidwys, thanks for the PR. I agree it would be nice to have higher granularity unit in the docs. However, `MemorySize#toString` does not only affect the docs, but also logs. There are cases that memory sizes are derived from a total size and a fraction (e.g. managed memory). In those cases, a memory size could be large but not integer with high granularity units. Therefore, I think it would make sense to print both the exact bytes for accuracy and the coarse high granularity value for readability (as in #10785). WDYT? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-15549) integer overflow in SpillingResettableMutableObjectIterator
[ https://issues.apache.org/jira/browse/FLINK-15549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caojian0613 updated FLINK-15549: Affects Version/s: (was: 1.9.1) (was: 1.9.0) 1.10.0 > integer overflow in SpillingResettableMutableObjectIterator > --- > > Key: FLINK-15549 > URL: https://issues.apache.org/jira/browse/FLINK-15549 > Project: Flink > Issue Type: Bug > Components: Runtime / Task >Affects Versions: 1.10.0 >Reporter: caojian0613 >Priority: Major > Labels: overflow > > The SpillingResettableMutableObjectIterator has a data overflow problem if > the number of elements in a single input exceeds Integer.MAX_VALUE. > The reason is inside the SpillingResettableMutableObjectIterator, it track > the total number of elements and the number of elements currently read with > two int type fileds (elementCount and currentElementNum), and if the number > of elements exceeds Integer.MAX_VALUE, it will overflow. > If there is an overflow, then in the next iteration, after reset the input , > the data will not be read or only part of the data will be read. > Therefore, we should changing the type of these two fields of > SpillingResettableIterator* from int to long, and we also need a pre-check > mechanism before such numerical. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9848: [FLINK-14332] [flink-metrics-signalfx] add flink-metrics-signalfx
flinkbot edited a comment on issue #9848: [FLINK-14332] [flink-metrics-signalfx] add flink-metrics-signalfx URL: https://github.com/apache/flink/pull/9848#issuecomment-538885706 ## CI report: * dd22d245973b8caf25f8f26a392a1ca95f863736 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/13062) * 6ae59be715e9454885ea94df650b61a5adc9d9f1 Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/131049194) * d0c22910828121d32e0a847ff33bd705fb9aae35 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/131050758) * 538f0528ff44a648a878e7c9eb5bac62b005c39c Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/131190746) * 62f76f23a0dae45d3efa9a773d37155f16565df8 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/131224128) * 3f43b51e9e9c8d3121786043901a4cbe3e4822a3 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/131246054) * 0c687c49cfa6d37560873461babdf460cee09303 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/131266147) * f92953f34e8c0bd59b684fd823b59cb13b68c689 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/132074402) * 542324b1fbe108ce517b3234c0a5b7867caf13f2 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/132618133) * a942e401d08ba8a5440c68bfdbb2872a8c8bde6a Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143979541) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4263) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13613) Update state migration tests for Flink 1.9
[
https://issues.apache.org/jira/browse/FLINK-13613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013338#comment-17013338
]
vinoyang commented on FLINK-13613:
--
bq. Not sure whether it makes sense to move all migration tests into the same
module.
What I mean about "centralized place" not always one module. It may be several
submodules belongs to one parent module named {{flink-migration-tests}} just
like those modules under {{flink-end-to-end-tests}}. So I said:
bq. However, It would also introduce another problem: there would be many new
modules.
It has some obvious pros:
* centralized management, currently, only little people know the context of
those migration tests, they only know more via those umbrella issues(e.g. this
one), every time when the community release major version, we must list all the
class names of migration tests;
* extract common logic and introduce common modules like what you said, the
centralized mechanism can make it easier;
* towards next plan (e.g. automatic testing), currently, these migration test
belongs to different modules(some of them are first-level, others are
second-level), unify them and reorganize them may make it easier;
> Update state migration tests for Flink 1.9
> --
>
> Key: FLINK-13613
> URL: https://issues.apache.org/jira/browse/FLINK-13613
> Project: Flink
> Issue Type: Task
> Components: Tests
>Affects Versions: 1.10.0
>Reporter: Till Rohrmann
>Assignee: Aljoscha Krettek
>Priority: Blocker
> Fix For: 1.10.0
>
>
> Once the Flink {{1.9.0}} release is out, we should update existing migration
> tests to cover restoring from {{1.9.0}} savepoints.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9848: [FLINK-14332] [flink-metrics-signalfx] add flink-metrics-signalfx
flinkbot edited a comment on issue #9848: [FLINK-14332] [flink-metrics-signalfx] add flink-metrics-signalfx URL: https://github.com/apache/flink/pull/9848#issuecomment-538885706 ## CI report: * dd22d245973b8caf25f8f26a392a1ca95f863736 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/13062) * 6ae59be715e9454885ea94df650b61a5adc9d9f1 Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/131049194) * d0c22910828121d32e0a847ff33bd705fb9aae35 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/131050758) * 538f0528ff44a648a878e7c9eb5bac62b005c39c Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/131190746) * 62f76f23a0dae45d3efa9a773d37155f16565df8 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/131224128) * 3f43b51e9e9c8d3121786043901a4cbe3e4822a3 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/131246054) * 0c687c49cfa6d37560873461babdf460cee09303 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/131266147) * f92953f34e8c0bd59b684fd823b59cb13b68c689 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/132074402) * 542324b1fbe108ce517b3234c0a5b7867caf13f2 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/132618133) * a942e401d08ba8a5440c68bfdbb2872a8c8bde6a Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143979541) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4263) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9848: [FLINK-14332] [flink-metrics-signalfx] add flink-metrics-signalfx
flinkbot edited a comment on issue #9848: [FLINK-14332] [flink-metrics-signalfx] add flink-metrics-signalfx URL: https://github.com/apache/flink/pull/9848#issuecomment-538885706 ## CI report: * dd22d245973b8caf25f8f26a392a1ca95f863736 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/13062) * 6ae59be715e9454885ea94df650b61a5adc9d9f1 Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/131049194) * d0c22910828121d32e0a847ff33bd705fb9aae35 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/131050758) * 538f0528ff44a648a878e7c9eb5bac62b005c39c Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/131190746) * 62f76f23a0dae45d3efa9a773d37155f16565df8 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/131224128) * 3f43b51e9e9c8d3121786043901a4cbe3e4822a3 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/131246054) * 0c687c49cfa6d37560873461babdf460cee09303 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/131266147) * f92953f34e8c0bd59b684fd823b59cb13b68c689 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/132074402) * 542324b1fbe108ce517b3234c0a5b7867caf13f2 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/132618133) * a942e401d08ba8a5440c68bfdbb2872a8c8bde6a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] lukess commented on issue #9848: [FLINK-14332] [flink-metrics-signalfx] add flink-metrics-signalfx
lukess commented on issue #9848: [FLINK-14332] [flink-metrics-signalfx] add flink-metrics-signalfx URL: https://github.com/apache/flink/pull/9848#issuecomment-573252493 @aljoscha I'm running the new SignalFXReporter in our environments. Current implementation: 1. current metric full name is from MetricGroup getScopeComponents(), and remove MetricGroup getAllVariables(), and finally append metricName from notifier. The getAllVariables() store as SFX dimensions that are similar to Graphite 1.1.x tags. 2. SFX AggregateMetricSender works as reporter to send protobuf data point to SFX backend 3. support Gauge, Meter and Counter 4. remove unnecessary tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10820: [FLINK-15512][statebackend] Refactor the mechanism to calculate the cache capacity shared among RocksDB instance(s)
flinkbot edited a comment on issue #10820: [FLINK-15512][statebackend] Refactor the mechanism to calculate the cache capacity shared among RocksDB instance(s) URL: https://github.com/apache/flink/pull/10820#issuecomment-572715611 ## CI report: * bfc6c4a4f7baa0bec85a9105b96014bb1fa5ce87 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143787485) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4241) * 64d8a9a6821c95f904ca567afd7a056be7f04f3a Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143932940) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4262) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-15557) Cannot connect to Azure Event Hub/Kafka since Jan 5th 2020. Kafka version issue
Chris created FLINK-15557: - Summary: Cannot connect to Azure Event Hub/Kafka since Jan 5th 2020. Kafka version issue Key: FLINK-15557 URL: https://issues.apache.org/jira/browse/FLINK-15557 Project: Flink Issue Type: Bug Components: Connectors / Kafka Environment: Java 8, Flink 1.9.1, Azure Event Hub Reporter: Chris As of Jan 5th we can no longer consume messages from azure event hub using Flink 1.9.1. I was able to fix the issue on several of our spring boot projects by upgrading to spring boot 2.2.2, kafka 2.3.1, and kafka-clients 2.3.1 2020-01-10 19:36:30,364 WARN org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=] Bootstrap broker *.servicebus.windows.net:9093 (id: -1 rack: null) disconnected -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10820: [FLINK-15512][statebackend] Refactor the mechanism to calculate the cache capacity shared among RocksDB instance(s)
flinkbot edited a comment on issue #10820: [FLINK-15512][statebackend] Refactor the mechanism to calculate the cache capacity shared among RocksDB instance(s) URL: https://github.com/apache/flink/pull/10820#issuecomment-572715611 ## CI report: * bfc6c4a4f7baa0bec85a9105b96014bb1fa5ce87 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143787485) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4241) * 64d8a9a6821c95f904ca567afd7a056be7f04f3a Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143932940) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4262) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10820: [FLINK-15512][statebackend] Refactor the mechanism to calculate the cache capacity shared among RocksDB instance(s)
flinkbot edited a comment on issue #10820: [FLINK-15512][statebackend] Refactor the mechanism to calculate the cache capacity shared among RocksDB instance(s) URL: https://github.com/apache/flink/pull/10820#issuecomment-572715611 ## CI report: * bfc6c4a4f7baa0bec85a9105b96014bb1fa5ce87 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143787485) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4241) * 64d8a9a6821c95f904ca567afd7a056be7f04f3a Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/143932940) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4262) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] tillrohrmann commented on issue #10682: [FLINK-15247][Runtime] Wait for all slots to be free before task executor services shutdown upon stopping
tillrohrmann commented on issue #10682: [FLINK-15247][Runtime] Wait for all slots to be free before task executor services shutdown upon stopping URL: https://github.com/apache/flink/pull/10682#issuecomment-573141077 @azagrebin is the PR ready for another round of review? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10665: [FLINK-15354] Start and stop minikube only in kubernetes related e2e tests
tillrohrmann commented on a change in pull request #10665: [FLINK-15354] Start
and stop minikube only in kubernetes related e2e tests
URL: https://github.com/apache/flink/pull/10665#discussion_r365354460
##
File path: flink-end-to-end-tests/test-scripts/common_kubernetes.sh
##
@@ -26,26 +26,56 @@ MINIKUBE_START_RETRIES=3
MINIKUBE_START_BACKOFF=5
RESULT_HASH="e682ec6622b5e83f2eb614617d5ab2cf"
+# If running tests on non-linux os, the kubectl and minikube should be
installed manually
+function setup_kubernetes_for_linux {
+# Download kubectl, which is a requirement for using minikube.
+if ! [ -x "$(command -v kubectl)" ]; then
+local version=$(curl -s
https://storage.googleapis.com/kubernetes-release/release/stable.txt)
+curl -Lo kubectl
https://storage.googleapis.com/kubernetes-release/release/$version/bin/linux/amd64/kubectl
&& \
+chmod +x kubectl && sudo mv kubectl /usr/local/bin/
+fi
+# Download minikube.
+if ! [ -x "$(command -v minikube)" ]; then
+curl -Lo minikube
https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 &&
\
+chmod +x minikube && sudo mv minikube /usr/local/bin/
+fi
+}
+
function check_kubernetes_status {
minikube status
return $?
}
function start_kubernetes_if_not_running {
if ! check_kubernetes_status; then
-minikube start
+start_command="minikube start"
+# We need sudo permission to set vm-driver to none in linux os.
+[[ "${OS_TYPE}" = "linux" ]] && start_command="sudo ${start_command}
--vm-driver=none"
Review comment:
Can we always assume that we have sudo rights? If not, then it might be
simpler to require that `minikube` and `kubectl` are installed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10665: [FLINK-15354] Start and stop minikube only in kubernetes related e2e tests
tillrohrmann commented on a change in pull request #10665: [FLINK-15354] Start
and stop minikube only in kubernetes related e2e tests
URL: https://github.com/apache/flink/pull/10665#discussion_r365356272
##
File path: flink-end-to-end-tests/test-scripts/test_kubernetes_session.sh
##
@@ -29,6 +29,7 @@ ARGS="--output ${OUTPUT_PATH}"
function cleanup {
kubectl delete service ${CLUSTER_ID}
kubectl delete clusterrolebinding ${CLUSTER_ROLE_BINDING}
+stop_kubernetes
Review comment:
This means that for every K8s test we start and stop `minikube`. This might
add some time.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10665: [FLINK-15354] Start and stop minikube only in kubernetes related e2e tests
tillrohrmann commented on a change in pull request #10665: [FLINK-15354] Start
and stop minikube only in kubernetes related e2e tests
URL: https://github.com/apache/flink/pull/10665#discussion_r365354011
##
File path: flink-end-to-end-tests/test-scripts/common_kubernetes.sh
##
@@ -26,26 +26,56 @@ MINIKUBE_START_RETRIES=3
MINIKUBE_START_BACKOFF=5
RESULT_HASH="e682ec6622b5e83f2eb614617d5ab2cf"
+# If running tests on non-linux os, the kubectl and minikube should be
installed manually
+function setup_kubernetes_for_linux {
+# Download kubectl, which is a requirement for using minikube.
+if ! [ -x "$(command -v kubectl)" ]; then
+local version=$(curl -s
https://storage.googleapis.com/kubernetes-release/release/stable.txt)
+curl -Lo kubectl
https://storage.googleapis.com/kubernetes-release/release/$version/bin/linux/amd64/kubectl
&& \
+chmod +x kubectl && sudo mv kubectl /usr/local/bin/
Review comment:
I think we cannot assume that we have sudo rights.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10665: [FLINK-15354] Start and stop minikube only in kubernetes related e2e tests
tillrohrmann commented on a change in pull request #10665: [FLINK-15354] Start
and stop minikube only in kubernetes related e2e tests
URL: https://github.com/apache/flink/pull/10665#discussion_r365354033
##
File path: flink-end-to-end-tests/test-scripts/common_kubernetes.sh
##
@@ -26,26 +26,56 @@ MINIKUBE_START_RETRIES=3
MINIKUBE_START_BACKOFF=5
RESULT_HASH="e682ec6622b5e83f2eb614617d5ab2cf"
+# If running tests on non-linux os, the kubectl and minikube should be
installed manually
+function setup_kubernetes_for_linux {
+# Download kubectl, which is a requirement for using minikube.
+if ! [ -x "$(command -v kubectl)" ]; then
+local version=$(curl -s
https://storage.googleapis.com/kubernetes-release/release/stable.txt)
+curl -Lo kubectl
https://storage.googleapis.com/kubernetes-release/release/$version/bin/linux/amd64/kubectl
&& \
+chmod +x kubectl && sudo mv kubectl /usr/local/bin/
+fi
+# Download minikube.
+if ! [ -x "$(command -v minikube)" ]; then
+curl -Lo minikube
https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 &&
\
+chmod +x minikube && sudo mv minikube /usr/local/bin/
Review comment:
Same here with `sudo`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10665: [FLINK-15354] Start and stop minikube only in kubernetes related e2e tests
tillrohrmann commented on a change in pull request #10665: [FLINK-15354] Start
and stop minikube only in kubernetes related e2e tests
URL: https://github.com/apache/flink/pull/10665#discussion_r365355152
##
File path: flink-end-to-end-tests/test-scripts/common_kubernetes.sh
##
@@ -26,26 +26,56 @@ MINIKUBE_START_RETRIES=3
MINIKUBE_START_BACKOFF=5
RESULT_HASH="e682ec6622b5e83f2eb614617d5ab2cf"
+# If running tests on non-linux os, the kubectl and minikube should be
installed manually
+function setup_kubernetes_for_linux {
+# Download kubectl, which is a requirement for using minikube.
+if ! [ -x "$(command -v kubectl)" ]; then
+local version=$(curl -s
https://storage.googleapis.com/kubernetes-release/release/stable.txt)
+curl -Lo kubectl
https://storage.googleapis.com/kubernetes-release/release/$version/bin/linux/amd64/kubectl
&& \
+chmod +x kubectl && sudo mv kubectl /usr/local/bin/
Review comment:
Ok I see that this was done before like this as well. Hence, there must be
sudo rights.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10665: [FLINK-15354] Start and stop minikube only in kubernetes related e2e tests
tillrohrmann commented on a change in pull request #10665: [FLINK-15354] Start
and stop minikube only in kubernetes related e2e tests
URL: https://github.com/apache/flink/pull/10665#discussion_r365355603
##
File path: flink-end-to-end-tests/test-scripts/test_kubernetes_embedded_job.sh
##
@@ -30,7 +30,7 @@ function cleanup {
kubectl delete job flink-job-cluster
kubectl delete service flink-job-cluster
kubectl delete deployment flink-task-manager
-rm -rf ${OUTPUT_VOLUME}
Review comment:
Does `stop_kubernetes` delete the `${OUTPUT_VOLUME}`? Or why is this no
longer needed?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10658: [FLINK-15359] Remove unused YarnConfigOptions, Tests, Docs
tillrohrmann commented on a change in pull request #10658: [FLINK-15359] Remove
unused YarnConfigOptions, Tests, Docs
URL: https://github.com/apache/flink/pull/10658#discussion_r365352138
##
File path:
flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java
##
@@ -434,17 +434,6 @@
@Deprecated
public static final String YARN_REALLOCATE_FAILED_CONTAINERS =
"yarn.reallocate-failed";
- /**
-* The maximum number of failed YARN containers before entirely stopping
-* the YARN session / job on YARN.
-*
-* By default, we take the number of initially requested containers.
-*
-* @deprecated in favor of {@code
YarnConfigOptions#MAX_FAILED_CONTAINERS}.
-*/
- @Deprecated
- public static final String YARN_MAX_FAILED_CONTAINERS =
"yarn.maximum-failed-containers";
-
Review comment:
Turning off the japicmp plugin check for `ConfigConstants` was a mistake.
One could introduce a special `ConfigOption` which throws an exception if
being used. That would have the same effect as breaking a program build after
upgrading.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-12122) Spread out tasks evenly across all available registered TaskManagers
[
https://issues.apache.org/jira/browse/FLINK-12122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013089#comment-17013089
]
Till Rohrmann commented on FLINK-12122:
---
[~ethanli] there are no plan to backport this feature to {{1.8}}.
[~liuyufei] the feature only guarantees spreading out tasks across the set of
TMs which are registered at the time of scheduling. Hence, when you are using
the active Yarn mode and submit the first job, then there won't be any TMs
registered. Consequently, Flink will allocate the first container, fill it up
and then only allocate a new container. However, if you start Flink in
standalone mode or after your first job finishes on Yarn there are still some
TMs registered, then the next job would be spread out.
[~shazeline] there is currently no random distribution mode. When activating
{{cluster.evenly-spread-out-slots}}, Flink will try to allocate slots from the
TMs so that they are evenly utilized. Additionally, Flink will try to spread
out the subtasks belonging to the same operator equally across the available
slots. Hence, Flink should be able to spread your source tasks out across the
available TMs.
> Spread out tasks evenly across all available registered TaskManagers
>
>
> Key: FLINK-12122
> URL: https://issues.apache.org/jira/browse/FLINK-12122
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.6.4, 1.7.2, 1.8.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.9.2, 1.10.0
>
> Attachments: image-2019-05-21-12-28-29-538.png,
> image-2019-05-21-13-02-50-251.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> With Flip-6, we changed the default behaviour how slots are assigned to
> {{TaskManages}}. Instead of evenly spreading it out over all registered
> {{TaskManagers}}, we randomly pick slots from {{TaskManagers}} with a
> tendency to first fill up a TM before using another one. This is a regression
> wrt the pre Flip-6 code.
> I suggest to change the behaviour so that we try to evenly distribute slots
> across all available {{TaskManagers}} by considering how many of their slots
> are already allocated.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10820: [FLINK-15512][statebackend] Refactor the mechanism to calculate the cache capacity shared among RocksDB instance(s)
flinkbot edited a comment on issue #10820: [FLINK-15512][statebackend] Refactor the mechanism to calculate the cache capacity shared among RocksDB instance(s) URL: https://github.com/apache/flink/pull/10820#issuecomment-572715611 ## CI report: * bfc6c4a4f7baa0bec85a9105b96014bb1fa5ce87 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143787485) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4241) * 64d8a9a6821c95f904ca567afd7a056be7f04f3a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15527) can not control the number of container on yarn single job module
[
https://issues.apache.org/jira/browse/FLINK-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013086#comment-17013086
]
Till Rohrmann commented on FLINK-15527:
---
Isn't the problem that the Hive component does some parallelism inference which
ends up to set the source parallelism to something quite large? The runtime
will just do what the JobGraph tells it to do.
I agree that it could be helpful to limit the number of containers when running
Flink with an active RM. I imagine something like {{--min-task-executors=10
--max-task-executors=20}} for example. But this should not have solved problem
since Flink weren't able to run a job with a parallelism exceeding {{20 *
number_of_slots}}.
> can not control the number of container on yarn single job module
> -
>
> Key: FLINK-15527
> URL: https://issues.apache.org/jira/browse/FLINK-15527
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.10.0
>Reporter: chenchencc
>Priority: Major
> Fix For: 1.10.0
>
> Attachments: flink-conf.yaml, image-2020-01-09-14-30-46-973.png,
> yarn_application.png
>
>
> when run yarn single job run many container but paralism set 4
> *scripts:*
> ./bin/flink run -m yarn-cluster -ys 3 -p 4 -yjm 1024m -ytm 4096m -yqu bi -c
> com.cc.test.HiveTest2 ./cc_jars/hive-1.0-SNAPSHOT.jar 11.txt test61 6
> _notes_: in 1.9.1 has cli paramter -yn to control the number of containers
> and in 1.10 remove it
> *result:*
> the number of containers is 500+
>
> *code use:*
> query the table and save it to the hdfs text
>
> the storge of table is 200g+
>
>
>
>
> *code:*
> com.cc.test.HiveTest2
> public static void main(String[] args) throws Exception
> { EnvironmentSettings settings =
> EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build();
> StreamExecutionEnvironment settings2 =
> StreamExecutionEnvironment.getExecutionEnvironment();
> settings2.setParallelism(Integer.valueOf(args[2]));
> StreamTableEnvironment tableEnv = StreamTableEnvironment.create(settings2,
> settings); String name = "myhive"; String defaultDatabase = "test"; String
> hiveConfDir = "/etc/hive/conf"; String version = "1.2.1"; // or 1.2.1 2.3.4
> HiveCatalog hive = new HiveCatalog(name, defaultDatabase, hiveConfDir,
> version); tableEnv.registerCatalog("myhive", hive);
> tableEnv.useCatalog("myhive"); tableEnv.listTables(); Table table =
> tableEnv.sqlQuery("select id from orderparent_test2 where id =
> 'A21204170176'"); tableEnv.toAppendStream(table, Row.class).print();
> tableEnv.toAppendStream(table, Row.class)
> .writeAsText("hdfs:///user/chenchao1/"+ args[0],
> FileSystem.WriteMode.OVERWRITE); tableEnv.execute(args[1]); }
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-13613) Update state migration tests for Flink 1.9
[
https://issues.apache.org/jira/browse/FLINK-13613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013076#comment-17013076
]
Till Rohrmann commented on FLINK-13613:
---
Not sure whether it makes sense to move all migration tests into the same
module. We are testing different components like the Kafka and Kinesis
connector. I think the migration tests belong to the respective component
module. If there is common logic which can be shared by the different tests,
then we should introduce something like a {{migration-test-commons}} module
where we add the common logic and then share it by depending on it.
> Update state migration tests for Flink 1.9
> --
>
> Key: FLINK-13613
> URL: https://issues.apache.org/jira/browse/FLINK-13613
> Project: Flink
> Issue Type: Task
> Components: Tests
>Affects Versions: 1.10.0
>Reporter: Till Rohrmann
>Assignee: Aljoscha Krettek
>Priority: Blocker
> Fix For: 1.10.0
>
>
> Once the Flink {{1.9.0}} release is out, we should update existing migration
> tests to cover restoring from {{1.9.0}} savepoints.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15529) Update upgrade compatibility table (docs/ops/upgrading.md) for 1.10.0
[ https://issues.apache.org/jira/browse/FLINK-15529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013070#comment-17013070 ] Till Rohrmann commented on FLINK-15529: --- Sounds good to me [~liyu]. Go ahead. > Update upgrade compatibility table (docs/ops/upgrading.md) for 1.10.0 > - > > Key: FLINK-15529 > URL: https://issues.apache.org/jira/browse/FLINK-15529 > Project: Flink > Issue Type: Task > Components: Documentation >Affects Versions: 1.10.0 >Reporter: Yu Li >Assignee: Yu Li >Priority: Blocker > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Update upgrade compatibility table (docs/ops/upgrading.md) for 1.10.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15504) Flink suppresses stdout/stderr during JobGraph generation from JAR
[
https://issues.apache.org/jira/browse/FLINK-15504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013068#comment-17013068
]
Maximilian Michels commented on FLINK-15504:
Backporting this also to {{release-1.10}}.
> Flink suppresses stdout/stderr during JobGraph generation from JAR
> --
>
> Key: FLINK-15504
> URL: https://issues.apache.org/jira/browse/FLINK-15504
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Flink uses the {{OptimizerPlanEnvironment}} which replaces stdout/stderr
> during job graph creation. This was intended only for previewing the plan,
> but other parts of Flink, e.g. the Rest API have started to use this code as
> well.
> It may be acceptable to suppress output during plan preview, but we should
> not suppress the output during execution, e.g. via the REST API.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15504) Flink suppresses stdout/stderr during JobGraph generation from JAR
[
https://issues.apache.org/jira/browse/FLINK-15504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maximilian Michels updated FLINK-15504:
---
Fix Version/s: (was: 1.10.0)
1.11.0
> Flink suppresses stdout/stderr during JobGraph generation from JAR
> --
>
> Key: FLINK-15504
> URL: https://issues.apache.org/jira/browse/FLINK-15504
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Flink uses the {{OptimizerPlanEnvironment}} which replaces stdout/stderr
> during job graph creation. This was intended only for previewing the plan,
> but other parts of Flink, e.g. the Rest API have started to use this code as
> well.
> It may be acceptable to suppress output during plan preview, but we should
> not suppress the output during execution, e.g. via the REST API.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-15504) Flink suppresses stdout/stderr during JobGraph generation from JAR
[
https://issues.apache.org/jira/browse/FLINK-15504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maximilian Michels closed FLINK-15504.
--
Fix Version/s: 1.10.0
Release Note: Allow outputting to stdout/stderr during JobGraph creation
from JAR
Resolution: Fixed
> Flink suppresses stdout/stderr during JobGraph generation from JAR
> --
>
> Key: FLINK-15504
> URL: https://issues.apache.org/jira/browse/FLINK-15504
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Flink uses the {{OptimizerPlanEnvironment}} which replaces stdout/stderr
> during job graph creation. This was intended only for previewing the plan,
> but other parts of Flink, e.g. the Rest API have started to use this code as
> well.
> It may be acceptable to suppress output during plan preview, but we should
> not suppress the output during execution, e.g. via the REST API.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] mxm merged pull request #10811: [FLINK-15504] Allow output to stdout/stderr during execution of PackagedProgram
mxm merged pull request #10811: [FLINK-15504] Allow output to stdout/stderr during execution of PackagedProgram URL: https://github.com/apache/flink/pull/10811 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10762: [FLINK-15115][kafka] Drop Kafka 0.8/0.9
tillrohrmann commented on a change in pull request #10762: [FLINK-15115][kafka]
Drop Kafka 0.8/0.9
URL: https://github.com/apache/flink/pull/10762#discussion_r365323005
##
File path:
flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/table/descriptors/KafkaValidator.java
##
@@ -38,7 +38,6 @@
public class KafkaValidator extends ConnectorDescriptorValidator {
public static final String CONNECTOR_TYPE_VALUE_KAFKA = "kafka";
- public static final String CONNECTOR_VERSION_VALUE_08 = "0.8";
Review comment:
`KafkaShortRetentionTestBase.java` contains in line 253 Kafka 0.8 specific
code. In line 254 the same class contains Kafka 0.9 specific code.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10762: [FLINK-15115][kafka] Drop Kafka 0.8/0.9
tillrohrmann commented on a change in pull request #10762: [FLINK-15115][kafka]
Drop Kafka 0.8/0.9
URL: https://github.com/apache/flink/pull/10762#discussion_r365321100
##
File path: docs/dev/connectors/kafka.zh.md
##
@@ -161,22 +151,18 @@ Flink 的 Kafka consumer 称为 `FlinkKafkaConsumer08`（或适用于
Kafka 0.9.
{% highlight java %}
Properties properties = new Properties();
properties.setProperty("bootstrap.servers", "localhost:9092");
-// 仅 Kafka 0.8 需要
-properties.setProperty("zookeeper.connect", "localhost:2181");
properties.setProperty("group.id", "test");
DataStream stream = env
- .addSource(new FlinkKafkaConsumer08<>("topic", new SimpleStringSchema(),
properties));
+ .addSource(new FlinkKafkaConsumer09<>("topic", new SimpleStringSchema(),
properties));
{% endhighlight %}
{% highlight scala %}
val properties = new Properties()
properties.setProperty("bootstrap.servers", "localhost:9092")
-// 仅 Kafka 0.8 需要
-properties.setProperty("zookeeper.connect", "localhost:2181")
properties.setProperty("group.id", "test")
stream = env
-.addSource(new FlinkKafkaConsumer08[String]("topic", new
SimpleStringSchema(), properties))
+.addSource(new FlinkKafkaConsumer09[String]("topic", new
SimpleStringSchema(), properties))
Review comment:
`docs/dev/datastream_api.md` and `docs/dev/datastream_api.zh.md` still
contain a reference to `FlinkKafkaConsumer08`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10762: [FLINK-15115][kafka] Drop Kafka 0.8/0.9
tillrohrmann commented on a change in pull request #10762: [FLINK-15115][kafka]
Drop Kafka 0.8/0.9
URL: https://github.com/apache/flink/pull/10762#discussion_r365328481
##
File path:
flink-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer010.java
##
@@ -173,7 +205,38 @@ public FlinkKafkaConsumer010(Pattern subscriptionPattern,
DeserializationSchema<
*/
@PublicEvolving
public FlinkKafkaConsumer010(Pattern subscriptionPattern,
KafkaDeserializationSchema deserializer, Properties props) {
- super(subscriptionPattern, deserializer, props);
+ this(null, subscriptionPattern, deserializer, props);
Review comment:
Can we avoid `null` and instead pass in an empty collection? If not, then
let's add `@Nullable` annotation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10762: [FLINK-15115][kafka] Drop Kafka 0.8/0.9
tillrohrmann commented on a change in pull request #10762: [FLINK-15115][kafka]
Drop Kafka 0.8/0.9
URL: https://github.com/apache/flink/pull/10762#discussion_r365322652
##
File path:
flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/table/descriptors/KafkaValidator.java
##
@@ -38,7 +38,6 @@
public class KafkaValidator extends ConnectorDescriptorValidator {
public static final String CONNECTOR_TYPE_VALUE_KAFKA = "kafka";
- public static final String CONNECTOR_VERSION_VALUE_08 = "0.8";
Review comment:
`ConnectorDescriptorValidator.java` contains a reference to Kafka 0.8 in
line 47.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10762: [FLINK-15115][kafka] Drop Kafka 0.8/0.9
tillrohrmann commented on a change in pull request #10762: [FLINK-15115][kafka]
Drop Kafka 0.8/0.9
URL: https://github.com/apache/flink/pull/10762#discussion_r365318969
##
File path:
flink-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThreadTest.java
##
@@ -875,109 +878,194 @@ void
reassignPartitions(List> newPartit
}
@SuppressWarnings("unchecked")
- private static KafkaConsumer createMockConsumer(
+ private static TestConsumer createMockConsumer(
final Map
mockConsumerAssignmentAndPosition,
final Map mockRetrievedPositions,
final boolean earlyWakeup,
final OneShotLatch midAssignmentLatch,
final OneShotLatch continueAssignmentLatch) {
- final KafkaConsumer mockConsumer =
mock(KafkaConsumer.class);
+ return new TestConsumer(mockConsumerAssignmentAndPosition,
mockRetrievedPositions, earlyWakeup, midAssignmentLatch,
continueAssignmentLatch);
+ }
- when(mockConsumer.assignment()).thenAnswer(new Answer()
{
- @Override
- public Object answer(InvocationOnMock invocationOnMock)
throws Throwable {
- if (midAssignmentLatch != null) {
- midAssignmentLatch.trigger();
- }
+ private static class TestConsumer implements Consumer {
+ private final Map
mockConsumerAssignmentAndPosition;
+ private final Map mockRetrievedPositions;
+ private final boolean earlyWakeup;
+ private final OneShotLatch midAssignmentLatch;
+ private final OneShotLatch continueAssignmentLatch;
+
+ private int numWakeupCalls = 0;
+
+ private TestConsumer(Map
mockConsumerAssignmentAndPosition, Map
mockRetrievedPositions, boolean earlyWakeup, OneShotLatch midAssignmentLatch,
OneShotLatch continueAssignmentLatch) {
+ this.mockConsumerAssignmentAndPosition =
mockConsumerAssignmentAndPosition;
+ this.mockRetrievedPositions = mockRetrievedPositions;
+ this.earlyWakeup = earlyWakeup;
+ this.midAssignmentLatch = midAssignmentLatch;
+ this.continueAssignmentLatch = continueAssignmentLatch;
+ }
- if (continueAssignmentLatch != null) {
+ @Override
+ public Set assignment() {
+ if (midAssignmentLatch != null) {
+ midAssignmentLatch.trigger();
+ }
+
+ if (continueAssignmentLatch != null) {
+ try {
continueAssignmentLatch.await();
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
}
- return
mockConsumerAssignmentAndPosition.keySet();
}
- });
+ return mockConsumerAssignmentAndPosition.keySet();
+ }
-
when(mockConsumer.poll(anyLong())).thenReturn(mock(ConsumerRecords.class));
+ @Override
+ public Set subscription() {
+ return null;
+ }
- if (!earlyWakeup) {
-
when(mockConsumer.position(any(TopicPartition.class))).thenAnswer(new
Answer() {
- @Override
- public Object answer(InvocationOnMock
invocationOnMock) throws Throwable {
- return
mockConsumerAssignmentAndPosition.get(invocationOnMock.getArgument(0));
- }
- });
- } else {
-
when(mockConsumer.position(any(TopicPartition.class))).thenThrow(new
WakeupException());
+ @Override
+ public void subscribe(List list) {
}
- doAnswer(new Answer() {
- @Override
- public Object answer(InvocationOnMock invocationOnMock)
throws Throwable {
- mockConsumerAssignmentAndPosition.clear();
+ @Override
+ public void subscribe(List list,
ConsumerRebalanceListener consumerRebalanceListener) {
+ }
- List assignedPartitions =
invocationOnMock.getArgument(0);
- for (TopicPartition assigned :
assignedPartitions) {
-
[GitHub] [flink] tillrohrmann commented on a change in pull request #10762: [FLINK-15115][kafka] Drop Kafka 0.8/0.9
tillrohrmann commented on a change in pull request #10762: [FLINK-15115][kafka]
Drop Kafka 0.8/0.9
URL: https://github.com/apache/flink/pull/10762#discussion_r365331071
##
File path:
flink-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThreadTest.java
##
@@ -1047,17 +1047,41 @@ public OffsetAndMetadata committed(TopicPartition
topicPartition) {
}
@Override
- public void pause(TopicPartition... topicPartitions) {
+ public Set paused() {
+ return null;
+ }
+
+ @Override
+ public void pause(Collection collection) {
+ }
+
+ @Override
+ public void resume(Collection collection) {
}
@Override
- public void resume(TopicPartition... topicPartitions) {
+ public Map
offsetsForTimes(Map map) {
+ return null;
Review comment:
I would suggest to either fail or to return an empty map but not `null`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10762: [FLINK-15115][kafka] Drop Kafka 0.8/0.9
tillrohrmann commented on a change in pull request #10762: [FLINK-15115][kafka]
Drop Kafka 0.8/0.9
URL: https://github.com/apache/flink/pull/10762#discussion_r365328689
##
File path:
flink-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer010.java
##
@@ -130,7 +162,7 @@ public FlinkKafkaConsumer010(List topics,
DeserializationSchema deser
* The properties that are used to configure both the fetcher
and the offset handler.
*/
public FlinkKafkaConsumer010(List topics,
KafkaDeserializationSchema deserializer, Properties props) {
- super(topics, deserializer, props);
+ this(topics, null, deserializer, props);
Review comment:
Same here with the `null` value for `Pattern`. I think it would be good to
avoid it if possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10762: [FLINK-15115][kafka] Drop Kafka 0.8/0.9
tillrohrmann commented on a change in pull request #10762: [FLINK-15115][kafka]
Drop Kafka 0.8/0.9
URL: https://github.com/apache/flink/pull/10762#discussion_r365318960
##
File path:
flink-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThreadTest.java
##
@@ -875,109 +878,194 @@ void
reassignPartitions(List> newPartit
}
@SuppressWarnings("unchecked")
- private static KafkaConsumer createMockConsumer(
+ private static TestConsumer createMockConsumer(
final Map
mockConsumerAssignmentAndPosition,
final Map mockRetrievedPositions,
final boolean earlyWakeup,
final OneShotLatch midAssignmentLatch,
final OneShotLatch continueAssignmentLatch) {
- final KafkaConsumer mockConsumer =
mock(KafkaConsumer.class);
+ return new TestConsumer(mockConsumerAssignmentAndPosition,
mockRetrievedPositions, earlyWakeup, midAssignmentLatch,
continueAssignmentLatch);
+ }
- when(mockConsumer.assignment()).thenAnswer(new Answer()
{
- @Override
- public Object answer(InvocationOnMock invocationOnMock)
throws Throwable {
- if (midAssignmentLatch != null) {
- midAssignmentLatch.trigger();
- }
+ private static class TestConsumer implements Consumer {
+ private final Map
mockConsumerAssignmentAndPosition;
+ private final Map mockRetrievedPositions;
+ private final boolean earlyWakeup;
+ private final OneShotLatch midAssignmentLatch;
+ private final OneShotLatch continueAssignmentLatch;
+
+ private int numWakeupCalls = 0;
+
+ private TestConsumer(Map
mockConsumerAssignmentAndPosition, Map
mockRetrievedPositions, boolean earlyWakeup, OneShotLatch midAssignmentLatch,
OneShotLatch continueAssignmentLatch) {
+ this.mockConsumerAssignmentAndPosition =
mockConsumerAssignmentAndPosition;
+ this.mockRetrievedPositions = mockRetrievedPositions;
+ this.earlyWakeup = earlyWakeup;
+ this.midAssignmentLatch = midAssignmentLatch;
+ this.continueAssignmentLatch = continueAssignmentLatch;
+ }
- if (continueAssignmentLatch != null) {
+ @Override
+ public Set assignment() {
+ if (midAssignmentLatch != null) {
+ midAssignmentLatch.trigger();
+ }
+
+ if (continueAssignmentLatch != null) {
+ try {
continueAssignmentLatch.await();
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
}
- return
mockConsumerAssignmentAndPosition.keySet();
}
- });
+ return mockConsumerAssignmentAndPosition.keySet();
+ }
-
when(mockConsumer.poll(anyLong())).thenReturn(mock(ConsumerRecords.class));
+ @Override
+ public Set subscription() {
+ return null;
+ }
- if (!earlyWakeup) {
-
when(mockConsumer.position(any(TopicPartition.class))).thenAnswer(new
Answer() {
- @Override
- public Object answer(InvocationOnMock
invocationOnMock) throws Throwable {
- return
mockConsumerAssignmentAndPosition.get(invocationOnMock.getArgument(0));
- }
- });
- } else {
-
when(mockConsumer.position(any(TopicPartition.class))).thenThrow(new
WakeupException());
+ @Override
+ public void subscribe(List list) {
}
- doAnswer(new Answer() {
- @Override
- public Object answer(InvocationOnMock invocationOnMock)
throws Throwable {
- mockConsumerAssignmentAndPosition.clear();
+ @Override
+ public void subscribe(List list,
ConsumerRebalanceListener consumerRebalanceListener) {
+ }
- List assignedPartitions =
invocationOnMock.getArgument(0);
- for (TopicPartition assigned :
assignedPartitions) {
-
[GitHub] [flink] tillrohrmann commented on a change in pull request #10762: [FLINK-15115][kafka] Drop Kafka 0.8/0.9
tillrohrmann commented on a change in pull request #10762: [FLINK-15115][kafka] Drop Kafka 0.8/0.9 URL: https://github.com/apache/flink/pull/10762#discussion_r365322000 ## File path: tools/travis/stage.sh ## @@ -128,11 +128,6 @@ flink-connectors/flink-sql-connector-kafka," MODULES_TESTS="\ flink-tests" -# we can only build the Kafka 0.8 connector when building for Scala 2.11 -if [[ $PROFILE == *"scala-2.11"* ]]; then - MODULES_CONNECTORS="$MODULES_CONNECTORS,flink-connectors/flink-connector-kafka-0.8" -fi - Review comment: `flink-runtime/pom.xml` contains reference to Kafka 0.8 in line 562. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10762: [FLINK-15115][kafka] Drop Kafka 0.8/0.9
tillrohrmann commented on a change in pull request #10762: [FLINK-15115][kafka]
Drop Kafka 0.8/0.9
URL: https://github.com/apache/flink/pull/10762#discussion_r365327750
##
File path: flink-connectors/flink-connector-kafka-0.10/pom.xml
##
@@ -46,7 +46,7 @@ under the License.
org.apache.flink
-
flink-connector-kafka-0.9_${scala.binary.version}
+
flink-connector-kafka-base_${scala.binary.version}
Review comment:
In line 117, there is still a dependency to `flink-connector-kafka-0.9`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10762: [FLINK-15115][kafka] Drop Kafka 0.8/0.9
tillrohrmann commented on a change in pull request #10762: [FLINK-15115][kafka]
Drop Kafka 0.8/0.9
URL: https://github.com/apache/flink/pull/10762#discussion_r365326108
##
File path: docs/dev/connectors/kafka.md
##
@@ -263,7 +255,7 @@ Example:
{% highlight java %}
final StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
-FlinkKafkaConsumer09 myConsumer = new FlinkKafkaConsumer09<>(...);
Review comment:
`docs/dev/event_time.md`, `docs/dev/event_time.zh.md`,
`docs/dev/event_timestamps_watermarks.md` and
`docs/dev/event_timestamps_watermarks.zh.md` contains a reference to
`FlinkKafkaConsumer09`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10762: [FLINK-15115][kafka] Drop Kafka 0.8/0.9
tillrohrmann commented on a change in pull request #10762: [FLINK-15115][kafka]
Drop Kafka 0.8/0.9
URL: https://github.com/apache/flink/pull/10762#discussion_r365331143
##
File path:
flink-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThreadTest.java
##
@@ -1047,17 +1047,41 @@ public OffsetAndMetadata committed(TopicPartition
topicPartition) {
}
@Override
- public void pause(TopicPartition... topicPartitions) {
+ public Set paused() {
+ return null;
+ }
+
+ @Override
+ public void pause(Collection collection) {
+ }
+
+ @Override
+ public void resume(Collection collection) {
}
@Override
- public void resume(TopicPartition... topicPartitions) {
+ public Map
offsetsForTimes(Map map) {
+ return null;
+ }
+
+ @Override
+ public Map
beginningOffsets(Collection collection) {
+ return null;
+ }
+
+ @Override
+ public Map
endOffsets(Collection collection) {
+ return null;
Review comment:
Same here with `null`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] GJL commented on issue #10814: [FLINK-15434][Tests]Fix unstable tests in JobMasterTest
GJL commented on issue #10814: [FLINK-15434][Tests]Fix unstable tests in JobMasterTest URL: https://github.com/apache/flink/pull/10814#issuecomment-573109310 Sorry, didn't have time to look at your PR yet but I will in the next days. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10831: [FLINK-13465][legal] Update javassist licensing
flinkbot edited a comment on issue #10831: [FLINK-13465][legal] Update javassist licensing URL: https://github.com/apache/flink/pull/10831#issuecomment-573065869 ## CI report: * d8c12a81e919646eb5cda292cfd5a70744c6c7ed Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143905039) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4261) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] GJL commented on a change in pull request #10804: [FLINK-15488] Obtain the JVM and TM param correctly
GJL commented on a change in pull request #10804: [FLINK-15488] Obtain the JVM
and TM param correctly
URL: https://github.com/apache/flink/pull/10804#discussion_r365322482
##
File path:
flink-dist/src/test/java/org/apache/flink/dist/BashJavaUtilsITCase.java
##
@@ -34,21 +36,30 @@
private static final String RUN_BASH_JAVA_UTILS_CMD_SCRIPT =
"src/test/bin/runBashJavaUtilsCmd.sh";
+ /**
+* Executes the given shell script wrapper and returns the last line.
+*/
+ private String executeScriptAndFetchLastLine(final String command)
throws IOException {
+ String[] commands = {RUN_BASH_JAVA_UTILS_CMD_SCRIPT, command};
+ String[] lines =
executeScript(commands).split(System.lineSeparator());
+ if (lines.length == 0) {
+ return "";
+ } else {
+ return lines[lines.length - 1];
+ }
+ }
+
@Test
public void testGetTmResourceDynamicConfigs() throws Exception {
- String[] command = {RUN_BASH_JAVA_UTILS_CMD_SCRIPT,
-
BashJavaUtils.Command.GET_TM_RESOURCE_DYNAMIC_CONFIGS.toString()};
- String result = executeScript(command);
+ String result =
executeScriptAndFetchLastLine(BashJavaUtils.Command.GET_TM_RESOURCE_DYNAMIC_CONFIGS.toString());
assertNotNull(result);
Review comment:
Would it be possible to invoke `BashJavaUtils#main` directly, and use
`System.setOut` to redirect the output to stdout to a different stream?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Comment Edited] (FLINK-15368) Add end-to-end test for controlling RocksDB memory usage
[
https://issues.apache.org/jira/browse/FLINK-15368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17007221#comment-17007221
]
Yu Li edited comment on FLINK-15368 at 1/10/20 4:26 PM:
Thanks for the efforts and update [~yunta]! Below are some comments and
suggestions according to the testing output and investigation result:
# We should write explicitly in our document not to set
{{optimizeForPointLookup}} when enabling RocksDB memory control.
# We cannot rely on {{strict_capacity_limit}} until
[RocksDB#6247|https://github.com/facebook/rocksdb/issues/6247] is resolved. In
another word, since the issue on {{strict_capacity_limit}} could hardly be
resolved soon, we need to work out a work-around solution for 1.10 release.
# From the [RocksDB document of
WriteBufferManager|https://github.com/facebook/rocksdb/wiki/Write-Buffer-Manager]
it should be able to limit the total memory of memtables. More specific, the
document says "a flush will be triggered if total mutable memtable size exceeds
90% of the limit". However, from the
[implementation|https://github.com/facebook/rocksdb/blob/e8263dbdaad0546c54bddd01a8454c2e750a86c2/include/rocksdb/write_buffer_manager.h#L55]
we could tell it's not following the document (write buffer could go up to
150% of the size), which is really a surprise.
# Based on the fact of #2 and #3, I suggest we work around by doing an internal
computation and setting the size of {{Cache}} and {{WriteBufferManager}}
accordingly: assume the memory limit is {{N}} and the write ratio is R, we have
{{1.5*size_of_write_buffer_manager=R*N}},
{{1.5*size_of_write_buffer_manager+size_of_others=N}}, and
{{size_of_write_buffer_manager+size_of_others=size_of_cache}}, then we know
{{size_of_write_buffer_manager=2*N*R/3}} and {{size_of_cache=N*(3-R)/3}}
# For the additional cost of pinned iterator and/or index, it should be
relatively small, and if it indeed cause out-of-memory, we suggest to use
{{taskmanager.memory.task.offheap}} to cover this part.
# Note that we will be able to get rid of the kind-of-complicated work-around
right after the rocksdb bug is fixed.
What do you think?
[~sewen] please also shed some lights here, thanks.
was (Author: carp84):
Thanks for the efforts and update [~yunta]! Below are some comments and
suggestions according to the testing output and investigation result:
# We should write explicitly in our document not to set
{{optimizeForPointLookup}} when enabling RocksDB memory control.
# We cannot rely on {{strict_capacity_limit}} until
[RocksDB#6247|https://github.com/facebook/rocksdb/issues/6247] is resolved. In
another word, since the issue on {{strict_capacity_limit}} could hardly be
resolved soon, we need to work out a work-around solution for 1.10 release.
# From the [RocksDB document of
WriteBufferManager|https://github.com/facebook/rocksdb/wiki/Write-Buffer-Manager]
it should be able to limit the total memory of memtables. More specific, the
document says "a flush will be triggered if total mutable memtable size exceeds
90% of the limit". However, from the
[implementation|https://github.com/facebook/rocksdb/blob/e8263dbdaad0546c54bddd01a8454c2e750a86c2/include/rocksdb/write_buffer_manager.h#L55]
we could tell it's not following the document (write buffer could go up to
150% of the size), which is really a surprise.
# Based on the fact of #2 and #3, I suggest we work around by doing an internal
computation and setting the size of {{Cache}} and {{WriteBufferManager}}
accordingly: assume the memory limit is {{N}} and the write ratio is R, we have
{{1.5*size_of_write_buffer_manager=R*size_of_cache}},
{{1.5*size_of_write_buffer_manager+size_of_others=N}}, and
{{size_of_write_buffer_manager+size_of_others=size_of_cache}}, then we know
{{size_of_write_buffer_manager=2NR/(3+R)}} and {{size_of_cache=3N/(3+R)}}
# For the additional cost of pinned iterator and/or index, it should be
relatively small, and if it indeed cause out-of-memory, we suggest to use
{{taskmanager.memory.task.offheap}} to cover this part.
# Note that we will be able to get rid of the kind-of-complicated work-around
right after the rocksdb bug is fixed.
What do you think?
[~sewen] please also shed some lights here, thanks.
> Add end-to-end test for controlling RocksDB memory usage
>
>
> Key: FLINK-15368
> URL: https://issues.apache.org/jira/browse/FLINK-15368
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
>Affects Versions: 1.10.0
>Reporter: Yu Li
>Assignee: Yun Tang
>Priority: Critical
> Fix For: 1.10.0
>
> Attachments: rocksdb-arena-block.png
>
>
> We need to add an end-to-end test to make sure the RocksDB memory usage
> control works well, especially under the
[GitHub] [flink] tillrohrmann commented on a change in pull request #10748: [FLINK-15458][conf][docs] Add support for whitelisting ambiguous options
tillrohrmann commented on a change in pull request #10748:
[FLINK-15458][conf][docs] Add support for whitelisting ambiguous options
URL: https://github.com/apache/flink/pull/10748#discussion_r365309969
##
File path:
flink-docs/src/test/java/org/apache/flink/docs/configuration/ConfigOptionsDocsCompletenessITCase.java
##
@@ -61,50 +64,105 @@
private static final Formatter htmlFormatter = new HtmlFormatter();
+ // options for which we allow distinct definitions
+ // this allows reporters to define their own options that are
technically only key suffixes
+ private static final Set WELL_DEFINED_WHITELIST = new
HashSet<>(Arrays.asList(
+ "host",
+ "port"
+ ));
+
@Test
public void testCommonSectionCompleteness() throws IOException,
ClassNotFoundException {
- Map documentedOptions =
parseDocumentedCommonOptions();
- Map existingOptions =
findExistingOptions(
+ Map> documentedOptions =
parseDocumentedCommonOptions();
+ Map> existingOptions =
findExistingOptions(
optionWithMetaInfo ->
optionWithMetaInfo.field.getAnnotation(Documentation.CommonOption.class) !=
null);
+ assertDocumentedOptionsAreWellDefined(documentedOptions);
+ assertExistingOptionsAreWellDefined(existingOptions);
+
compareDocumentedAndExistingOptions(documentedOptions,
existingOptions);
}
@Test
public void testFullReferenceCompleteness() throws IOException,
ClassNotFoundException {
- Map documentedOptions =
parseDocumentedOptions();
- Map existingOptions =
findExistingOptions(ignored -> true);
+ Map> documentedOptions =
parseDocumentedOptions();
+ Map> existingOptions =
findExistingOptions(ignored -> true);
+
+ assertDocumentedOptionsAreWellDefined(documentedOptions);
+ assertExistingOptionsAreWellDefined(existingOptions);
compareDocumentedAndExistingOptions(documentedOptions,
existingOptions);
}
- private static void compareDocumentedAndExistingOptions(Map documentedOptions, Map
existingOptions) {
+ private static void assertDocumentedOptionsAreWellDefined(Map> documentedOptions) {
+ assertOptionsAreWellDefined(documentedOptions, (option1,
option2) -> {
+ // found a ConfigOption pair with the same key
that aren't equal
+ // we fail here outright as this is not a
documentation-completeness problem
+ if
(!option1.defaultValue.equals(option2.defaultValue)) {
+ throw new AssertionError("Documentation
contains distinct defaults for " +
+ option1.key + " in " +
option1.containingFile + " and " + option2.containingFile + '.');
+ } else {
+ throw new AssertionError("Documentation
contains distinct descriptions for " +
+ option1.key + " in " +
option1.containingFile + " and " + option2.containingFile + '.');
+ }
+ });
+ }
+
+ private static void assertExistingOptionsAreWellDefined(Map> existingOptions) {
+ assertOptionsAreWellDefined(existingOptions, (option1, option2)
-> {
+ // found a ConfigOption pair with the same key
that aren't equal
+ // we fail here outright as this is not a
documentation-completeness problem
+ if
(!option1.defaultValue.equals(option2.defaultValue)) {
+ throw new AssertionError("Ambiguous
option " + option1.key + " due to distinct default values (" +
option1.defaultValue + " vs " + option2.defaultValue + ").");
+ } else {
+ throw new AssertionError("Ambiguous
option " + option1.key + " due to distinct descriptions.");
+ }
+ });
+ }
+
+ private static void assertOptionsAreWellDefined(Map> allOptions, BiFunction duplicateHandler) {
+ allOptions.entrySet().stream()
+ .filter(entry ->
!WELL_DEFINED_WHITELIST.contains(entry.getKey()))
+ .map(Map.Entry::getValue)
+ .forEach(options -> options.stream().reduce((option1,
option2) -> {
+ if (option1.equals(option2)) {
+ // we allow multiple instances of
ConfigOptions with the same key if they are identical
+ return option1;
+ } else {
+
[GitHub] [flink] flinkbot edited a comment on issue #10830: [FLINK-15554][azure] Bump jetty-util to 3.1.2
flinkbot edited a comment on issue #10830: [FLINK-15554][azure] Bump jetty-util to 3.1.2 URL: https://github.com/apache/flink/pull/10830#issuecomment-573052596 ## CI report: * 8aba8ccd54ff402b95b302453317d534febb5df9 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143899788) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4260) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10831: [FLINK-13465][legal] Update javassist licensing
flinkbot edited a comment on issue #10831: [FLINK-13465][legal] Update javassist licensing URL: https://github.com/apache/flink/pull/10831#issuecomment-573065869 ## CI report: * d8c12a81e919646eb5cda292cfd5a70744c6c7ed Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143905039) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4261) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10828: [FLINK-15460][JDBC] planner dependencies won't be necessary for JDBC …
flinkbot edited a comment on issue #10828: [FLINK-15460][JDBC] planner dependencies won't be necessary for JDBC … URL: https://github.com/apache/flink/pull/10828#issuecomment-573039067 ## CI report: * 2715875b652966235103eac77e95f1e4bc345c73 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143894187) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4258) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (FLINK-15490) KafkaITCase.testOneSourceMultiplePartitions fails on Travis
[
https://issues.apache.org/jira/browse/FLINK-15490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiangjie Qin resolved FLINK-15490.
--
Resolution: Fixed
Merged.
master: f0f9343a35ff21017e2406614b34a9b1f2712330
release-1.10: a1ba3f98c400fbad0d9719b251470f0c00a51db3
> KafkaITCase.testOneSourceMultiplePartitions fails on Travis
> ---
>
> Key: FLINK-15490
> URL: https://issues.apache.org/jira/browse/FLINK-15490
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Tests
>Affects Versions: 1.10.0
>Reporter: Till Rohrmann
>Assignee: Jiangjie Qin
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The test {{KafkaITCase.testOneSourceMultiplePartitions}} failed on Travis
> because it received a duplicate value:
> {code}
> 13:10:49,276 INFO
> org.apache.flink.streaming.connectors.kafka.testutils.FailingIdentityMapper
> - > Failing mapper 1: count=2802, totalCount=3167
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
> at
> org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:648)
> at
> org.apache.flink.streaming.util.TestStreamEnvironment.execute(TestStreamEnvironment.java:77)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1628)
> at org.apache.flink.test.util.TestUtils.tryExecute(TestUtils.java:35)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaConsumerTestBase.runOneSourceMultiplePartitionsExactlyOnceTest(KafkaConsumerTestBase.java:912)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaITCase.testOneSourceMultiplePartitions(KafkaITCase.java:102)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.flink.runtime.JobException: Recovery is suppressed by
> FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=1,
> backoffTimeMS=0)
> at
> org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:110)
> at
> org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:76)
> at
> org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:186)
> at
> org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:181)
> at
> org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:175)
> at
> org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:476)
> at
> org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:380)
> at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:279)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:194)
> at
> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
> at
[GitHub] [flink] becketqin commented on issue #10794: [FLINK-15490][kafka][test-stability] Enable idempotence producing in …
becketqin commented on issue #10794: [FLINK-15490][kafka][test-stability] Enable idempotence producing in … URL: https://github.com/apache/flink/pull/10794#issuecomment-573092217 Merged. master: f0f9343a35ff21017e2406614b34a9b1f2712330 release-1.10: a1ba3f98c400fbad0d9719b251470f0c00a51db3 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] becketqin closed pull request #10794: [FLINK-15490][kafka][test-stability] Enable idempotence producing in …
becketqin closed pull request #10794: [FLINK-15490][kafka][test-stability] Enable idempotence producing in … URL: https://github.com/apache/flink/pull/10794 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10830: [FLINK-15554][azure] Bump jetty-util to 3.1.2
tillrohrmann commented on a change in pull request #10830: [FLINK-15554][azure] Bump jetty-util to 3.1.2 URL: https://github.com/apache/flink/pull/10830#discussion_r365294294 ## File path: flink-filesystems/flink-azure-fs-hadoop/pom.xml ## @@ -36,6 +36,7 @@ under the License. 1.16.0 2.9.4 + 9.3.24.v20180605 Review comment: The `NOTICE` file needs to be updated as well. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] StephanEwen commented on issue #10812: [FLINK-15523][conf] Japicmp checks ConfigConstants
StephanEwen commented on issue #10812: [FLINK-15523][conf] Japicmp checks ConfigConstants URL: https://github.com/apache/flink/pull/10812#issuecomment-573082392 Looks good, thank you for fixing this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10829: [FLINK-14853][docs] Use higher granularity units in generated docs for Duration & MemorySize if possible
flinkbot edited a comment on issue #10829: [FLINK-14853][docs] Use higher granularity units in generated docs for Duration & MemorySize if possible URL: https://github.com/apache/flink/pull/10829#issuecomment-573052559 ## CI report: * c289f16e336e54931105f5c3ec143f8a9fd69021 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143899744) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4259) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10831: [FLINK-13465][legal] Update javassist licensing
flinkbot edited a comment on issue #10831: [FLINK-13465][legal] Update javassist licensing URL: https://github.com/apache/flink/pull/10831#issuecomment-573065869 ## CI report: * d8c12a81e919646eb5cda292cfd5a70744c6c7ed Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/143905039) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4261) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] StephanEwen commented on issue #10560: FLINK-15174 added certificate pinning for SSL mutual auth to further protect cluster
StephanEwen commented on issue #10560: FLINK-15174 added certificate pinning for SSL mutual auth to further protect cluster URL: https://github.com/apache/flink/pull/10560#issuecomment-573075148 @flinkbot approve all +1 to merge this, looks good from my side! I would add a minor refinement of the docs in a followup commit. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] Myasuka commented on a change in pull request #10820: [FLINK-15512][statebackend] Refactor the mechanism to calculate the cache capacity shared among RocksDB instance(s)
Myasuka commented on a change in pull request #10820: [FLINK-15512][statebackend] Refactor the mechanism to calculate the cache capacity shared among RocksDB instance(s) URL: https://github.com/apache/flink/pull/10820#discussion_r365275814 ## File path: docs/_includes/generated/rocks_db_configuration.html ## @@ -30,7 +30,7 @@ state.backend.rocksdb.memory.high-prio-pool-ratio 0.1 Double -The fraction of cache memory that is reserved for high-priority data like index, filter, and compression dictionary blocks. This option only has an effect when 'state.backend.rocksdb.memory.managed' or 'state.backend.rocksdb.memory.fixed-per-slot' are configured. +The fraction of total shared memory that is reserved for high-priority data like index, filter, and compression dictionary blocks. This option only has an effect when 'state.backend.rocksdb.memory.managed' or 'state.backend.rocksdb.memory.fixed-per-slot' are configured. Review comment: Thanks for pointing out, it should not be changed here since the ratio is really used by the cache. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] link3280 edited a comment on issue #10674: [FLINK-15220][Connector/Kafka][Table] Add startFromTimestamp in KafkaTableSource
link3280 edited a comment on issue #10674: [FLINK-15220][Connector/Kafka][Table] Add startFromTimestamp in KafkaTableSource URL: https://github.com/apache/flink/pull/10674#issuecomment-573066105 @dawidwys I agree that `OffsetDateTime` is more deterministic and reliable. I didn't give much thought to it, because I saw many temporal functions are using UTC time zone, but this could indeed be a problem. Ping Jark @wuchong who would possibly be in favor of `LocalDateTime` for his opinion. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10831: [FLINK-13465][legal] Update javassist licensing
flinkbot commented on issue #10831: [FLINK-13465][legal] Update javassist licensing URL: https://github.com/apache/flink/pull/10831#issuecomment-573065869 ## CI report: * d8c12a81e919646eb5cda292cfd5a70744c6c7ed UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] link3280 commented on issue #10674: [FLINK-15220][Connector/Kafka][Table] Add startFromTimestamp in KafkaTableSource
link3280 commented on issue #10674: [FLINK-15220][Connector/Kafka][Table] Add startFromTimestamp in KafkaTableSource URL: https://github.com/apache/flink/pull/10674#issuecomment-573066105 @dawidwys I agree that `OffsetDateTime` is more deterministic and reliable. I didn't give much thought to it, because I saw many temporal functions are using UTC time zone. Ping Jark @wuchong who would possibly be in favor of `LocalDateTime` for his opinion. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10830: [FLINK-15554][azure] Bump jetty-util to 3.1.2
flinkbot edited a comment on issue #10830: [FLINK-15554][azure] Bump jetty-util to 3.1.2 URL: https://github.com/apache/flink/pull/10830#issuecomment-573052596 ## CI report: * 8aba8ccd54ff402b95b302453317d534febb5df9 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/143899788) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4260) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10829: [FLINK-14853][docs] Use higher granularity units in generated docs for Duration & MemorySize if possible
flinkbot edited a comment on issue #10829: [FLINK-14853][docs] Use higher granularity units in generated docs for Duration & MemorySize if possible URL: https://github.com/apache/flink/pull/10829#issuecomment-573052559 ## CI report: * c289f16e336e54931105f5c3ec143f8a9fd69021 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/143899744) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4259) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10828: [FLINK-15460][JDBC] planner dependencies won't be necessary for JDBC …
flinkbot edited a comment on issue #10828: [FLINK-15460][JDBC] planner dependencies won't be necessary for JDBC … URL: https://github.com/apache/flink/pull/10828#issuecomment-573039067 ## CI report: * 2715875b652966235103eac77e95f1e4bc345c73 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143894187) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4258) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10779: [FLINK-15327][runtime] No warning of InterruptedException during cancel.
flinkbot edited a comment on issue #10779: [FLINK-15327][runtime] No warning of InterruptedException during cancel. URL: https://github.com/apache/flink/pull/10779#issuecomment-571250275 ## CI report: * b30a4d297496a904582cb24d036cbb4c5b647149 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143281199) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4137) * e48e4c08207d6e135026bf683c1af8a9d1310a76 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143356039) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4152) * 19d4be405c97cf220c291475d6804e4183bedf68 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143466456) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4172) * 3982e03edebfff4a20b6acd4a611400f3ce6d44e Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143566671) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4192) * 7530decdd623b9cdc3489e8b6ee1afbaf0643f78 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143588686) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4194) * 2634381d115390c6720bffaf5d4b2e932187dd49 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143758039) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4234) * c5b4ba1c360a7b04cea60caea894d0233f531e67 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143863271) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4252) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-12004) Consider about POJO using method chaining when extract TypeInformation
[
https://issues.apache.org/jira/browse/FLINK-12004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dawid Wysakowicz closed FLINK-12004.
Fix Version/s: 1.11.0
Resolution: Implemented
Implemented in fb12bcfaa82e10fb85e5f4c45e5d9ab503a3d891
> Consider about POJO using method chaining when extract TypeInformation
> --
>
> Key: FLINK-12004
> URL: https://issues.apache.org/jira/browse/FLINK-12004
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream, API / Type Serialization System
>Affects Versions: 1.7.2
>Reporter: jiawei chen
>Assignee: jiawei chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I have a POJO class using method chaining such as :
> {code:java}
> public static class CustomChainingPojoType {
> private String myField1;
> private int myField2;
> public CustomChainingPojoType() {
> }
> public CustomChainingPojoType setMyField1(String myField1) {
> this.myField1 = myField1;
> return this;
> }
> public CustomChainingPojoType setMyField2(int myField2) {
> this.myField2 = myField2;
> return this;
> }
> public String getMyField1() {
> return myField1;
> }
> public int getMyField2() {
> return myField2;
> }
> }{code}
>
> It can not be analyze as POJO in TypeExtractor because the set method return
> type is not Void, How about add a condition that return type can be class
> self.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-12004) Consider about POJO using method chaining when extract TypeInformation
[
https://issues.apache.org/jira/browse/FLINK-12004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dawid Wysakowicz reassigned FLINK-12004:
Assignee: jiawei chen
> Consider about POJO using method chaining when extract TypeInformation
> --
>
> Key: FLINK-12004
> URL: https://issues.apache.org/jira/browse/FLINK-12004
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream, API / Type Serialization System
>Affects Versions: 1.7.2
>Reporter: jiawei chen
>Assignee: jiawei chen
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I have a POJO class using method chaining such as :
> {code:java}
> public static class CustomChainingPojoType {
> private String myField1;
> private int myField2;
> public CustomChainingPojoType() {
> }
> public CustomChainingPojoType setMyField1(String myField1) {
> this.myField1 = myField1;
> return this;
> }
> public CustomChainingPojoType setMyField2(int myField2) {
> this.myField2 = myField2;
> return this;
> }
> public String getMyField1() {
> return myField1;
> }
> public int getMyField2() {
> return myField2;
> }
> }{code}
>
> It can not be analyze as POJO in TypeExtractor because the set method return
> type is not Void, How about add a condition that return type can be class
> self.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] dawidwys commented on issue #8045: [FLINK-12004][TypeInformation] analyze pojo using method chaining
dawidwys commented on issue #8045: [FLINK-12004][TypeInformation] analyze pojo using method chaining URL: https://github.com/apache/flink/pull/8045#issuecomment-573062515 Thank you for your contribution @fmyblack. Merged This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] dawidwys closed pull request #8045: [FLINK-12004][TypeInformation] analyze pojo using method chaining
dawidwys closed pull request #8045: [FLINK-12004][TypeInformation] analyze pojo using method chaining URL: https://github.com/apache/flink/pull/8045 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10831: [FLINK-13465][legal] Update licensing
flinkbot commented on issue #10831: [FLINK-13465][legal] Update licensing URL: https://github.com/apache/flink/pull/10831#issuecomment-573061824 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit d8c12a81e919646eb5cda292cfd5a70744c6c7ed (Fri Jan 10 14:41:57 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zentol opened a new pull request #10831: [FLINK-13465][legal] Update licensing
zentol opened a new pull request #10831: [FLINK-13465][legal] Update licensing URL: https://github.com/apache/flink/pull/10831 Javassist was bumped in FLINK-13465 but I forgot to update the NOTICE file of flink-dist. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15534) YARNSessionCapacitySchedulerITCase#perJobYarnClusterWithParallelism failed due to NPE
[
https://issues.apache.org/jira/browse/FLINK-15534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17012912#comment-17012912
]
Yang Wang commented on FLINK-15534:
---
[~liyu] Thanks. [YARN-7007|https://issues.apache.org/jira/browse/YARN-7007] has
been added to [hadoop-2.8.6
release|https://issues.apache.org/jira/projects/YARN/versions/12344056].
Unfortunately, it do not has clear release date. I will keep an eye on it.
> YARNSessionCapacitySchedulerITCase#perJobYarnClusterWithParallelism failed
> due to NPE
> -
>
> Key: FLINK-15534
> URL: https://issues.apache.org/jira/browse/FLINK-15534
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.11.0
>Reporter: Yu Li
>Assignee: Yang Wang
>Priority: Blocker
>
> As titled, travis run fails with below error:
> {code}
> 07:29:22.417 [ERROR]
> perJobYarnClusterWithParallelism(org.apache.flink.yarn.YARNSessionCapacitySchedulerITCase)
> Time elapsed: 16.263 s <<< ERROR!
> java.lang.NullPointerException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptMetrics.getAggregateAppResourceUsage(RMAppAttemptMetrics.java:128)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.getApplicationResourceUsageReport(RMAppAttemptImpl.java:900)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.createAndGetApplicationReport(RMAppImpl.java:660)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:930)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplications(ApplicationClientProtocolPBServiceImpl.java:273)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:507)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:847)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:790)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2486)
> at
> org.apache.flink.yarn.YARNSessionCapacitySchedulerITCase.perJobYarnClusterWithParallelism(YARNSessionCapacitySchedulerITCase.java:405)
> Caused by: org.apache.hadoop.ipc.RemoteException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptMetrics.getAggregateAppResourceUsage(RMAppAttemptMetrics.java:128)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.getApplicationResourceUsageReport(RMAppAttemptImpl.java:900)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.createAndGetApplicationReport(RMAppImpl.java:660)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:930)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplications(ApplicationClientProtocolPBServiceImpl.java:273)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:507)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:847)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:790)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2486)
> at
> org.apache.flink.yarn.YARNSessionCapacitySchedulerITCase.perJobYarnClusterWithParallelism(YARNSessionCapacitySchedulerITCase.java:405)
> {code}
> https://api.travis-ci.org/v3/job/634588108/log.txt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] GJL commented on a change in pull request #10804: [FLINK-15488] Obtain the JVM and TM param correctly
GJL commented on a change in pull request #10804: [FLINK-15488] Obtain the JVM
and TM param correctly
URL: https://github.com/apache/flink/pull/10804#discussion_r365249851
##
File path:
flink-dist/src/test/java/org/apache/flink/dist/BashJavaUtilsITCase.java
##
@@ -34,21 +36,30 @@
private static final String RUN_BASH_JAVA_UTILS_CMD_SCRIPT =
"src/test/bin/runBashJavaUtilsCmd.sh";
+ /**
+* Executes the given shell script wrapper and returns the last line.
+*/
+ private String executeScriptAndFetchLastLine(final String command)
throws IOException {
+ String[] commands = {RUN_BASH_JAVA_UTILS_CMD_SCRIPT, command};
+ String[] lines =
executeScript(commands).split(System.lineSeparator());
+ if (lines.length == 0) {
+ return "";
+ } else {
+ return lines[lines.length - 1];
+ }
+ }
+
@Test
public void testGetTmResourceDynamicConfigs() throws Exception {
- String[] command = {RUN_BASH_JAVA_UTILS_CMD_SCRIPT,
-
BashJavaUtils.Command.GET_TM_RESOURCE_DYNAMIC_CONFIGS.toString()};
- String result = executeScript(command);
+ String result =
executeScriptAndFetchLastLine(BashJavaUtils.Command.GET_TM_RESOURCE_DYNAMIC_CONFIGS.toString());
assertNotNull(result);
Review comment:
Sorry, I forgot that the prefix is removed using `sed` at that point.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (FLINK-15556) Add a switch for PushProjectIntoTableSourceScanRule
hailong wang created FLINK-15556: Summary: Add a switch for PushProjectIntoTableSourceScanRule Key: FLINK-15556 URL: https://issues.apache.org/jira/browse/FLINK-15556 Project: Flink Issue Type: Improvement Components: Table SQL / Planner Affects Versions: 1.10.0 Reporter: hailong wang Fix For: 1.11.0 In some case, For the project push down, The digest of tableSource maybe different. For example, If we create a jdbc tablesource, and is used twice after it for different column, the source can not be reused for the different digest. In this case, the query io will more precious, So we can add a switch to turn off project push down, and the reused the source. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] KarmaGYZ commented on a change in pull request #10804: [FLINK-15488] Obtain the JVM and TM param correctly
KarmaGYZ commented on a change in pull request #10804: [FLINK-15488] Obtain the
JVM and TM param correctly
URL: https://github.com/apache/flink/pull/10804#discussion_r365253539
##
File path:
flink-dist/src/test/java/org/apache/flink/dist/BashJavaUtilsITCase.java
##
@@ -34,21 +36,30 @@
private static final String RUN_BASH_JAVA_UTILS_CMD_SCRIPT =
"src/test/bin/runBashJavaUtilsCmd.sh";
+ /**
+* Executes the given shell script wrapper and returns the last line.
+*/
+ private String executeScriptAndFetchLastLine(final String command)
throws IOException {
+ String[] commands = {RUN_BASH_JAVA_UTILS_CMD_SCRIPT, command};
+ String[] lines =
executeScript(commands).split(System.lineSeparator());
+ if (lines.length == 0) {
+ return "";
+ } else {
+ return lines[lines.length - 1];
+ }
+ }
+
@Test
public void testGetTmResourceDynamicConfigs() throws Exception {
- String[] command = {RUN_BASH_JAVA_UTILS_CMD_SCRIPT,
-
BashJavaUtils.Command.GET_TM_RESOURCE_DYNAMIC_CONFIGS.toString()};
- String result = executeScript(command);
+ String result =
executeScriptAndFetchLastLine(BashJavaUtils.Command.GET_TM_RESOURCE_DYNAMIC_CONFIGS.toString());
assertNotNull(result);
Review comment:
Yes indeed. Currently, the test case here is more like an end-to-end test.
However, I think it's necessary to test the bash function. What do you think is
the best way to cover those code path?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10829: [FLINK-14853][docs] Use higher granularity units in generated docs for Duration & MemorySize if possible
flinkbot commented on issue #10829: [FLINK-14853][docs] Use higher granularity units in generated docs for Duration & MemorySize if possible URL: https://github.com/apache/flink/pull/10829#issuecomment-573052559 ## CI report: * c289f16e336e54931105f5c3ec143f8a9fd69021 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10830: [FLINK-15554][azure] Bump jetty-util to 3.1.2
flinkbot commented on issue #10830: [FLINK-15554][azure] Bump jetty-util to 3.1.2 URL: https://github.com/apache/flink/pull/10830#issuecomment-573052596 ## CI report: * 8aba8ccd54ff402b95b302453317d534febb5df9 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10828: [FLINK-15460][JDBC] planner dependencies won't be necessary for JDBC …
flinkbot edited a comment on issue #10828: [FLINK-15460][JDBC] planner dependencies won't be necessary for JDBC … URL: https://github.com/apache/flink/pull/10828#issuecomment-573039067 ## CI report: * 2715875b652966235103eac77e95f1e4bc345c73 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/143894187) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4258) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10779: [FLINK-15327][runtime] No warning of InterruptedException during cancel.
flinkbot edited a comment on issue #10779: [FLINK-15327][runtime] No warning of InterruptedException during cancel. URL: https://github.com/apache/flink/pull/10779#issuecomment-571250275 ## CI report: * b30a4d297496a904582cb24d036cbb4c5b647149 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143281199) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4137) * e48e4c08207d6e135026bf683c1af8a9d1310a76 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143356039) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4152) * 19d4be405c97cf220c291475d6804e4183bedf68 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143466456) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4172) * 3982e03edebfff4a20b6acd4a611400f3ce6d44e Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143566671) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4192) * 7530decdd623b9cdc3489e8b6ee1afbaf0643f78 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143588686) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4194) * 2634381d115390c6720bffaf5d4b2e932187dd49 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143758039) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4234) * c5b4ba1c360a7b04cea60caea894d0233f531e67 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/143863271) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4252) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] GJL commented on issue #10804: [FLINK-15488] Obtain the JVM and TM param correctly
GJL commented on issue #10804: [FLINK-15488] Obtain the JVM and TM param correctly URL: https://github.com/apache/flink/pull/10804#issuecomment-573049886 LGTM. Merging This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] GJL commented on a change in pull request #10804: [FLINK-15488] Obtain the JVM and TM param correctly
GJL commented on a change in pull request #10804: [FLINK-15488] Obtain the JVM
and TM param correctly
URL: https://github.com/apache/flink/pull/10804#discussion_r365249851
##
File path:
flink-dist/src/test/java/org/apache/flink/dist/BashJavaUtilsITCase.java
##
@@ -34,21 +36,30 @@
private static final String RUN_BASH_JAVA_UTILS_CMD_SCRIPT =
"src/test/bin/runBashJavaUtilsCmd.sh";
+ /**
+* Executes the given shell script wrapper and returns the last line.
+*/
+ private String executeScriptAndFetchLastLine(final String command)
throws IOException {
+ String[] commands = {RUN_BASH_JAVA_UTILS_CMD_SCRIPT, command};
+ String[] lines =
executeScript(commands).split(System.lineSeparator());
+ if (lines.length == 0) {
+ return "";
+ } else {
+ return lines[lines.length - 1];
+ }
+ }
+
@Test
public void testGetTmResourceDynamicConfigs() throws Exception {
- String[] command = {RUN_BASH_JAVA_UTILS_CMD_SCRIPT,
-
BashJavaUtils.Command.GET_TM_RESOURCE_DYNAMIC_CONFIGS.toString()};
- String result = executeScript(command);
+ String result =
executeScriptAndFetchLastLine(BashJavaUtils.Command.GET_TM_RESOURCE_DYNAMIC_CONFIGS.toString());
assertNotNull(result);
Review comment:
Sorry, I forgot that the prefix is removed using `sed` at that point. For a
unit test, I think it would be easier to invoke main from BashJavaUtils
directly, and redirect System.out using `System.setOut` but that's outside the
scope of this PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] langdangjushi commented on issue #8400: [FLINK-12472][yarn] Support setting attemptFailuresValidityInterval o…
langdangjushi commented on issue #8400: [FLINK-12472][yarn] Support setting attemptFailuresValidityInterval o… URL: https://github.com/apache/flink/pull/8400#issuecomment-573049344 @jiasheng55 大哥, 测试过没啊?? 你写的默认10s, 然后yarn max attempts 至少得是2吧, 那就是说如果在10s内没有做到失败2次, 那么app master将进入无限重启啊哥... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-15555) Delete TABLE_OPTIMIZER_REUSE_SOURCE_ENABLED option for subplaner reuse
hailong wang created FLINK-1:
Summary: Delete TABLE_OPTIMIZER_REUSE_SOURCE_ENABLED option for
subplaner reuse
Key: FLINK-1
URL: https://issues.apache.org/jira/browse/FLINK-1
Project: Flink
Issue Type: Improvement
Components: Table SQL / Runtime
Affects Versions: 1.10.0
Reporter: hailong wang
Fix For: 1.11.0
Blink planner supports subplan reuse. If TABLE_OPTIMIZER_REUSE_SUB_PLAN_ENABLED
is true, the optimizer will try to find out duplicated sub-plans and reuse
them. And if TABLE_OPTIMIZER_REUSE_SOURCE_ENABLED is true, the optimizer will
try to find out duplicated table sources and reuse them.
The option of TABLE_OPTIMIZER_REUSE_SOURCE_ENABLED used to defined whether
TableSourceScan should be reused.
But if the parent's relNode of TableSourceScan can be reused, it will be also
reused even if TABLE_OPTIMIZER_REUSE_SOURCE_ENABLED is false, just like follow
sql:
{code:java}
WITH t AS (SELECT a, b, e FROM x, y WHERE x.a = y.d)
SELECT t1.*, t2.* FROM t t1, t t2 WHERE t1.b = t2.e AND t1.a < 10 AND t2.a > 5
{code}
the plan may be as follow:
{code:java}
HashJoin(joinType=[InnerJoin], where=[=(b, e0)], select=[a, b, e, a0, b0, e0],
build=[right])
:- Exchange(distribution=[hash[b]], shuffle_mode=[BATCH])
: +- Calc(select=[a, b, e])
: +- HashJoin(joinType=[InnerJoin], where=[=(a, d)], select=[a, b, d, e],
build=[left])
::- Exchange(distribution=[hash[a]])
:: +- Calc(select=[a, b], where=[<(a, 10)])
:: +- TableSourceScan(table=[[default_catalog, default_database, x,
source: [TestTableSource(a, b, c)]]], fields=[a, b, c])
:+- Exchange(distribution=[hash[d]], reuse_id=[1])
: +- Calc(select=[d, e])
: +- TableSourceScan(table=[[default_catalog, default_database, y,
source: [TestTableSource(d, e, f)]]], fields=[d, e, f])
+- Exchange(distribution=[hash[e]])
+- Calc(select=[a, b, e])
+- HashJoin(joinType=[InnerJoin], where=[=(a, d)], select=[a, b, d, e],
build=[left])
:- Exchange(distribution=[hash[a]])
: +- Calc(select=[a, b], where=[>(a, 5)])
: +- TableSourceScan(table=[[default_catalog, default_database, x,
source: [TestTableSource(a, b, c)]]], fields=[a, b, c])
+- Reused(reference_id=[1])
{code}
So I think it is useless to defined this option, only
TABLE_OPTIMIZER_REUSE_SUB_PLAN_ENABLED will be ok.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
