[jira] [Assigned] (FLINK-15823) Sync download page translation
[ https://issues.apache.org/jira/browse/FLINK-15823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hequn Cheng reassigned FLINK-15823: --- Assignee: Hequn Cheng > Sync download page translation > -- > > Key: FLINK-15823 > URL: https://issues.apache.org/jira/browse/FLINK-15823 > Project: Flink > Issue Type: Improvement > Components: chinese-translation, Project Website >Reporter: Chesnay Schepler >Assignee: Hequn Cheng >Priority: Major > > Adjust the chinese version of `downloads.md` for the changes made in > https://issues.apache.org/jira/browse/FLINK-15800 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15823) Sync download page translation
Chesnay Schepler created FLINK-15823: Summary: Sync download page translation Key: FLINK-15823 URL: https://issues.apache.org/jira/browse/FLINK-15823 Project: Flink Issue Type: Improvement Components: chinese-translation, Project Website Reporter: Chesnay Schepler Adjust the chinese version of `downloads.md` for the changes made in https://issues.apache.org/jira/browse/FLINK-15800 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-15800) Refactor download page to make hadoop less prominant
[ https://issues.apache.org/jira/browse/FLINK-15800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-15800. Resolution: Fixed asf-site: c88862aa77f77055f737209830b06385fad7b868 > Refactor download page to make hadoop less prominant > > > Key: FLINK-15800 > URL: https://issues.apache.org/jira/browse/FLINK-15800 > Project: Flink > Issue Type: Improvement > Components: Project Website >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > The downloads page is listing shaded-hadoop downloads for every Flink release. > This is ultimately redundant since we don't touch the packaging, and it > reinforces that putting shaded-hadoop into /lib is the "default" way of > adding hadoop, when users should first try exposing the hadoop classpath. > As such we should move shaded-hadoop into the additional components section. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10980: [FLINK-12343][yarn] add yarn file replication option
flinkbot edited a comment on issue #10980: [FLINK-12343][yarn] add yarn file replication option URL: https://github.com/apache/flink/pull/10980#issuecomment-580609306 ## CI report: * 1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/146878202) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10970: [FLINK-15789][kubernetes] Do not wrap the InterruptedException and throw a different one in ActionWatcher.await
flinkbot edited a comment on issue #10970: [FLINK-15789][kubernetes] Do not wrap the InterruptedException and throw a different one in ActionWatcher.await URL: https://github.com/apache/flink/pull/10970#issuecomment-580116823 ## CI report: * d3bc38e0e541eba36972ff7ad63ed8c73137aafc Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/146711098) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4667) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10971: [FLINK-15791][kubernetes] Use explicit I/O Executor for asynchronous operations in Fabric8FlinkKubeClient
flinkbot edited a comment on issue #10971: [FLINK-15791][kubernetes] Use explicit I/O Executor for asynchronous operations in Fabric8FlinkKubeClient URL: https://github.com/apache/flink/pull/10971#issuecomment-580116900 ## CI report: * f769194571216372e83fdff8d9898fd693079498 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/146711109) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4668) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15822) Rethink the necessity of the internal Service
[ https://issues.apache.org/jira/browse/FLINK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027240#comment-17027240 ] Canbin Zheng commented on FLINK-15822: -- cc [~yangwang166] [~tison] > Rethink the necessity of the internal Service > - > > Key: FLINK-15822 > URL: https://issues.apache.org/jira/browse/FLINK-15822 > Project: Flink > Issue Type: Sub-task > Components: Deployment / Kubernetes >Affects Versions: 1.10.0 >Reporter: Canbin Zheng >Priority: Major > Fix For: 1.11.0, 1.10.1 > > > The current design introduces two Kubernetes Services when deploying a new > session cluster. The rest Service serves external communication while the > internal Service mainly serves two purposes: > # A discovery service directs the communication from TaskManagers to the > JobManager Pod that has labels containing the internal Service’s selector in > the non-HA mode, so that the TM Pods could re-register to the new JM Pod once > a JM Pod failover occurs, while in the HA mode, there could be one active and > multiple standby JM Pods, so we use the Pod IP of the active one for internal > communication instead of using the internal Service . > # The OwnerReference of all other Kubernetes Resources, including the rest > Service, the ConfigMap and the JobManager Deployment. > Is it possible that we just create one single Service instead of two? I think > things could work quite well with only the rest Service, meanwhile the design > and code could be more succinct. > This ticket proposes to remove the internal Service, the core changes > including > # In the non-HA mode, we use the rest Service as the JobManager Pod > discovery service. > # Set the JobManager Deployment as the OwnerReference of all the other > Kubernetes Resources, including the rest Service and the ConfigMap. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15822) Rethink the necessity of the internal Service
Canbin Zheng created FLINK-15822: Summary: Rethink the necessity of the internal Service Key: FLINK-15822 URL: https://issues.apache.org/jira/browse/FLINK-15822 Project: Flink Issue Type: Sub-task Components: Deployment / Kubernetes Affects Versions: 1.10.0 Reporter: Canbin Zheng Fix For: 1.11.0, 1.10.1 The current design introduces two Kubernetes Services when deploying a new session cluster. The rest Service serves external communication while the internal Service mainly serves two purposes: # A discovery service directs the communication from TaskManagers to the JobManager Pod that has labels containing the internal Service’s selector in the non-HA mode, so that the TM Pods could re-register to the new JM Pod once a JM Pod failover occurs, while in the HA mode, there could be one active and multiple standby JM Pods, so we use the Pod IP of the active one for internal communication instead of using the internal Service . # The OwnerReference of all other Kubernetes Resources, including the rest Service, the ConfigMap and the JobManager Deployment. Is it possible that we just create one single Service instead of two? I think things could work quite well with only the rest Service, meanwhile the design and code could be more succinct. This ticket proposes to remove the internal Service, the core changes including # In the non-HA mode, we use the rest Service as the JobManager Pod discovery service. # Set the JobManager Deployment as the OwnerReference of all the other Kubernetes Resources, including the rest Service and the ConfigMap. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on issue #10980: [FLINK-12343][yarn] add yarn file replication option
flinkbot commented on issue #10980: [FLINK-12343][yarn] add yarn file replication option URL: https://github.com/apache/flink/pull/10980#issuecomment-580609306 ## CI report: * 1bca661ca3f42c4c98234ac90fb5359516c0b6c9 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wangyang0918 commented on issue #10970: [FLINK-15789][kubernetes] Do not wrap the InterruptedException and throw a different one in ActionWatcher.await
wangyang0918 commented on issue #10970: [FLINK-15789][kubernetes] Do not wrap the InterruptedException and throw a different one in ActionWatcher.await URL: https://github.com/apache/flink/pull/10970#issuecomment-580606349 @flinkbot run travis This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wangyang0918 commented on issue #10971: [FLINK-15791][kubernetes] Use explicit I/O Executor for asynchronous operations in Fabric8FlinkKubeClient
wangyang0918 commented on issue #10971: [FLINK-15791][kubernetes] Use explicit I/O Executor for asynchronous operations in Fabric8FlinkKubeClient URL: https://github.com/apache/flink/pull/10971#issuecomment-580606278 @flinkbot run travis This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-12343) Allow set file.replication in Yarn Configuration
[ https://issues.apache.org/jira/browse/FLINK-12343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027222#comment-17027222 ] Zhenqiu Huang commented on FLINK-12343: --- [~trohrmann] I created a new PR for this ticket. Please take a look. > Allow set file.replication in Yarn Configuration > > > Key: FLINK-12343 > URL: https://issues.apache.org/jira/browse/FLINK-12343 > Project: Flink > Issue Type: Improvement > Components: Command Line Client, Deployment / YARN >Affects Versions: 1.6.4, 1.7.2, 1.8.0 >Reporter: Zhenqiu Huang >Assignee: Zhenqiu Huang >Priority: Minor > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Currently, FlinkYarnSessionCli upload jars into hdfs with default 3 > replications. From our production experience, we find that 3 replications > will block big job (256 containers) to launch, when the HDFS is slow due to > big workload for batch pipelines. Thus, we want to make the factor > customizable from FlinkYarnSessionCli by adding an option. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on issue #10980: [FLINK-12343][yarn] add yarn file replication option
flinkbot commented on issue #10980: [FLINK-12343][yarn] add yarn file replication option URL: https://github.com/apache/flink/pull/10980#issuecomment-580598389 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 1bca661ca3f42c4c98234ac90fb5359516c0b6c9 (Fri Jan 31 06:16:16 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] HuangZhenQiu opened a new pull request #10980: [FLINK-12343][yarn] add yarn file replication option
HuangZhenQiu opened a new pull request #10980: [FLINK-12343][yarn] add yarn file replication option URL: https://github.com/apache/flink/pull/10980 ## What is the purpose of the change Add yarn file replication option, so that users can specify how many replications of a hdfs file (for example, the flink libs job jars) should be created. ## Brief change log Added yarn config option yarn.file-replication, so that Flink user can add replication number for large jobs ( more than 256 containers) to efficiently download jars into each node manager and start containers. ## Verifying this change This change added tests and can be verified as follows: - Added integration tests for end-to-end deployment the replication option configured to 4. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-15821) Allow configuring Kafka partition / topic discovery in StateFun Kafka ingress
[
https://issues.apache.org/jira/browse/FLINK-15821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tzu-Li (Gordon) Tai updated FLINK-15821:
Parent: FLINK-15819
Issue Type: Sub-task (was: Bug)
> Allow configuring Kafka partition / topic discovery in StateFun Kafka ingress
> -
>
> Key: FLINK-15821
> URL: https://issues.apache.org/jira/browse/FLINK-15821
> Project: Flink
> Issue Type: Sub-task
> Components: Stateful Functions
>Affects Versions: statefun-1.1
>Reporter: Tzu-Li (Gordon) Tai
>Priority: Major
>
> This is already implemented in the {{FlinkKafkaConsumer}}, so it would only
> be a matter of exposing it through Stateful Function's Kafka ingress.
> Proposed API:
> {code}
> KafkaIngressBuilder#withTopics(java.util.regex.Pattern regexPattern)
> KafkaIngressBuilder#enableDiscovery(java.time.Duration discoveryInterval)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-15821) Allow configuring Kafka partition / topic discovery in StateFun Kafka ingress
Tzu-Li (Gordon) Tai created FLINK-15821:
---
Summary: Allow configuring Kafka partition / topic discovery in
StateFun Kafka ingress
Key: FLINK-15821
URL: https://issues.apache.org/jira/browse/FLINK-15821
Project: Flink
Issue Type: Bug
Components: Stateful Functions
Affects Versions: statefun-1.1
Reporter: Tzu-Li (Gordon) Tai
This is already implemented in the {{FlinkKafkaConsumer}}, so it would only be
a matter of exposing it through Stateful Function's Kafka ingress.
Proposed API:
{code}
KafkaIngressBuilder#withTopics(java.util.regex.Pattern regexPattern)
KafkaIngressBuilder#enableDiscovery(java.time.Duration discoveryInterval)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-15820) Allow disabling / enabling auto offset committing to Kafka in StateFun Kafka ingress
Tzu-Li (Gordon) Tai created FLINK-15820:
---
Summary: Allow disabling / enabling auto offset committing to
Kafka in StateFun Kafka ingress
Key: FLINK-15820
URL: https://issues.apache.org/jira/browse/FLINK-15820
Project: Flink
Issue Type: Sub-task
Components: Stateful Functions
Affects Versions: statefun-1.1
Reporter: Tzu-Li (Gordon) Tai
This is already supported in {{FlinkKafkaConsumer}}, so it would only be a
matter of exposing it through the Stateful Functions Kafka ingress.
It would be reasonable to support this with it being disabled by default, so
that users do not always need to set the consumer group id and have a minimal
setup to get started with using the ingress.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15819) Add more StateFun Kafka ingress features that are already available in FlinkKafkaConsumer
[
https://issues.apache.org/jira/browse/FLINK-15819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tzu-Li (Gordon) Tai updated FLINK-15819:
Affects Version/s: statefun-1.1
> Add more StateFun Kafka ingress features that are already available in
> FlinkKafkaConsumer
> -
>
> Key: FLINK-15819
> URL: https://issues.apache.org/jira/browse/FLINK-15819
> Project: Flink
> Issue Type: New Feature
> Components: Stateful Functions
>Affects Versions: statefun-1.1
>Reporter: Tzu-Li (Gordon) Tai
>Priority: Major
>
> This is an umbrella JIRA to list all pending {{FlinkKafkaConsumer}} features
> that are reasonable to expose through Stateful Function's Kafka ingress.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-15819) Add more StateFun Kafka ingress features that are already available in FlinkKafkaConsumer
Tzu-Li (Gordon) Tai created FLINK-15819:
---
Summary: Add more StateFun Kafka ingress features that are already
available in FlinkKafkaConsumer
Key: FLINK-15819
URL: https://issues.apache.org/jira/browse/FLINK-15819
Project: Flink
Issue Type: New Feature
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
This is an umbrella JIRA to list all pending {{FlinkKafkaConsumer}} features
that are reasonable to expose through Stateful Function's Kafka ingress.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15818) Add Kafka ingress startup position config to YAML support
[
https://issues.apache.org/jira/browse/FLINK-15818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tzu-Li (Gordon) Tai updated FLINK-15818:
Description:
With FLINK-15769, we've added startup position configuration to the Java Kafka
Ingress.
The same functionality should also be added to the YAML support.
Part of this work should also cover unifying source creation logic between the
{{ProtobufKafkaSourceProvider}} and {{KafkaSourceProvider}} so that
configuration validation is shared.
was:
With FLINK-15769, we've added startup position configuration to the Java Kafka
Ingress.
The same functionality should also be added to the YAML support.
Part of this work should also cover unifying source creation logic between the
`ProtobufKafkaSourceProvider` and `KafkaSourceProvider` so that configuration
validation is shared.
> Add Kafka ingress startup position config to YAML support
> -
>
> Key: FLINK-15818
> URL: https://issues.apache.org/jira/browse/FLINK-15818
> Project: Flink
> Issue Type: New Feature
> Components: Stateful Functions
>Affects Versions: statefun-1.1
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Major
>
> With FLINK-15769, we've added startup position configuration to the Java
> Kafka Ingress.
> The same functionality should also be added to the YAML support.
> Part of this work should also cover unifying source creation logic between
> the {{ProtobufKafkaSourceProvider}} and {{KafkaSourceProvider}} so that
> configuration validation is shared.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-15818) Add Kafka ingress startup position config to YAML support
Tzu-Li (Gordon) Tai created FLINK-15818: --- Summary: Add Kafka ingress startup position config to YAML support Key: FLINK-15818 URL: https://issues.apache.org/jira/browse/FLINK-15818 Project: Flink Issue Type: New Feature Components: Stateful Functions Affects Versions: statefun-1.1 Reporter: Tzu-Li (Gordon) Tai Assignee: Tzu-Li (Gordon) Tai With FLINK-15769, we've added startup position configuration to the Java Kafka Ingress. The same functionality should also be added to the YAML support. Part of this work should also cover unifying source creation logic between the `ProtobufKafkaSourceProvider` and `KafkaSourceProvider` so that configuration validation is shared. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink-statefun] tzulitai commented on issue #5: [FLINK-15769] [kafka-io] Add configuring startup position for Kafka ingress
tzulitai commented on issue #5: [FLINK-15769] [kafka-io] Add configuring startup position for Kafka ingress URL: https://github.com/apache/flink-statefun/pull/5#issuecomment-580576641 Sorry @sjwiesman, I had to do a force push when addressing your comments because I accidentally squashed my fixup commits already This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15393) Change 'ReturnExpiredIfNotCleanedUp' to 'NeverReturnExpired' in JoinRecordStateViews#createTtlConfig
[
https://issues.apache.org/jira/browse/FLINK-15393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027170#comment-17027170
]
Shi Feng commented on FLINK-15393:
--
Hi [~hailong wang], I knew. But it does not work when using blink planner :)
> Change 'ReturnExpiredIfNotCleanedUp' to 'NeverReturnExpired' in
> JoinRecordStateViews#createTtlConfig
> --
>
> Key: FLINK-15393
> URL: https://issues.apache.org/jira/browse/FLINK-15393
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
>Affects Versions: 1.10.0
>Reporter: hailong wang
>Priority: Major
>
> In Blink planner, we use ttl state to clean expired data by rocksdb for
> regular joins. The StateTtlConfig is:
> {code:java}
> StateTtlConfig
>.newBuilder(Time.milliseconds(retentionTime))
>.setUpdateType(StateTtlConfig.UpdateType.OnCreateAndWrite)
>
> .setStateVisibility(StateTtlConfig.StateVisibility.ReturnExpiredIfNotCleanedUp)
>.build();
> {code}
> I think we should use StateTtlConfig.StateVisibility.NeverReturnExpired to
> prevent to join expired data.
> BTW, Only regular joins use rocksdb filter to clean expired data, should we
> change timer to rocksdb filter in other sql operator such as
> TemporalTableJoin?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10978: [FLINK-15584] Give nested data type of ROWs in ValidationException
flinkbot edited a comment on issue #10978: [FLINK-15584] Give nested data type of ROWs in ValidationException URL: https://github.com/apache/flink/pull/10978#issuecomment-580392596 ## CI report: * a55aa8516067969246aad8fffa1fed5272046cc6 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/146804306) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4702) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-15817) Kubernetes Resource leak while deployment exception happens
Canbin Zheng created FLINK-15817: Summary: Kubernetes Resource leak while deployment exception happens Key: FLINK-15817 URL: https://issues.apache.org/jira/browse/FLINK-15817 Project: Flink Issue Type: Sub-task Components: Deployment / Kubernetes Affects Versions: 1.10.0 Reporter: Canbin Zheng Fix For: 1.11.0, 1.10.1 When we deploy a new session cluster on Kubernetes cluster, usually there are four steps to create the Kubernetes components, and the creation order is as below: internal Service -> rest Service -> ConfigMap -> JobManager Deployment. After the internal Service is created, any Exceptions that fail the cluster deployment progress would cause Kubernetes Resource leak, for example: # If failed to create rest Service due to service name constraint([FLINK-15816|https://issues.apache.org/jira/browse/FLINK-15816]), the internal Service would not be cleaned up when the deploy progress terminates. # If failed to create JobManager Deployment(a case is that _jobmanager.heap.size_ is too small such as 512M, which is less than the default configuration value of 'containerized.heap-cutoff-min'), the internal Service, the rest Service, and the ConfigMap all leaks. This ticket proposes to do some clean-ups(cleans the residual Services and ConfigMap) if the cluster deployment progress terminates accidentally on the client-side. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10978: [FLINK-15584] Give nested data type of ROWs in ValidationException
flinkbot edited a comment on issue #10978: [FLINK-15584] Give nested data type of ROWs in ValidationException URL: https://github.com/apache/flink/pull/10978#issuecomment-580392596 ## CI report: * a55aa8516067969246aad8fffa1fed5272046cc6 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/146804306) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4702) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] yanghua commented on issue #10972: [FLINK-15810] Add more description for FlinkKafkaProducerMigrationOperatorTest
yanghua commented on issue #10972: [FLINK-15810] Add more description for FlinkKafkaProducerMigrationOperatorTest URL: https://github.com/apache/flink/pull/10972#issuecomment-580558540 cc @aljoscha This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (FLINK-15652) Support for imagePullSecrets k8s option
[ https://issues.apache.org/jira/browse/FLINK-15652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen reassigned FLINK-15652: - Assignee: Canbin Zheng > Support for imagePullSecrets k8s option > --- > > Key: FLINK-15652 > URL: https://issues.apache.org/jira/browse/FLINK-15652 > Project: Flink > Issue Type: Sub-task > Components: Deployment / Kubernetes >Reporter: Canbin Zheng >Assignee: Canbin Zheng >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > Time Spent: 10m > Remaining Estimate: 0h > > It's likely that one would pull images from the private image registry, > credentials can be passed with the Pod specification through the > `imagePullSecrets` parameter, > which refers to the k8s secret by name. Implementation wise we expose a new > configuration option to the users and then pass it along to the K8S. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-15647) Support to set annotations for JM/TM Pods
[ https://issues.apache.org/jira/browse/FLINK-15647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen reassigned FLINK-15647: - Assignee: Canbin Zheng > Support to set annotations for JM/TM Pods > - > > Key: FLINK-15647 > URL: https://issues.apache.org/jira/browse/FLINK-15647 > Project: Flink > Issue Type: Sub-task > Components: Deployment / Kubernetes >Reporter: Canbin Zheng >Assignee: Canbin Zheng >Priority: Major > Fix For: 1.11.0 > > > One can use either labels or annotations to attach metadata to Kubernetes > objects. Labels can be used to select objects and to find collections of > objects that satisfy certain conditions. In contrast, annotations are not > used to identify and select objects. The metadata in an annotation can be > small or large, structured or unstructured, and can include characters not > permitted by labels. > > [Kubernetes > Annotation|https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] ayushtkn commented on issue #10978: [FLINK-15584] Give nested data type of ROWs in ValidationException
ayushtkn commented on issue #10978: [FLINK-15584] Give nested data type of ROWs in ValidationException URL: https://github.com/apache/flink/pull/10978#issuecomment-580555602 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15791) Don't use ForkJoinPool#commonPool() for executing asynchronous operations in Fabric8FlinkKubeClient
[
https://issues.apache.org/jira/browse/FLINK-15791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027133#comment-17027133
]
Yang Wang commented on FLINK-15791:
---
[~trohrmann] thanks. It's my bad. I will always wait for the assignee next time.
> Don't use ForkJoinPool#commonPool() for executing asynchronous operations in
> Fabric8FlinkKubeClient
> ---
>
> Key: FLINK-15791
> URL: https://issues.apache.org/jira/browse/FLINK-15791
> Project: Flink
> Issue Type: Sub-task
> Components: Deployment / Kubernetes
>Affects Versions: 1.10.0
>Reporter: Till Rohrmann
>Assignee: Yang Wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0, 1.10.1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We should not use the {{ForkJoinPool#commonPool()}} in order to run
> asynchronous operations in the {{Fabric8FlinkKubeClient}} as it is done
> [here|https://github.com/apache/flink/blob/master/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/Fabric8FlinkKubeClient.java#L315].
> Since we don't know which other component is using this pool, it can be
> quite dangerous to use it as there might be congestion.
> Instead, I propose to provide an explicit I/O {{Executor}} which is used for
> running asynchronous operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-15816) Limit the maximum length of the value of kubernetes.cluster-id configuration option
[
https://issues.apache.org/jira/browse/FLINK-15816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027126#comment-17027126
]
Canbin Zheng edited comment on FLINK-15816 at 1/31/20 1:51 AM:
---
cc [~yangwang166] [~trohrmann]
was (Author: felixzheng):
cc [~yangwang166] @[~trohrmann]
> Limit the maximum length of the value of kubernetes.cluster-id configuration
> option
> ---
>
> Key: FLINK-15816
> URL: https://issues.apache.org/jira/browse/FLINK-15816
> Project: Flink
> Issue Type: Sub-task
> Components: Deployment / Kubernetes
>Affects Versions: 1.10.0
>Reporter: Canbin Zheng
>Priority: Minor
> Fix For: 1.11.0, 1.10.1
>
>
> Two Kubernetes Service will be created when a session cluster is deployed,
> one is the internal Service and the other is the rest Service, we set the
> internal Service name to the value of the _kubernetes.cluster-id_
> configuration option and then set the rest Service name to
> _${kubernetes.cluster-id}_ with a suffix *-rest* appended, said if we set the
> _kubernetes.cluster-id_ to *session-test*, then the internal Service name
> will be *session-test* and the rest Service name be *session-test-rest;*
> there is a constraint in Kubernetes that the Service name must be no more
> than 63 characters, for the current naming convention it leads to that the
> value of _kubernetes.cluster-id_ should not be more than 58 characters,
> otherwise there are scenarios that the internal Service is created
> successfully then comes up with a ClusterDeploymentException when trying to
> create the rest Service.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15816) Limit the maximum length of the value of kubernetes.cluster-id configuration option
[
https://issues.apache.org/jira/browse/FLINK-15816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027126#comment-17027126
]
Canbin Zheng commented on FLINK-15816:
--
cc [~yangwang166] @[~trohrmann]
> Limit the maximum length of the value of kubernetes.cluster-id configuration
> option
> ---
>
> Key: FLINK-15816
> URL: https://issues.apache.org/jira/browse/FLINK-15816
> Project: Flink
> Issue Type: Sub-task
> Components: Deployment / Kubernetes
>Affects Versions: 1.10.0
>Reporter: Canbin Zheng
>Priority: Minor
> Fix For: 1.11.0, 1.10.1
>
>
> Two Kubernetes Service will be created when a session cluster is deployed,
> one is the internal Service and the other is the rest Service, we set the
> internal Service name to the value of the _kubernetes.cluster-id_
> configuration option and then set the rest Service name to
> _${kubernetes.cluster-id}_ with a suffix *-rest* appended, said if we set the
> _kubernetes.cluster-id_ to *session-test*, then the internal Service name
> will be *session-test* and the rest Service name be *session-test-rest;*
> there is a constraint in Kubernetes that the Service name must be no more
> than 63 characters, for the current naming convention it leads to that the
> value of _kubernetes.cluster-id_ should not be more than 58 characters,
> otherwise there are scenarios that the internal Service is created
> successfully then comes up with a ClusterDeploymentException when trying to
> create the rest Service.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-15816) Limit the maximum length of the value of kubernetes.cluster-id configuration option
Canbin Zheng created FLINK-15816:
Summary: Limit the maximum length of the value of
kubernetes.cluster-id configuration option
Key: FLINK-15816
URL: https://issues.apache.org/jira/browse/FLINK-15816
Project: Flink
Issue Type: Sub-task
Components: Deployment / Kubernetes
Affects Versions: 1.10.0
Reporter: Canbin Zheng
Fix For: 1.11.0, 1.10.1
Two Kubernetes Service will be created when a session cluster is deployed, one
is the internal Service and the other is the rest Service, we set the internal
Service name to the value of the _kubernetes.cluster-id_ configuration option
and then set the rest Service name to _${kubernetes.cluster-id}_ with a suffix
*-rest* appended, said if we set the _kubernetes.cluster-id_ to *session-test*,
then the internal Service name will be *session-test* and the rest Service name
be *session-test-rest;* there is a constraint in Kubernetes that the Service
name must be no more than 63 characters, for the current naming convention it
leads to that the value of _kubernetes.cluster-id_ should not be more than 58
characters, otherwise there are scenarios that the internal Service is created
successfully then comes up with a ClusterDeploymentException when trying to
create the rest Service.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10979: [FLINK-15814][connectors] Log warning when StreamingFileSink is used with bulk output formats and an ambiguous S3 scheme
flinkbot edited a comment on issue #10979: [FLINK-15814][connectors] Log warning when StreamingFileSink is used with bulk output formats and an ambiguous S3 scheme URL: https://github.com/apache/flink/pull/10979#issuecomment-580523238 ## CI report: * a5ed5755f47351d60f5dec965ae273eb67dade75 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/146853266) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4705) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15646) Configurable K8s context support
[
https://issues.apache.org/jira/browse/FLINK-15646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027113#comment-17027113
]
Canbin Zheng commented on FLINK-15646:
--
Thanks for the reminder [~trohrmann]
> Configurable K8s context support
>
>
> Key: FLINK-15646
> URL: https://issues.apache.org/jira/browse/FLINK-15646
> Project: Flink
> Issue Type: Sub-task
> Components: Deployment / Kubernetes
>Affects Versions: 1.10.0
>Reporter: Canbin Zheng
>Assignee: Canbin Zheng
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0, 1.10.1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> One would be forced to first {{kubectl config use-context }} to
> switch to the desired context if working with multiple K8S clusters or having
> multiple K8S "users" for interacting with the specified cluster, so it is an
> important improvement to add an option(kubernetes.context) for configuring
> arbitrary contexts when deploying a Flink cluster. If that option is not
> specified, then the current context in the config file would be used.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10979: [FLINK-15814][connectors] Log warning when StreamingFileSink is used with bulk output formats and an ambiguous S3 scheme
flinkbot edited a comment on issue #10979: [FLINK-15814][connectors] Log warning when StreamingFileSink is used with bulk output formats and an ambiguous S3 scheme URL: https://github.com/apache/flink/pull/10979#issuecomment-580523238 ## CI report: * a5ed5755f47351d60f5dec965ae273eb67dade75 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/146853266) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4705) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10979: [FLINK-15814][connectors] Log warning when StreamingFileSink is used with bulk output formats and an ambiguous S3 scheme
flinkbot commented on issue #10979: [FLINK-15814][connectors] Log warning when StreamingFileSink is used with bulk output formats and an ambiguous S3 scheme URL: https://github.com/apache/flink/pull/10979#issuecomment-580523238 ## CI report: * a5ed5755f47351d60f5dec965ae273eb67dade75 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10979: [FLINK-15814][connectors] Log warning when StreamingFileSink is used with bulk output formats and an ambiguous S3 scheme
flinkbot commented on issue #10979: [FLINK-15814][connectors] Log warning when StreamingFileSink is used with bulk output formats and an ambiguous S3 scheme URL: https://github.com/apache/flink/pull/10979#issuecomment-580514392 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit a5ed5755f47351d60f5dec965ae273eb67dade75 (Thu Jan 30 23:44:04 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-15814) Log warning when StreamingFileSink is used with bulk output formats and an ambiguous S3 scheme
[ https://issues.apache.org/jira/browse/FLINK-15814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-15814: --- Labels: pull-request-available (was: ) > Log warning when StreamingFileSink is used with bulk output formats and an > ambiguous S3 scheme > -- > > Key: FLINK-15814 > URL: https://issues.apache.org/jira/browse/FLINK-15814 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Minor > Labels: pull-request-available > Fix For: 1.11.0 > > > StreamingFileSink does not properly work with bulk output formats and > s3-presto. We should log a warning when the provided scheme is 's3p' or 's3' > (which is ambiguous) explaining the potential issues and encourage users to > only use 's3a' which always delegates to hadoop. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] sjwiesman opened a new pull request #10979: [FLINK-15814][connectors] Log warning when StreamingFileSink is used with bulk output formats and an ambiguous S3 scheme
sjwiesman opened a new pull request #10979: [FLINK-15814][connectors] Log warning when StreamingFileSink is used with bulk output formats and an ambiguous S3 scheme URL: https://github.com/apache/flink/pull/10979 ## What is the purpose of the change StreamingFileSink does not properly work with bulk output formats and s3-presto. We should log a warning when the provided scheme is 's3p' or 's3' (which is ambiguous) explaining the potential issues and encourage users to only use 's3a' which always delegates to hadoop. ## Brief change log Add a WARN log when creating a bulk format for s3 and s3p scheme's. ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-2167) Register HCatalog as external catalog in TableEnvironment
[
https://issues.apache.org/jira/browse/FLINK-2167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027071#comment-17027071
]
Brandon Bevans commented on FLINK-2167:
---
Is this ticket still applicable given the Blink / Catalog changes in 1.9?
> Register HCatalog as external catalog in TableEnvironment
> -
>
> Key: FLINK-2167
> URL: https://issues.apache.org/jira/browse/FLINK-2167
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / Ecosystem
>Affects Versions: 0.9
>Reporter: Fabian Hueske
>Priority: Minor
> Labels: starter
>
> Register Hive's HCatalog as external catalog in the {{TableEnvironment}}.
> After registration, Table API or SQL queries should be able to access tables
> registered in HCatalog.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10954: [FLINK-15785][travis][e2e] Rework E2E test activations
flinkbot edited a comment on issue #10954: [FLINK-15785][travis][e2e] Rework E2E test activations URL: https://github.com/apache/flink/pull/10954#issuecomment-579242201 ## CI report: * 0a4cdba97fcfbb788402b890e6e958cb2949646b Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146389068) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4638) * 73b8f36264985692b35cf2e2ddb3c619df97d299 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146815794) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4704) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (FLINK-15814) Log warning when StreamingFileSink is used with bulk output formats and an ambiguous S3 scheme
[ https://issues.apache.org/jira/browse/FLINK-15814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nico Kruber reassigned FLINK-15814: --- Assignee: Seth Wiesman > Log warning when StreamingFileSink is used with bulk output formats and an > ambiguous S3 scheme > -- > > Key: FLINK-15814 > URL: https://issues.apache.org/jira/browse/FLINK-15814 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Minor > Fix For: 1.11.0 > > > StreamingFileSink does not properly work with bulk output formats and > s3-presto. We should log a warning when the provided scheme is 's3p' or 's3' > (which is ambiguous) explaining the potential issues and encourage users to > only use 's3a' which always delegates to hadoop. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10436: [FLINK-14920] [flink-end-to-end-perf-tests] Set up environment to run performance e2e tests
flinkbot edited a comment on issue #10436: [FLINK-14920] [flink-end-to-end-perf-tests] Set up environment to run performance e2e tests URL: https://github.com/apache/flink/pull/10436#issuecomment-562103759 ## CI report: * 3441142c77315169ce7a3b48d4d59e710c1016c2 Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/139518645) * 68bc1d0dac16c85a35644a3444777dde5b38257c Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/139548818) * 60710d385f3fc62c641d36da4029813d779caf6d Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/139555833) * f4d638d6b1e578bc631df874eb67efdeb4673601 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/139575528) * 841bea9216e1c07d2fce9093d720f64f6d6c889f Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/139812364) * 87a8eb32399114c449f75a94d0243afed93adda5 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/139852611) * ed6a1e4c6e0fea5c0432c0dde6312c7b73ae447c Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140126741) * c1a1151bd3eb16de5851105700fd48928fe15e7b Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140166474) * a08d23d559e36ff93480728a9168e92319941a68 Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/140528959) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3436) * faf3ac41b617d6aad093e61dd89445947f2898c0 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140533954) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3441) * 88c6ebcfb9d6b73fade99ce198ec9a7fe74f6c2d Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140587691) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3464) * 93ecb7520afc244ca00956b0d4b5aa89f953561b Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140604882) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3475) * bc7ca35a50d44cc4e89b5c7fb8d49986afbdd4dd Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143599767) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4197) * 8dd50647a22991315ead23b7c29b32a0727730a2 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143849156) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4247) * 8957e09a4854d6200f75ac306d42fdae87b301aa Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/145256639) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4513) * 0dcd1e2f5c7d20a23bbb5699f4e0c6c32583daba Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146808912) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4703) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10978: [FLINK-15584] Give nested data type of ROWs in ValidationException
flinkbot edited a comment on issue #10978: [FLINK-15584] Give nested data type of ROWs in ValidationException URL: https://github.com/apache/flink/pull/10978#issuecomment-580392596 ## CI report: * a55aa8516067969246aad8fffa1fed5272046cc6 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/146804306) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4702) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10954: [FLINK-15785][travis][e2e] Rework E2E test activations
flinkbot edited a comment on issue #10954: [FLINK-15785][travis][e2e] Rework E2E test activations URL: https://github.com/apache/flink/pull/10954#issuecomment-579242201 ## CI report: * 0a4cdba97fcfbb788402b890e6e958cb2949646b Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146389068) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4638) * 73b8f36264985692b35cf2e2ddb3c619df97d299 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146815794) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4704) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10436: [FLINK-14920] [flink-end-to-end-perf-tests] Set up environment to run performance e2e tests
flinkbot edited a comment on issue #10436: [FLINK-14920] [flink-end-to-end-perf-tests] Set up environment to run performance e2e tests URL: https://github.com/apache/flink/pull/10436#issuecomment-562103759 ## CI report: * 3441142c77315169ce7a3b48d4d59e710c1016c2 Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/139518645) * 68bc1d0dac16c85a35644a3444777dde5b38257c Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/139548818) * 60710d385f3fc62c641d36da4029813d779caf6d Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/139555833) * f4d638d6b1e578bc631df874eb67efdeb4673601 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/139575528) * 841bea9216e1c07d2fce9093d720f64f6d6c889f Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/139812364) * 87a8eb32399114c449f75a94d0243afed93adda5 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/139852611) * ed6a1e4c6e0fea5c0432c0dde6312c7b73ae447c Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140126741) * c1a1151bd3eb16de5851105700fd48928fe15e7b Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140166474) * a08d23d559e36ff93480728a9168e92319941a68 Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/140528959) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3436) * faf3ac41b617d6aad093e61dd89445947f2898c0 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140533954) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3441) * 88c6ebcfb9d6b73fade99ce198ec9a7fe74f6c2d Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140587691) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3464) * 93ecb7520afc244ca00956b0d4b5aa89f953561b Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140604882) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3475) * bc7ca35a50d44cc4e89b5c7fb8d49986afbdd4dd Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143599767) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4197) * 8dd50647a22991315ead23b7c29b32a0727730a2 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143849156) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4247) * 8957e09a4854d6200f75ac306d42fdae87b301aa Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/145256639) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4513) * 0dcd1e2f5c7d20a23bbb5699f4e0c6c32583daba Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146808912) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4703) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10978: [FLINK-15584] Give nested data type of ROWs in ValidationException
flinkbot edited a comment on issue #10978: [FLINK-15584] Give nested data type of ROWs in ValidationException URL: https://github.com/apache/flink/pull/10978#issuecomment-580392596 ## CI report: * a55aa8516067969246aad8fffa1fed5272046cc6 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/146804306) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4702) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10954: [FLINK-15785][travis][e2e] Rework E2E test activations
flinkbot edited a comment on issue #10954: [FLINK-15785][travis][e2e] Rework E2E test activations URL: https://github.com/apache/flink/pull/10954#issuecomment-579242201 ## CI report: * 0a4cdba97fcfbb788402b890e6e958cb2949646b Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146389068) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4638) * 73b8f36264985692b35cf2e2ddb3c619df97d299 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10436: [FLINK-14920] [flink-end-to-end-perf-tests] Set up environment to run performance e2e tests
flinkbot edited a comment on issue #10436: [FLINK-14920] [flink-end-to-end-perf-tests] Set up environment to run performance e2e tests URL: https://github.com/apache/flink/pull/10436#issuecomment-562103759 ## CI report: * 3441142c77315169ce7a3b48d4d59e710c1016c2 Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/139518645) * 68bc1d0dac16c85a35644a3444777dde5b38257c Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/139548818) * 60710d385f3fc62c641d36da4029813d779caf6d Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/139555833) * f4d638d6b1e578bc631df874eb67efdeb4673601 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/139575528) * 841bea9216e1c07d2fce9093d720f64f6d6c889f Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/139812364) * 87a8eb32399114c449f75a94d0243afed93adda5 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/139852611) * ed6a1e4c6e0fea5c0432c0dde6312c7b73ae447c Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140126741) * c1a1151bd3eb16de5851105700fd48928fe15e7b Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140166474) * a08d23d559e36ff93480728a9168e92319941a68 Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/140528959) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3436) * faf3ac41b617d6aad093e61dd89445947f2898c0 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140533954) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3441) * 88c6ebcfb9d6b73fade99ce198ec9a7fe74f6c2d Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140587691) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3464) * 93ecb7520afc244ca00956b0d4b5aa89f953561b Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140604882) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3475) * bc7ca35a50d44cc4e89b5c7fb8d49986afbdd4dd Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143599767) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4197) * 8dd50647a22991315ead23b7c29b32a0727730a2 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143849156) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4247) * 8957e09a4854d6200f75ac306d42fdae87b301aa Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/145256639) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4513) * 0dcd1e2f5c7d20a23bbb5699f4e0c6c32583daba Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/146808912) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4703) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-15814) Log warning when StreamingFileSink is used with bulk output formats and an ambiguous S3 scheme
[ https://issues.apache.org/jira/browse/FLINK-15814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman updated FLINK-15814: - Component/s: Connectors / FileSystem > Log warning when StreamingFileSink is used with bulk output formats and an > ambiguous S3 scheme > -- > > Key: FLINK-15814 > URL: https://issues.apache.org/jira/browse/FLINK-15814 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Reporter: Seth Wiesman >Priority: Minor > Fix For: 1.11.0 > > > StreamingFileSink does not properly work with bulk output formats and > s3-presto. We should log a warning when the provided scheme is 's3p' or 's3' > (which is ambiguous) explaining the potential issues and encourage users to > only use 's3a' which always delegates to hadoop. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10978: [FLINK-15584] Give nested data type of ROWs in ValidationException
flinkbot edited a comment on issue #10978: [FLINK-15584] Give nested data type of ROWs in ValidationException URL: https://github.com/apache/flink/pull/10978#issuecomment-580392596 ## CI report: * a55aa8516067969246aad8fffa1fed5272046cc6 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/146804306) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4702) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10964: [FLINK-15806][yarn] Log recommended way how to stop a detached Yarn session cluster
flinkbot edited a comment on issue #10964: [FLINK-15806][yarn] Log recommended way how to stop a detached Yarn session cluster URL: https://github.com/apache/flink/pull/10964#issuecomment-579861664 ## CI report: * ca1a1bd8f2a853008dc6dbbeb088090f535b9e14 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/146630418) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4660) * 773cdd25c14ec123411e09baabd9522467cd6dbf Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/146799827) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4700) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10436: [FLINK-14920] [flink-end-to-end-perf-tests] Set up environment to run performance e2e tests
flinkbot edited a comment on issue #10436: [FLINK-14920] [flink-end-to-end-perf-tests] Set up environment to run performance e2e tests URL: https://github.com/apache/flink/pull/10436#issuecomment-562103759 ## CI report: * 3441142c77315169ce7a3b48d4d59e710c1016c2 Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/139518645) * 68bc1d0dac16c85a35644a3444777dde5b38257c Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/139548818) * 60710d385f3fc62c641d36da4029813d779caf6d Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/139555833) * f4d638d6b1e578bc631df874eb67efdeb4673601 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/139575528) * 841bea9216e1c07d2fce9093d720f64f6d6c889f Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/139812364) * 87a8eb32399114c449f75a94d0243afed93adda5 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/139852611) * ed6a1e4c6e0fea5c0432c0dde6312c7b73ae447c Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140126741) * c1a1151bd3eb16de5851105700fd48928fe15e7b Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140166474) * a08d23d559e36ff93480728a9168e92319941a68 Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/140528959) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3436) * faf3ac41b617d6aad093e61dd89445947f2898c0 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140533954) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3441) * 88c6ebcfb9d6b73fade99ce198ec9a7fe74f6c2d Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140587691) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3464) * 93ecb7520afc244ca00956b0d4b5aa89f953561b Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140604882) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3475) * bc7ca35a50d44cc4e89b5c7fb8d49986afbdd4dd Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143599767) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4197) * 8dd50647a22991315ead23b7c29b32a0727730a2 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143849156) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4247) * 8957e09a4854d6200f75ac306d42fdae87b301aa Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/145256639) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4513) * 0dcd1e2f5c7d20a23bbb5699f4e0c6c32583daba UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-15623) Buildling flink-python with maven profile docs-and-source fails
[
https://issues.apache.org/jira/browse/FLINK-15623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-15623.
Resolution: Fixed
Small rework to exclude everything under target/ from checkstyle:
master: e8f4f67c4ef45564d089dfe152b216165618d8bf
> Buildling flink-python with maven profile docs-and-source fails
> ---
>
> Key: FLINK-15623
> URL: https://issues.apache.org/jira/browse/FLINK-15623
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.10.0
> Environment: rev: 91d96abe5f42bd088a326870b4885d79611fccb5
>Reporter: Gary Yao
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> *Description*
> Building flink-python with maven profile {{docs-and-source}} fails due to
> checkstyle violations.
> *How to reproduce*
> Running
> {noformat}
> mvn clean install -pl flink-python -Pdocs-and-source -DskipTests
> -DretryFailedDeploymentCount=10
> {noformat}
> should fail with the following error
> {noformat}
> [...]
> [ERROR]
> generated-sources/org/apache/flink/fnexecution/v1/FlinkFnApi.java:[8343]
> (regexp) RegexpSinglelineJava: Line has leading space characters; indentation
> should be performed with tabs only.
> [ERROR]
> generated-sources/org/apache/flink/fnexecution/v1/FlinkFnApi.java:[8344]
> (regexp) RegexpSinglelineJava: Line has leading space characters; indentation
> should be performed with tabs only.
> [ERROR]
> generated-sources/org/apache/flink/fnexecution/v1/FlinkFnApi.java:[8345]
> (regexp) RegexpSinglelineJava: Line has leading space characters; indentation
> should be performed with tabs only.
> [ERROR]
> generated-sources/org/apache/flink/fnexecution/v1/FlinkFnApi.java:[8346]
> (regexp) RegexpSinglelineJava: Line has leading space characters; indentation
> should be performed with tabs only.
> [ERROR]
> generated-sources/org/apache/flink/fnexecution/v1/FlinkFnApi.java:[8347]
> (regexp) RegexpSinglelineJava: Line has leading space characters; indentation
> should be performed with tabs only.
> [ERROR]
> generated-sources/org/apache/flink/fnexecution/v1/FlinkFnApi.java:[8348]
> (regexp) RegexpSinglelineJava: Line has leading space characters; indentation
> should be performed with tabs only.
> [ERROR]
> generated-sources/org/apache/flink/fnexecution/v1/FlinkFnApi.java:[8349]
> (regexp) RegexpSinglelineJava: Line has leading space characters; indentation
> should be performed with tabs only.
> [ERROR]
> generated-sources/org/apache/flink/fnexecution/v1/FlinkFnApi.java:[8350]
> (regexp) RegexpSinglelineJava: Line has leading space characters; indentation
> should be performed with tabs only.
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 18.046 s
> [INFO] Finished at: 2020-01-16T16:44:01+00:00
> [INFO] Final Memory: 158M/2826M
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-checkstyle-plugin:2.17:check (validate) on
> project flink-python_2.11: You have 7603 Checkstyle violations. -> [Help 1]
> [ERROR]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zentol merged pull request #10886: [FLINK-15623][checkstyle] Exclude target directory
zentol merged pull request #10886: [FLINK-15623][checkstyle] Exclude target directory URL: https://github.com/apache/flink/pull/10886 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zentol commented on a change in pull request #10954: [FLINK-15785][travis][e2e] Rework E2E test activations
zentol commented on a change in pull request #10954: [FLINK-15785][travis][e2e] Rework E2E test activations URL: https://github.com/apache/flink/pull/10954#discussion_r373134908 ## File path: flink-end-to-end-tests/pom.xml ## @@ -35,8 +35,19 @@ under the License. flink-end-to-end-tests - org.apache.flink.tests.util.categories.Dummy - + + + + org.apache.flink.tests.util.categories.Dummy + + false + false + false + false + false + false + true + true Review comment: done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-15815) Disable inclusion of original pom under META-INF/maven
Chesnay Schepler created FLINK-15815: Summary: Disable inclusion of original pom under META-INF/maven Key: FLINK-15815 URL: https://issues.apache.org/jira/browse/FLINK-15815 Project: Flink Issue Type: Improvement Components: BuildSystem / Shaded Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: shaded-10.0 The META-INF/maven directory contains the original poms. These are pretty much just noise and are safe to exclude, so let's do just that. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10498: [FLINK-14495][docs] Add documentation for memory control of RocksDB state backend
flinkbot edited a comment on issue #10498: [FLINK-14495][docs] Add documentation for memory control of RocksDB state backend URL: https://github.com/apache/flink/pull/10498#issuecomment-563242008 ## CI report: * dd347fe423313a8bb7c975e3f04ba626fe5dd1b2 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/140238394) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3366) * 2a89d2f7c213fe6137c45c613f6db2062ff1 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/140394883) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3403) * dafe1e061fce4ddda35fc9c4806906ca04c776d8 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146421997) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4641) * f9b43a7e166caeea959525864d7d73bfb92a60e8 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/146789254) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4699) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10978: [FLINK-15584] Give nested data type of ROWs in ValidationException
flinkbot commented on issue #10978: [FLINK-15584] Give nested data type of ROWs in ValidationException URL: https://github.com/apache/flink/pull/10978#issuecomment-580392596 ## CI report: * a55aa8516067969246aad8fffa1fed5272046cc6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10977: [hotfix] Log warning when ambigous S3 scheme is used with bulk output formats and the StreamingFileSink
flinkbot edited a comment on issue #10977: [hotfix] Log warning when ambigous S3 scheme is used with bulk output formats and the StreamingFileSink URL: https://github.com/apache/flink/pull/10977#issuecomment-580379827 ## CI report: * ceaba67d6a940580ac66023600c7da8dee9b Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/146799868) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4701) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] sjwiesman closed pull request #10977: [hotfix] Log warning when ambigous S3 scheme is used with bulk output formats and the StreamingFileSink
sjwiesman closed pull request #10977: [hotfix] Log warning when ambigous S3 scheme is used with bulk output formats and the StreamingFileSink URL: https://github.com/apache/flink/pull/10977 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] sjwiesman commented on issue #10977: [hotfix] Log warning when ambigous S3 scheme is used with bulk output formats and the StreamingFileSink
sjwiesman commented on issue #10977: [hotfix] Log warning when ambigous S3 scheme is used with bulk output formats and the StreamingFileSink URL: https://github.com/apache/flink/pull/10977#issuecomment-580392323 closing in favor of: https://issues.apache.org/jira/browse/FLINK-15814 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-15814) Log warning when StreamingFileSink is used with bulk output formats and an ambiguous S3 scheme
Seth Wiesman created FLINK-15814: Summary: Log warning when StreamingFileSink is used with bulk output formats and an ambiguous S3 scheme Key: FLINK-15814 URL: https://issues.apache.org/jira/browse/FLINK-15814 Project: Flink Issue Type: Improvement Reporter: Seth Wiesman Fix For: 1.11.0 StreamingFileSink does not properly work with bulk output formats and s3-presto. We should log a warning when the provided scheme is 's3p' or 's3' (which is ambiguous) explaining the potential issues and encourage users to only use 's3a' which always delegates to hadoop. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10964: [FLINK-15806][yarn] Log recommended way how to stop a detached Yarn session cluster
flinkbot edited a comment on issue #10964: [FLINK-15806][yarn] Log recommended way how to stop a detached Yarn session cluster URL: https://github.com/apache/flink/pull/10964#issuecomment-579861664 ## CI report: * ca1a1bd8f2a853008dc6dbbeb088090f535b9e14 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/146630418) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4660) * 773cdd25c14ec123411e09baabd9522467cd6dbf Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/146799827) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4700) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10435: [FLINK-13955][runtime] migrate ContinuousFileReaderOperator to the mailbox execution model
flinkbot edited a comment on issue #10435: [FLINK-13955][runtime] migrate ContinuousFileReaderOperator to the mailbox execution model URL: https://github.com/apache/flink/pull/10435#issuecomment-562048103 ## CI report: * d07737bd54c586aa87204c6ac298309f432cc06c Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/139496045) * 1e3e2ff9f65ba15c64fa275fc8567b352eb21c5e Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/139712355) * 58350d6bdc50078b6b15ca70b9d832d3b35aded3 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/139795244) * 287209ff3f521e5fd03df1800ca6425ff6b5a191 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140374178) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3398) * 53161ebd8667e4b01df1e31cf61e38f667c74a32 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/14060) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3479) * b7985f7c1d209fda5c9c05ab1ba16c9ef9e8c5fb Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/140864128) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3539) * 8eff8ec051009be299234ffc8f2afcb6764a1186 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146473490) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4643) * 37198e4d40054bf076b8e53f2ffc24027ee2cc0c Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146551712) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4646) * 28f9eef96938e13005b5106aa89e0a1afdfb4722 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146558623) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4648) * 80bddbc69d17ebb7a202bb12c6d7b4f6124bd59f Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146580700) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4652) * b67aa24b28b0fdf3053def2bebaad56d3b27f64f UNKNOWN * 497f2a70e1c8ff056f8c398167da5cc6b5760c77 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/146624350) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4659) * e830116faf20ea6c889514875c8b07bb77044804 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/146783279) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4698) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10978: [FLINK-15584] Give nested data type of ROWs in ValidationException
flinkbot commented on issue #10978: [FLINK-15584] Give nested data type of ROWs in ValidationException URL: https://github.com/apache/flink/pull/10978#issuecomment-580385511 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit a55aa8516067969246aad8fffa1fed5272046cc6 (Thu Jan 30 18:17:56 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-15584).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15584) Give nested data type of ROWs in ValidationException
[
https://issues.apache.org/jira/browse/FLINK-15584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026897#comment-17026897
]
Ayush Saxena commented on FLINK-15584:
--
Thanx [~BenoitParis] for the report. Makes sense to change.
Have raised PR.
Someone Pls help review!!!
> Give nested data type of ROWs in ValidationException
>
>
> Key: FLINK-15584
> URL: https://issues.apache.org/jira/browse/FLINK-15584
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Affects Versions: 1.9.1, 1.10.0, 1.11.0
>Reporter: Benoît Paris
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In
> {code:java}
> INSERT INTO baz_sink
> SELECT
> a,
> ROW(b, c)
> FROM foo_source{code}
> Schema mismatch mistakes will not get proper detail level, yielding the
> following:
> Caused by: org.apache.flink.table.api.ValidationException: Field types of
> query result and registered TableSink [baz_sink] do not match.
> Query result schema: [a: Integer, EXPR$2: Row]
> TableSink schema: [a: Integer, payload: Row]
> Leaving the user with an opaque 'Row' type to debug.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15584) Give nested data type of ROWs in ValidationException
[
https://issues.apache.org/jira/browse/FLINK-15584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-15584:
---
Labels: pull-request-available (was: )
> Give nested data type of ROWs in ValidationException
>
>
> Key: FLINK-15584
> URL: https://issues.apache.org/jira/browse/FLINK-15584
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Affects Versions: 1.9.1, 1.10.0, 1.11.0
>Reporter: Benoît Paris
>Priority: Minor
> Labels: pull-request-available
>
> In
> {code:java}
> INSERT INTO baz_sink
> SELECT
> a,
> ROW(b, c)
> FROM foo_source{code}
> Schema mismatch mistakes will not get proper detail level, yielding the
> following:
> Caused by: org.apache.flink.table.api.ValidationException: Field types of
> query result and registered TableSink [baz_sink] do not match.
> Query result schema: [a: Integer, EXPR$2: Row]
> TableSink schema: [a: Integer, payload: Row]
> Leaving the user with an opaque 'Row' type to debug.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] ayushtkn opened a new pull request #10978: [FLINK-15584] Give nested data type of ROWs in ValidationException
ayushtkn opened a new pull request #10978: [FLINK-15584] Give nested data type of ROWs in ValidationException URL: https://github.com/apache/flink/pull/10978 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zentol commented on a change in pull request #10954: [FLINK-15785][travis][e2e] Rework E2E test activations
zentol commented on a change in pull request #10954: [FLINK-15785][travis][e2e] Rework E2E test activations URL: https://github.com/apache/flink/pull/10954#discussion_r373106382 ## File path: flink-end-to-end-tests/pom.xml ## @@ -90,6 +162,66 @@ under the License. true + + org.codehaus.gmavenplus + gmavenplus-plugin + 1.8.1 + + + org.codehaus.groovy + groovy-all + 2.5.9 + runtime + pom + + + + + merge-categories + initialize + + execute + + + +
[GitHub] [flink] flinkbot edited a comment on issue #10498: [FLINK-14495][docs] Add documentation for memory control of RocksDB state backend
flinkbot edited a comment on issue #10498: [FLINK-14495][docs] Add documentation for memory control of RocksDB state backend URL: https://github.com/apache/flink/pull/10498#issuecomment-563242008 ## CI report: * dd347fe423313a8bb7c975e3f04ba626fe5dd1b2 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/140238394) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3366) * 2a89d2f7c213fe6137c45c613f6db2062ff1 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/140394883) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3403) * dafe1e061fce4ddda35fc9c4806906ca04c776d8 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146421997) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4641) * f9b43a7e166caeea959525864d7d73bfb92a60e8 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/146789254) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4699) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10977: [hotfix] Log warning when ambigous S3 scheme is used with bulk output formats and the StreamingFileSink
flinkbot commented on issue #10977: [hotfix] Log warning when ambigous S3 scheme is used with bulk output formats and the StreamingFileSink URL: https://github.com/apache/flink/pull/10977#issuecomment-580379827 ## CI report: * ceaba67d6a940580ac66023600c7da8dee9b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10964: [FLINK-15806][yarn] Log recommended way how to stop a detached Yarn session cluster
flinkbot edited a comment on issue #10964: [FLINK-15806][yarn] Log recommended way how to stop a detached Yarn session cluster URL: https://github.com/apache/flink/pull/10964#issuecomment-579861664 ## CI report: * ca1a1bd8f2a853008dc6dbbeb088090f535b9e14 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/146630418) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4660) * 773cdd25c14ec123411e09baabd9522467cd6dbf UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zentol commented on a change in pull request #10954: [FLINK-15785][travis][e2e] Rework E2E test activations
zentol commented on a change in pull request #10954: [FLINK-15785][travis][e2e] Rework E2E test activations URL: https://github.com/apache/flink/pull/10954#discussion_r373096880 ## File path: flink-end-to-end-tests/pom.xml ## @@ -90,6 +162,66 @@ under the License. true + + org.codehaus.gmavenplus + gmavenplus-plugin + 1.8.1 + + + org.codehaus.groovy + groovy-all + 2.5.9 + runtime + pom + + + + + merge-categories + initialize + + execute + + + +
[GitHub] [flink] zentol commented on a change in pull request #10954: [FLINK-15785][travis][e2e] Rework E2E test activations
zentol commented on a change in pull request #10954: [FLINK-15785][travis][e2e] Rework E2E test activations URL: https://github.com/apache/flink/pull/10954#discussion_r373096066 ## File path: flink-end-to-end-tests/pom.xml ## @@ -35,8 +35,19 @@ under the License. flink-end-to-end-tests - org.apache.flink.tests.util.categories.Dummy - + + + + org.apache.flink.tests.util.categories.Dummy + + false + false + false + false + false + false + true + true Review comment: we can. technically it's not necessary, but it's a good idea to do so for clarity. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10977: [hotfix] Log warning when ambigous S3 scheme is used with bulk output formats and the StreamingFileSink
flinkbot commented on issue #10977: [hotfix] Log warning when ambigous S3 scheme is used with bulk output formats and the StreamingFileSink URL: https://github.com/apache/flink/pull/10977#issuecomment-580370694 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit ceaba67d6a940580ac66023600c7da8dee9b (Thu Jan 30 17:41:45 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] sjwiesman opened a new pull request #10977: [hotfix] Log warning when ambigous S3 scheme is used with bulk output formats and the StreamingFileSink
sjwiesman opened a new pull request #10977: [hotfix] Log warning when ambigous S3 scheme is used with bulk output formats and the StreamingFileSink URL: https://github.com/apache/flink/pull/10977 ## What is the purpose of the change The interaction of the multiple s3 filesystems and the streaming file sink is a common mistake for new users. This change logs a warning when the streaming file sink is used ambiguously. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10498: [FLINK-14495][docs] Add documentation for memory control of RocksDB state backend
flinkbot edited a comment on issue #10498: [FLINK-14495][docs] Add documentation for memory control of RocksDB state backend URL: https://github.com/apache/flink/pull/10498#issuecomment-563242008 ## CI report: * dd347fe423313a8bb7c975e3f04ba626fe5dd1b2 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/140238394) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3366) * 2a89d2f7c213fe6137c45c613f6db2062ff1 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/140394883) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3403) * dafe1e061fce4ddda35fc9c4806906ca04c776d8 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146421997) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4641) * f9b43a7e166caeea959525864d7d73bfb92a60e8 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/146789254) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4699) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (FLINK-15789) ActionWatcher.await should throw InterruptedException
[
https://issues.apache.org/jira/browse/FLINK-15789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann reassigned FLINK-15789:
-
Assignee: Yang Wang
> ActionWatcher.await should throw InterruptedException
> -
>
> Key: FLINK-15789
> URL: https://issues.apache.org/jira/browse/FLINK-15789
> Project: Flink
> Issue Type: Sub-task
> Components: Deployment / Kubernetes
>Affects Versions: 1.10.0
>Reporter: Till Rohrmann
>Assignee: Yang Wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0, 1.10.1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The {{ActionWatcher.await}} method should declare that it throws an
> {{InterruptedException}} instead of catching it, not resetting the
> interrupted flag and then throwing a different exception.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10435: [FLINK-13955][runtime] migrate ContinuousFileReaderOperator to the mailbox execution model
flinkbot edited a comment on issue #10435: [FLINK-13955][runtime] migrate ContinuousFileReaderOperator to the mailbox execution model URL: https://github.com/apache/flink/pull/10435#issuecomment-562048103 ## CI report: * d07737bd54c586aa87204c6ac298309f432cc06c Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/139496045) * 1e3e2ff9f65ba15c64fa275fc8567b352eb21c5e Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/139712355) * 58350d6bdc50078b6b15ca70b9d832d3b35aded3 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/139795244) * 287209ff3f521e5fd03df1800ca6425ff6b5a191 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140374178) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3398) * 53161ebd8667e4b01df1e31cf61e38f667c74a32 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/14060) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3479) * b7985f7c1d209fda5c9c05ab1ba16c9ef9e8c5fb Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/140864128) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3539) * 8eff8ec051009be299234ffc8f2afcb6764a1186 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146473490) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4643) * 37198e4d40054bf076b8e53f2ffc24027ee2cc0c Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146551712) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4646) * 28f9eef96938e13005b5106aa89e0a1afdfb4722 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146558623) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4648) * 80bddbc69d17ebb7a202bb12c6d7b4f6124bd59f Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146580700) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4652) * b67aa24b28b0fdf3053def2bebaad56d3b27f64f UNKNOWN * 497f2a70e1c8ff056f8c398167da5cc6b5760c77 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/146624350) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4659) * e830116faf20ea6c889514875c8b07bb77044804 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/146783279) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4698) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] tillrohrmann commented on issue #10964: [FLINK-15806][yarn] Log recommended way how to stop a detached Yarn session cluster
tillrohrmann commented on issue #10964: [FLINK-15806][yarn] Log recommended way how to stop a detached Yarn session cluster URL: https://github.com/apache/flink/pull/10964#issuecomment-580365091 Good point @kl0u. I will update the PR accordingly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (FLINK-15791) Don't use ForkJoinPool#commonPool() for executing asynchronous operations in Fabric8FlinkKubeClient
[
https://issues.apache.org/jira/browse/FLINK-15791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann reassigned FLINK-15791:
-
Assignee: Yang Wang
> Don't use ForkJoinPool#commonPool() for executing asynchronous operations in
> Fabric8FlinkKubeClient
> ---
>
> Key: FLINK-15791
> URL: https://issues.apache.org/jira/browse/FLINK-15791
> Project: Flink
> Issue Type: Sub-task
> Components: Deployment / Kubernetes
>Affects Versions: 1.10.0
>Reporter: Till Rohrmann
>Assignee: Yang Wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0, 1.10.1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We should not use the {{ForkJoinPool#commonPool()}} in order to run
> asynchronous operations in the {{Fabric8FlinkKubeClient}} as it is done
> [here|https://github.com/apache/flink/blob/master/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/Fabric8FlinkKubeClient.java#L315].
> Since we don't know which other component is using this pool, it can be
> quite dangerous to use it as there might be congestion.
> Instead, I propose to provide an explicit I/O {{Executor}} which is used for
> running asynchronous operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15791) Don't use ForkJoinPool#commonPool() for executing asynchronous operations in Fabric8FlinkKubeClient
[
https://issues.apache.org/jira/browse/FLINK-15791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026861#comment-17026861
]
Till Rohrmann commented on FLINK-15791:
---
Before opening PRs it would be good to wait for a committer to assign you to
the issue [~fly_in_gis]. Otherwise we cannot ensure that the PRs get reviewed
and merged in a timely manner.
> Don't use ForkJoinPool#commonPool() for executing asynchronous operations in
> Fabric8FlinkKubeClient
> ---
>
> Key: FLINK-15791
> URL: https://issues.apache.org/jira/browse/FLINK-15791
> Project: Flink
> Issue Type: Sub-task
> Components: Deployment / Kubernetes
>Affects Versions: 1.10.0
>Reporter: Till Rohrmann
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0, 1.10.1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We should not use the {{ForkJoinPool#commonPool()}} in order to run
> asynchronous operations in the {{Fabric8FlinkKubeClient}} as it is done
> [here|https://github.com/apache/flink/blob/master/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/Fabric8FlinkKubeClient.java#L315].
> Since we don't know which other component is using this pool, it can be
> quite dangerous to use it as there might be congestion.
> Instead, I propose to provide an explicit I/O {{Executor}} which is used for
> running asynchronous operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-15646) Configurable K8s context support
[
https://issues.apache.org/jira/browse/FLINK-15646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann reassigned FLINK-15646:
-
Assignee: Canbin Zheng
> Configurable K8s context support
>
>
> Key: FLINK-15646
> URL: https://issues.apache.org/jira/browse/FLINK-15646
> Project: Flink
> Issue Type: Sub-task
> Components: Deployment / Kubernetes
>Affects Versions: 1.10.0
>Reporter: Canbin Zheng
>Assignee: Canbin Zheng
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0, 1.10.1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> One would be forced to first {{kubectl config use-context }} to
> switch to the desired context if working with multiple K8S clusters or having
> multiple K8S "users" for interacting with the specified cluster, so it is an
> important improvement to add an option(kubernetes.context) for configuring
> arbitrary contexts when deploying a Flink cluster. If that option is not
> specified, then the current context in the config file would be used.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15646) Configurable K8s context support
[
https://issues.apache.org/jira/browse/FLINK-15646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026859#comment-17026859
]
Till Rohrmann commented on FLINK-15646:
---
Before opening PRs, please ask a committer to assign you to this ticket
[~felixzheng].
> Configurable K8s context support
>
>
> Key: FLINK-15646
> URL: https://issues.apache.org/jira/browse/FLINK-15646
> Project: Flink
> Issue Type: Sub-task
> Components: Deployment / Kubernetes
>Affects Versions: 1.10.0
>Reporter: Canbin Zheng
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0, 1.10.1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> One would be forced to first {{kubectl config use-context }} to
> switch to the desired context if working with multiple K8S clusters or having
> multiple K8S "users" for interacting with the specified cluster, so it is an
> important improvement to add an option(kubernetes.context) for configuring
> arbitrary contexts when deploying a Flink cluster. If that option is not
> specified, then the current context in the config file would be used.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink-statefun] tzulitai edited a comment on issue #5: [FLINK-15769] [kafka-io] Add configuring startup position for Kafka ingress
tzulitai edited a comment on issue #5: [FLINK-15769] [kafka-io] Add configuring startup position for Kafka ingress URL: https://github.com/apache/flink-statefun/pull/5#issuecomment-580363749 > I was going to ask if this should be configurable via YAML but I'm not really sure the state of those configurations and if that would be out of scope for this ticket. @sjwiesman yes, I'll add YAML support as part of this PR as well (or maybe even a follow-up PR if this one gets a bit too big; they should be fairly independent code-wise). Wanted to open a PR for the current API to have a preview of the behaviour and what to expose before moving on. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10965: [FLINK-15797][k8s] Reduce log noise of Fabric8FlinkKubeClient
tillrohrmann commented on a change in pull request #10965: [FLINK-15797][k8s]
Reduce log noise of Fabric8FlinkKubeClient
URL: https://github.com/apache/flink/pull/10965#discussion_r373087104
##
File path:
flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/Fabric8FlinkKubeClient.java
##
@@ -269,14 +271,14 @@ public void eventReceived(Action action, Pod pod) {
callbackHandler.onDeleted(Collections.singletonList(new
KubernetesPod(flinkConfig, pod)));
break;
default:
- LOG.info("Skip handling {}
event for pod {}", action, pod.getMetadata().getName());
+ LOG.debug("Ignore handling {}
event for pod {}", action, pod.getMetadata().getName());
break;
}
}
@Override
public void onClose(KubernetesClientException e) {
- LOG.error("Pods watcher onClose", e);
+ LOG.debug("The pods watcher is closing.", e);
Review comment:
Maybe @wangyang0918 can tell us why this is ok. If not then this should be
properly forwarded and finally handled. This could then be another subtask of
https://issues.apache.org/jira/browse/FLINK-15788.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink-statefun] tzulitai commented on issue #5: [FLINK-15769] [kafka-io] Add configuring startup position for Kafka ingress
tzulitai commented on issue #5: [FLINK-15769] [kafka-io] Add configuring startup position for Kafka ingress URL: https://github.com/apache/flink-statefun/pull/5#issuecomment-580363749 > I was going to ask if this should be configurable via YAML but I'm not really sure the state of those configurations and if that would be out of scope for this ticket. @sjwiesman yes, I'll add YAML support as part of this PR as well (or maybe even a follow-up PR if this one gets a bit too big; they should be fairly independent code-wise). Wanted to open the current API to have a preview of the behaviour and what to expose before moving on. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink-statefun] sjwiesman commented on issue #5: [FLINK-15769] [kafka-io] Add configuring startup position for Kafka ingress
sjwiesman commented on issue #5: [FLINK-15769] [kafka-io] Add configuring startup position for Kafka ingress URL: https://github.com/apache/flink-statefun/pull/5#issuecomment-580361520 I was going to ask if this should be configurable via YAML but I'm not really sure the state of those configurations and if that would be out of scope for this ticket. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (FLINK-11911) KafkaTopicPartition is not a valid POJO
[ https://issues.apache.org/jira/browse/FLINK-11911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026852#comment-17026852 ] Oleksandr Nitavskyi edited comment on FLINK-11911 at 1/30/20 5:19 PM: -- What if we try to use custom TypeInformationFactory (https://ci.apache.org/projects/flink/flink-docs-stable/dev/types_serialization.html#defining-type-information-using-a-factory), where we can fallback to Kryo if we deserialize old Kryo state? was (Author: oleksandr nitavskyi): Sure, makes sense. What if we try to use custom TypeInformationFactory (https://ci.apache.org/projects/flink/flink-docs-stable/dev/types_serialization.html#defining-type-information-using-a-factory), where we can fallback to Kryo if we deserialize old Kryo state? > KafkaTopicPartition is not a valid POJO > --- > > Key: FLINK-11911 > URL: https://issues.apache.org/jira/browse/FLINK-11911 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: 1.8.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > > KafkaTopicPartition is not a POJO, and therefore it cannot be serialized > efficiently. This is using the KafkaDeserializationSchema. > When enforcing POJO's: > ``` > java.lang.UnsupportedOperationException: Generic types have been disabled in > the ExecutionConfig and type > org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition is > treated as a generic type. > at > org.apache.flink.api.java.typeutils.GenericTypeInfo.createSerializer(GenericTypeInfo.java:86) > at > org.apache.flink.api.java.typeutils.TupleTypeInfo.createSerializer(TupleTypeInfo.java:107) > at > org.apache.flink.api.java.typeutils.TupleTypeInfo.createSerializer(TupleTypeInfo.java:52) > at > org.apache.flink.api.java.typeutils.ListTypeInfo.createSerializer(ListTypeInfo.java:102) > at > org.apache.flink.api.common.state.StateDescriptor.initializeSerializerUnlessSet(StateDescriptor.java:288) > at > org.apache.flink.runtime.state.DefaultOperatorStateBackend.getListState(DefaultOperatorStateBackend.java:289) > at > org.apache.flink.runtime.state.DefaultOperatorStateBackend.getUnionListState(DefaultOperatorStateBackend.java:219) > at > org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.initializeState(FlinkKafkaConsumerBase.java:856) > at > org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178) > at > org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160) > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96) > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:278) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711) > at java.lang.Thread.run(Thread.java:748) > ``` > And in the logs: > ``` > 2019-03-13 16:41:28,217 INFO > org.apache.flink.api.java.typeutils.TypeExtractor - class > org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition > does not contain a setter for field topic > 2019-03-13 16:41:28,221 INFO > org.apache.flink.api.java.typeutils.TypeExtractor - Class class > org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition > cannot be used as a POJO type because not all fields are valid POJO fields, > and must be processed as GenericType. Please read the Flink documentation on > "Data Types & Serialization" for details of the effect on performance. > ``` -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-11911) KafkaTopicPartition is not a valid POJO
[ https://issues.apache.org/jira/browse/FLINK-11911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026852#comment-17026852 ] Oleksandr Nitavskyi commented on FLINK-11911: - Sure, makes sense. What if we try to use custom TypeInformationFactory (https://ci.apache.org/projects/flink/flink-docs-stable/dev/types_serialization.html#defining-type-information-using-a-factory), where we can fallback to Kryo if we deserialize old Kryo state? > KafkaTopicPartition is not a valid POJO > --- > > Key: FLINK-11911 > URL: https://issues.apache.org/jira/browse/FLINK-11911 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: 1.8.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > > KafkaTopicPartition is not a POJO, and therefore it cannot be serialized > efficiently. This is using the KafkaDeserializationSchema. > When enforcing POJO's: > ``` > java.lang.UnsupportedOperationException: Generic types have been disabled in > the ExecutionConfig and type > org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition is > treated as a generic type. > at > org.apache.flink.api.java.typeutils.GenericTypeInfo.createSerializer(GenericTypeInfo.java:86) > at > org.apache.flink.api.java.typeutils.TupleTypeInfo.createSerializer(TupleTypeInfo.java:107) > at > org.apache.flink.api.java.typeutils.TupleTypeInfo.createSerializer(TupleTypeInfo.java:52) > at > org.apache.flink.api.java.typeutils.ListTypeInfo.createSerializer(ListTypeInfo.java:102) > at > org.apache.flink.api.common.state.StateDescriptor.initializeSerializerUnlessSet(StateDescriptor.java:288) > at > org.apache.flink.runtime.state.DefaultOperatorStateBackend.getListState(DefaultOperatorStateBackend.java:289) > at > org.apache.flink.runtime.state.DefaultOperatorStateBackend.getUnionListState(DefaultOperatorStateBackend.java:219) > at > org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.initializeState(FlinkKafkaConsumerBase.java:856) > at > org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178) > at > org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160) > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96) > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:278) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711) > at java.lang.Thread.run(Thread.java:748) > ``` > And in the logs: > ``` > 2019-03-13 16:41:28,217 INFO > org.apache.flink.api.java.typeutils.TypeExtractor - class > org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition > does not contain a setter for field topic > 2019-03-13 16:41:28,221 INFO > org.apache.flink.api.java.typeutils.TypeExtractor - Class class > org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition > cannot be used as a POJO type because not all fields are valid POJO fields, > and must be processed as GenericType. Please read the Flink documentation on > "Data Types & Serialization" for details of the effect on performance. > ``` -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] sjwiesman commented on a change in pull request #10498: [FLINK-14495][docs] Add documentation for memory control of RocksDB state backend
sjwiesman commented on a change in pull request #10498: [FLINK-14495][docs] Add documentation for memory control of RocksDB state backend URL: https://github.com/apache/flink/pull/10498#discussion_r373083295 ## File path: docs/ops/state/large_state_tuning.md ## @@ -210,6 +211,71 @@ and not from the JVM. Any memory you assign to RocksDB will have to be accounted of the TaskManagers by the same amount. Not doing that may result in YARN/Mesos/etc terminating the JVM processes for allocating more memory than configured. +### Bounding RocksDB Memory Usage + +RocksDB allocates native memory outside of the JVM, which could lead the process to exceed the total memory budget. +This can be especially problematic in containerized environments such as Kubernetes that kill processes who exceed their memory budgets. +Flink limit total memory usage of RocksDB instance(s) per slot by leveraging shareable [cache](https://github.com/facebook/rocksdb/wiki/Block-Cache) +and [write buffer manager](https://github.com/facebook/rocksdb/wiki/Write-Buffer-Manager) among all instances in a single slot by default. +The shared cache will place an upper limit on the [three components](https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB) that use the majority of memory +when RocksDB is deployed as a state backend: block cache, index and bloom filters, and MemTables. +This feature is enabled by default and could be controlled by two ways: + - Integrate with managed memory of task manager: turn `state.backend.rocksdb.memory.managed` as true. If so, RocksDB state backend will use the managed memory budget of the task slot to set the capacity of that shared cache object. + This operation is enabled by default, which means Flink would always choose to integrate RocksDB memory usage with the managed memory first. Review comment: Yes, the documentation is versioned so we don’t need to specify versions inline. Just say: This feature is enabled by default along with managed memory. Flink will use the managed memory budget as the per-slot memory limit for RocksDB state backend(s). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink-statefun] sjwiesman commented on a change in pull request #5: [FLINK-15769] [kafka-io] Add configuring startup position for Kafka ingress
sjwiesman commented on a change in pull request #5: [FLINK-15769] [kafka-io] Add configuring startup position for Kafka ingress URL: https://github.com/apache/flink-statefun/pull/5#discussion_r373081346 ## File path: statefun-kafka-io/src/main/java/org/apache/flink/statefun/sdk/kafka/KafkaIngressBuilder.java ## @@ -39,8 +39,7 @@ private Class> deserializerClass; private String kafkaAddress; private KafkaIngressAutoResetPosition autoResetPosition = KafkaIngressAutoResetPosition.LATEST; - private KafkaIngressStartupPosition startupPosition = - KafkaIngressStartupPosition.fromGroupOffsets(); + private KafkaIngressStartupPosition startupPosition = KafkaIngressStartupPosition.fromLatest(); Review comment: Sounds good to me, I don't have a strong opinion on this one This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10886: [FLINK-15623][checkstyle] Exclude target directory
tillrohrmann commented on a change in pull request #10886: [FLINK-15623][checkstyle] Exclude target directory URL: https://github.com/apache/flink/pull/10886#discussion_r373076627 ## File path: tools/maven/suppressions.xml ## @@ -32,13 +32,6 @@ under the License. - - - Review comment: Thanks for the clarification. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink-statefun] tzulitai commented on a change in pull request #5: [FLINK-15769] [kafka-io] Add configuring startup position for Kafka ingress
tzulitai commented on a change in pull request #5: [FLINK-15769] [kafka-io] Add
configuring startup position for Kafka ingress
URL: https://github.com/apache/flink-statefun/pull/5#discussion_r373075875
##
File path:
statefun-kafka-io/src/main/java/org/apache/flink/statefun/sdk/kafka/KafkaIngressStartupPosition.java
##
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.statefun.sdk.kafka;
+
+import java.util.Date;
+import java.util.Map;
+
+/**
+ * Position for the ingress to start consuming Kafka partitions.
+ */
+@SuppressWarnings("WeakerAccess, unused")
+public class KafkaIngressStartupPosition {
Review comment:
Good point! Will add this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink-statefun] tzulitai commented on issue #5: [FLINK-15769] [kafka-io] Add configuring startup position for Kafka ingress
tzulitai commented on issue #5: [FLINK-15769] [kafka-io] Add configuring startup position for Kafka ingress URL: https://github.com/apache/flink-statefun/pull/5#issuecomment-580354457 > Can you add something to the docs about this? It can just be one line saying the starting positions are configurable and then a list of the options. Will do! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink-statefun] tzulitai commented on a change in pull request #5: [FLINK-15769] [kafka-io] Add configuring startup position for Kafka ingress
tzulitai commented on a change in pull request #5: [FLINK-15769] [kafka-io] Add configuring startup position for Kafka ingress URL: https://github.com/apache/flink-statefun/pull/5#discussion_r373075738 ## File path: statefun-kafka-io/src/main/java/org/apache/flink/statefun/sdk/kafka/KafkaIngressBuilder.java ## @@ -39,8 +39,7 @@ private Class> deserializerClass; private String kafkaAddress; private KafkaIngressAutoResetPosition autoResetPosition = KafkaIngressAutoResetPosition.LATEST; - private KafkaIngressStartupPosition startupPosition = - KafkaIngressStartupPosition.fromGroupOffsets(); + private KafkaIngressStartupPosition startupPosition = KafkaIngressStartupPosition.fromLatest(); Review comment: As for latest v.s. earliest, for a typical Kafka consumer (non-Flink/StateFun), when consuming from a topic that has no previous record of committed offsets, that's the default behaviour AFAIK. See the section about `auto.offset.reset` in the Kafka docs here: https://docs.confluent.io/current/clients/consumer.html#id1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink-statefun] tzulitai commented on a change in pull request #5: [FLINK-15769] [kafka-io] Add configuring startup position for Kafka ingress
tzulitai commented on a change in pull request #5: [FLINK-15769] [kafka-io] Add configuring startup position for Kafka ingress URL: https://github.com/apache/flink-statefun/pull/5#discussion_r373074470 ## File path: statefun-kafka-io/src/main/java/org/apache/flink/statefun/sdk/kafka/KafkaIngressBuilder.java ## @@ -39,8 +39,7 @@ private Class> deserializerClass; private String kafkaAddress; private KafkaIngressAutoResetPosition autoResetPosition = KafkaIngressAutoResetPosition.LATEST; - private KafkaIngressStartupPosition startupPosition = - KafkaIngressStartupPosition.fromGroupOffsets(); + private KafkaIngressStartupPosition startupPosition = KafkaIngressStartupPosition.fromLatest(); Review comment: The `FlinkKafkaConsumer` actually uses committed group offsets as the default: https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumerBase.java#L159 The reason we (after a discussion with @igalshilman) chose to change this to latest, is because group offsets startup require that the user must set the consumer group id. We wanted to go with a default that requires the minimal setup to get started with using the ingress. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
