[jira] [Created] (FLINK-22130) Test row based operations in Python Table API
Huang Xingbo created FLINK-22130: Summary: Test row based operations in Python Table API Key: FLINK-22130 URL: https://issues.apache.org/jira/browse/FLINK-22130 Project: Flink Issue Type: Test Components: API / Python Affects Versions: 1.13.0 Reporter: Huang Xingbo Assignee: Huang Xingbo Fix For: 1.13.0 It includes but not limited to the following testing items: * map/flat_map/aggregate/flat_aggregate works well * performance test -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22131) Fix the bug of general udf and pandas udf chained together in map operation
Huang Xingbo created FLINK-22131: Summary: Fix the bug of general udf and pandas udf chained together in map operation Key: FLINK-22131 URL: https://issues.apache.org/jira/browse/FLINK-22131 Project: Flink Issue Type: Sub-task Components: API / Python Affects Versions: 1.13.0 Reporter: Huang Xingbo Assignee: Huang Xingbo Fix For: 1.13.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22132) Test unaligned checkpoints rescaling manually on a real cluster
Piotr Nowojski created FLINK-22132: -- Summary: Test unaligned checkpoints rescaling manually on a real cluster Key: FLINK-22132 URL: https://issues.apache.org/jira/browse/FLINK-22132 Project: Flink Issue Type: Test Components: Runtime / Checkpointing Affects Versions: 1.13.0 Reporter: Piotr Nowojski Fix For: 1.13.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22133) SplitEmumerator does not provide checkpoint id in snapshot
Brian Zhou created FLINK-22133: -- Summary: SplitEmumerator does not provide checkpoint id in snapshot Key: FLINK-22133 URL: https://issues.apache.org/jira/browse/FLINK-22133 Project: Flink Issue Type: Bug Components: Connectors / Common Affects Versions: 1.12.0 Reporter: Brian Zhou In ExternallyInducedSource API, the checkpoint trigger exposes the checkpoint Id for the external client to identify the checkpoint. However, in the FLIP-27 source, the SplitEmumerator::snapshot() is an no-arg method. The connector cannot track the checkpoint ID from Flink -- This message was sent by Atlassian Jira (v8.3.4#803005)
[DISCUSS] Backport FLIP-27 Kafka source connector fixes with API change to release-1.12.
Hi folks, I'd like to start a discussion thread about backporting some FLIP-27 Kafka source connector fixes to release-1.12. These fixes include some API changes and thus needs a public discussion. The tickets in question are following: https://issues.apache.org/jira/browse/FLINK-20379 https://issues.apache.org/jira/browse/FLINK-20114 https://issues.apache.org/jira/browse/FLINK-21817 Without these fixes, the FLIP-27 Kafka source in release-1.12 is not really usable, and the API changes only affect the Kafka Source. So it seems breaking the API in this case is still worthwhile. It would be good to see what others think. Thanks, Jiangjie (Becket) Qin
[jira] [Created] (FLINK-22134) Test the reactive mode
Till Rohrmann created FLINK-22134: - Summary: Test the reactive mode Key: FLINK-22134 URL: https://issues.apache.org/jira/browse/FLINK-22134 Project: Flink Issue Type: Task Components: Runtime / Coordination Affects Versions: 1.13.0 Reporter: Till Rohrmann Fix For: 1.13.0 The newly introduced reactive mode (FLINK-10407) allows Flink to make use of newly arriving resources while the job is running. The feature documentation with the current set of limitations can be found here [1]. In order to test this new feature I recommend to follow the documentation and to try it out wrt the stated limitations. Everything which is not explicitly contained in the set of limitations should work. [1] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/elastic_scaling/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22135) Test the adaptive scheduler
Till Rohrmann created FLINK-22135:
-
Summary: Test the adaptive scheduler
Key: FLINK-22135
URL: https://issues.apache.org/jira/browse/FLINK-22135
Project: Flink
Issue Type: Task
Components: Runtime / Coordination
Affects Versions: 1.13.0
Reporter: Till Rohrmann
Fix For: 1.13.0
With FLINK-21075, we introduced a new scheduler type which first waits for
resources before deciding on the actual parallelism. This allows to continue
executing a job even if the cluster loses a {{TaskManager}} permanently. We
should test that this feature works as described by its documentation [1] (w/o
using the reactive mode).
[1]
https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/elastic_scaling/#adaptive-scheduler
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-22136) Device application for unaligned checkpoint test on cluster
Arvid Heise created FLINK-22136: --- Summary: Device application for unaligned checkpoint test on cluster Key: FLINK-22136 URL: https://issues.apache.org/jira/browse/FLINK-22136 Project: Flink Issue Type: Sub-task Reporter: Arvid Heise To test unaligned checkpoints, we should use a few different applications that use different features: * Mixing forward/rescale channels with keyby or other shuffle operations * Unions * 2 or n-ary operators * Associated state ((keyed) process function) * Correctness verifications The sinks should not be mocked but rather should be able to induce a fair amount of backpressure into the system. Quite possibly, it would be a good idea to have a way to add more backpressure to the sink by running the respective system on the cluster and be able to add/remove parallel instances. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22137) Execute unaligned checkpoint test on a cluster
Arvid Heise created FLINK-22137: --- Summary: Execute unaligned checkpoint test on a cluster Key: FLINK-22137 URL: https://issues.apache.org/jira/browse/FLINK-22137 Project: Flink Issue Type: Sub-task Reporter: Arvid Heise Start application and at some point cancel/induce failure, the user needs to restart from a retained checkpoint with * lower * same * higher degree of parallelism. To enable unaligned checkpoints, set * execution.checkpointing.unaligned: true * execution.checkpointing.alignment-timeout to 0s, 10s, 1min (for high backpressure) The primary objective is to check if all data is recovered properly and if the semantics is correct (does state match input?). The secondary objective is to check if Flink UI shows the information correctly: * unaligned checkpoint enabled on job level * timeout on job level * for each checkpoint, if it's unaligned or not; how much data was written -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22138) Better support structured types as toDataStream output
Timo Walther created FLINK-22138:
Summary: Better support structured types as toDataStream output
Key: FLINK-22138
URL: https://issues.apache.org/jira/browse/FLINK-22138
Project: Flink
Issue Type: Sub-task
Components: Table SQL / API
Reporter: Timo Walther
Assignee: Timo Walther
There are still some minor barriers that prevent using {{toDataStream}} to its
full extent.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [DISCUSS] Backport FLIP-27 Kafka source connector fixes with API change to release-1.12.
Hi Becket, If I remember correctly, then we deliberately not documented the Kafka connector in the 1.12 release. Hence, from this point there should be no need to backport any fixes because users are not aware of this feature. On the other hand this also means that we should be able to break anything we want to. Consequently, backporting these fixes should be possible. The question would probably be whether we want to ship new features with a bug fix release. Do we know of any users who want to use the new Kafka source, are using the 1.12 version and cannot upgrade to 1.13 once it is released? If this is the case, then this could be an argument for shipping this feature with a bug fix release. If not, then we could save some work by not backporting it. Cheers, Till On Wed, Apr 7, 2021 at 10:43 AM Becket Qin wrote: > Hi folks, > > I'd like to start a discussion thread about backporting some FLIP-27 Kafka > source connector fixes to release-1.12. These fixes include some API > changes and thus needs a public discussion. > > The tickets in question are following: > https://issues.apache.org/jira/browse/FLINK-20379 > https://issues.apache.org/jira/browse/FLINK-20114 > https://issues.apache.org/jira/browse/FLINK-21817 > > Without these fixes, the FLIP-27 Kafka source in release-1.12 is not really > usable, and the API changes only affect the Kafka Source. So it seems > breaking the API in this case is still worthwhile. > > It would be good to see what others think. > > Thanks, > > Jiangjie (Becket) Qin >
[jira] [Created] (FLINK-22139) Flink Jobmanager & Task Manger logs are not writing to the logs files
Bhagi created FLINK-22139:
-
Summary: Flink Jobmanager & Task Manger logs are not writing to
the logs files
Key: FLINK-22139
URL: https://issues.apache.org/jira/browse/FLINK-22139
Project: Flink
Issue Type: Bug
Components: Deployment / Kubernetes
Affects Versions: 1.12.2
Environment: on kubernetes flink standalone deployment with jobmanager
HA is enabled.
Reporter: Bhagi
Hi Team,
I am submitting the jobs and restarting the job manager and task manager pods..
Log files are generating with the name task manager and job manager.
but job manager & task manager log file size is '0', i am not sure any
configuration missed..why logs are not writing to their log files..
# Task Manager pod###
flink@flink-taskmanager-85b6585b7-hhgl7:~$ ls -lart log/
total 0
-rw-r--r-- 1 flink flink 0 Apr 7 09:35
flink--taskexecutor-0-flink-taskmanager-85b6585b7-hhgl7.log
flink@flink-taskmanager-85b6585b7-hhgl7:~$
### Jobmanager pod Logs #
flink@flink-jobmanager-f6db89b7f-lq4ps:~$
-rw-r--r-- 1 7148739 flink 0 Apr 7 06:36
flink--standalonesession-0-flink-jobmanager-f6db89b7f-gtkx5.log
-rw-r--r-- 1 7148739 flink 0 Apr 7 06:36
flink--standalonesession-0-flink-jobmanager-f6db89b7f-wnrfm.log
-rw-r--r-- 1 7148739 flink 0 Apr 7 06:37
flink--standalonesession-0-flink-jobmanager-f6db89b7f-2b2fs.log
-rw-r--r-- 1 7148739 flink 0 Apr 7 06:37
flink--standalonesession-0-flink-jobmanager-f6db89b7f-7kdhh.log
-rw-r--r-- 1 7148739 flink 0 Apr 7 09:35
flink--standalonesession-0-flink-jobmanager-f6db89b7f-twhkt.log
drwxrwxrwx 2 7148739 flink35 Apr 7 09:35 .
-rw-r--r-- 1 7148739 flink 0 Apr 7 09:35
flink--standalonesession-0-flink-jobmanager-f6db89b7f-lq4ps.log
flink@flink-jobmanager-f6db89b7f-lq4ps:~$
I configured log4j.properties for flink
log4j.properties: |+
monitorInterval=30
rootLogger.level = INFO
rootLogger.appenderRef.file.ref = MainAppender
logger.flink.name = org.apache.flink
logger.flink.level = INFO
logger.akka.name = akka
logger.akka.level = INFO
appender.main.name = MainAppender
appender.main.type = RollingFile
appender.main.append = true
appender.main.fileName = ${sys:log.file}
appender.main.filePattern = ${sys:log.file}.%i
appender.main.layout.type = PatternLayout
appender.main.layout.pattern = %d{-MM-dd HH:mm:ss,SSS} %-5p %-60c %x -
%m%n
appender.main.policies.type = Policies
appender.main.policies.size.type = SizeBasedTriggeringPolicy
appender.main.policies.size.size = 100MB
appender.main.policies.startup.type = OnStartupTriggeringPolicy
appender.main.strategy.type = DefaultRolloverStrategy
appender.main.strategy.max = ${env:MAX_LOG_FILE_NUMBER:-10}
logger.netty.name =
org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
logger.netty.level = OFF
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[ANNOUNCE] Apache Flink-shaded 13.0 released
The Apache Flink community is very happy to announce the release of Apache Flink-shaded 13.0. The flink-shaded project contains a number of shaded dependencies for Apache Flink. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The release is available for download at: https://flink.apache.org/downloads.html We would like to thank all contributors of the Apache Flink community who made this release possible! Regards, Release Manager
[jira] [Created] (FLINK-22140) Test the unified binary savepoint
Dawid Wysakowicz created FLINK-22140: Summary: Test the unified binary savepoint Key: FLINK-22140 URL: https://issues.apache.org/jira/browse/FLINK-22140 Project: Flink Issue Type: Task Components: Runtime / State Backends Affects Versions: 1.13.0 Reporter: Dawid Wysakowicz Fix For: 1.13.0 With https://issues.apache.org/jira/browse/FLINK-20976 we introduced a unified binary savepoint format which should let you switch between different state backends. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22141) Manually test exactly-once JDBC sink
Roman Khachatryan created FLINK-22141: - Summary: Manually test exactly-once JDBC sink Key: FLINK-22141 URL: https://issues.apache.org/jira/browse/FLINK-22141 Project: Flink Issue Type: Test Components: Connectors / JDBC Reporter: Roman Khachatryan Fix For: 1.13.0 In FLINK-15578, an API and its implementation were added to JDBC connector to support exactly-once semantics for sinks. The implementation uses JDBC XA transactions. The scope of this task is to make sure: # The feature is well-documented # The API is reasonably easy to use # The implementation works as expected ## normal case: database is updated on checkpointing ## failure and recovery case: no duplicates inserted, no records skipped ## several DBs: postgressql, mssql, oracle (mysql has a known issue: FLINK-21743) ## concurrent checkpoints > 1, DoP > 1 # Logging is meaningful -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22142) Remove console logging for Kafka connector for AZP runs
Till Rohrmann created FLINK-22142: - Summary: Remove console logging for Kafka connector for AZP runs Key: FLINK-22142 URL: https://issues.apache.org/jira/browse/FLINK-22142 Project: Flink Issue Type: Improvement Components: Connectors / Kafka, Test Infrastructure Affects Versions: 1.12.2, 1.13.0 Reporter: Till Rohrmann Assignee: Till Rohrmann Fix For: 1.13.0 For the Kafka connector we do log to the console. These logging statements clutter the AZP output considerably. I propose to remove this logic. Moreover, we still have some DEBUG logging for FLINK-16383 which has been fixed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22143) Flink returns less rows than expected when using limit in SQL
Peng Yu created FLINK-22143:
---
Summary: Flink returns less rows than expected when using limit in
SQL
Key: FLINK-22143
URL: https://issues.apache.org/jira/browse/FLINK-22143
Project: Flink
Issue Type: Bug
Components: Table SQL / Runtime
Affects Versions: 1.13.0
Reporter: Peng Yu
Fix For: 1.13.0
Flink's blink runtime returns less rows than expected when querying Hive tables
with limit.
{code:java}
// sql
select i_item_sk from tpcds_1g_snappy.item limit 5000;
{code}
Above query will return only *4998* lines in some cases.
This problem can be re-produced on below conditions:
# A Hive table with parquet format.
# Running SQL with limit using blink planner since Flink version 1.12.0
# The input table is small. (With only 1 data file in which there is only 1
row group, e.g. 1 GB of TPCDS benchmark data)
# The requested count of lines by `limit` is above the batch size (2048 by
default)
After investigation, a bug is found lying in the *LimitableBulkFormat* class.
In this class, for each batch, *numRead* will be increased *1* more than actual
count of rows returned by reader.readBatch().
The reason is that *numRead* get increased even when next() reaches then end of
current batch.
If there is only 1 input split, no more lines will be merged into the final
result.
As a result, less lines will be returned by Flink.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[ANNOUNCE] Release 1.13 branch
Hi committers! I'd like to remind that we agreed we won't cut off the 1.13 branch until we are a bit more confident with the stability of the master branch. Therefore I'd kindly ask not to push the release-1.13 branch yet. I did push that branch accidentally when creating the rc0, but I reported that to the ASF infra and removed the branch in the morning. Unfortunately the branch has been pushed again since then ;). That's why I am writing this email. I'll remove the branch again and we will most probably perform a proper cut early next week. Does anyone have a different opinion? Would anyone benefit from cutting off the branch already? If so, please let me know. Best, Dawid OpenPGP_signature Description: OpenPGP digital signature
[jira] [Created] (FLINK-22144) Test display last n exceptions/causes for job restarts in Web UI
Till Rohrmann created FLINK-22144: - Summary: Test display last n exceptions/causes for job restarts in Web UI Key: FLINK-22144 URL: https://issues.apache.org/jira/browse/FLINK-22144 Project: Flink Issue Type: Task Components: Runtime / Coordination Affects Versions: 1.13.0 Reporter: Till Rohrmann Fix For: 1.13.0 This is the testing task for FLINK-6042. We should test whether the root causes for multiple restarts are properly displayed in the web UI. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSS] Backport FLIP-27 Kafka source connector fixes with API change to release-1.12.
Hi, Thanks for fixing the new KafkaSource issues. I'm interested in using these fixes with 1.12 for experimental purposes. +1 for backporting. 1.12 is the current stable release and users who would like to try the FLIP-27 sources are likely to use that release. Thomas On Wed, Apr 7, 2021 at 2:50 AM Till Rohrmann wrote: > Hi Becket, > > If I remember correctly, then we deliberately not documented the Kafka > connector in the 1.12 release. Hence, from this point there should be no > need to backport any fixes because users are not aware of this feature. > > On the other hand this also means that we should be able to break anything > we want to. Consequently, backporting these fixes should be possible. > > The question would probably be whether we want to ship new features with a > bug fix release. Do we know of any users who want to use the new Kafka > source, are using the 1.12 version and cannot upgrade to 1.13 once it is > released? If this is the case, then this could be an argument for shipping > this feature with a bug fix release. If not, then we could save some work > by not backporting it. > > Cheers, > Till > > On Wed, Apr 7, 2021 at 10:43 AM Becket Qin wrote: > > > Hi folks, > > > > I'd like to start a discussion thread about backporting some FLIP-27 > Kafka > > source connector fixes to release-1.12. These fixes include some API > > changes and thus needs a public discussion. > > > > The tickets in question are following: > > https://issues.apache.org/jira/browse/FLINK-20379 > > https://issues.apache.org/jira/browse/FLINK-20114 > > https://issues.apache.org/jira/browse/FLINK-21817 > > > > Without these fixes, the FLIP-27 Kafka source in release-1.12 is not > really > > usable, and the API changes only affect the Kafka Source. So it seems > > breaking the API in this case is still worthwhile. > > > > It would be good to see what others think. > > > > Thanks, > > > > Jiangjie (Becket) Qin > > >
Re: [VOTE] Apache Flink Stateful Functions 3.0.0, release candidate #1
This jar contains a com/google/protobuf classfile, which is not declared in any NOTICE file (and doesn't ship the license file of protobuf): https://repository.apache.org/content/repositories/orgapacheflink-1415/org/apache/flink/statefun-flink-common/3.0.0/statefun-flink-common-3.0.0.jar I fear that this could be a blocker for the release? Otherwise, I did the following check: - src distribution looks fine: No binaries, js related files are declared (the copyright in the NOTICE file could be updated to 2021, but that's not a blocker) On Fri, Apr 2, 2021 at 8:29 AM Yu Li wrote: > +1 (binding) > > Checked sums and signatures: OK > Checked RAT and end-to-end tests: OK > Checked version in pom/README/setup.py files: OK > Checked release notes: OK > Checked docker PR: OK > > Thanks for driving this release, Gordon! > > Best Regards, > Yu > > > On Fri, 2 Apr 2021 at 09:22, Seth Wiesman wrote: > > > +1 (non-binding) > > > > - Built from source and executed end to end tests > > - Checked licenses and signatures > > - Deployed remote Java SDK to gke cluster > > - Took savepoint and statefully rescaled > > > > Seth > > > > On Thu, Apr 1, 2021 at 9:05 AM Konstantin Knauf > wrote: > > > > > +1 (non-binding) > > > > > > - mvn clean install -Prun-e2e-tests (java 8) from source > > > - python3 -m unittest tests > > > - spin up Statefun Cluster on EKS with an image built from the > > Dockerfiles > > > of [1] > > > - run Python & Java Greeter example on AWS Lambda > > > - read through documentation (opened [2] to fix some tpoys) > > > > > > [1] https://github.com/apache/flink-statefun-docker/pull/13 > > > [2] https://github.com/apache/flink-statefun/pull/219 > > > > > > On Thu, Apr 1, 2021 at 6:46 AM Tzu-Li (Gordon) Tai < > tzuli...@apache.org> > > > wrote: > > > > > > > +1 (binding) > > > > > > > > - verified signatures and hashes > > > > - NOTICE and LICENSE files in statefun-flink-distribution, > > > > statefun-protobuf-shaded, and statefun-sdk-java looks sane > > > > - maven clean install -Prun-e2e-tests (java 8) from source > > > > - ran all examples and tutorials in apache/flink-statefun-playground > > with > > > > the new artifacts > > > > - Ran my SDK verifier utility [1] against the new Java and Python > SDKs. > > > > > > > > Cheers, > > > > Gordon > > > > > > > > [1] https://github.com/tzulitai/statefun-sdk-verifier > > > > > > > > On Wed, Mar 31, 2021 at 8:50 PM Igal Shilman > > wrote: > > > > > > > > > Thanks Gordon for managing the release! > > > > > > > > > > +1 (non binding) from my side: > > > > > > > > > > Here are the results of my testing: > > > > > - verified the signatures > > > > > - verified that the source distribution doesn't contain any binary > > > files > > > > > - ran mvn clean install -Prun-e2e-tests with java8 > > > > > - ran the smoke test that sends 100 million messages locally. > > > > > - extended the smoke test to include the remote sdks (1 function in > > the > > > > > Java SDK, 1 function in the Python SDK), and it passes. > > > > > - deployed to kubernetes with minio as an S3 replacement. > > > > > > > > > > > > > > > On Tue, Mar 30, 2021 at 12:29 PM Tzu-Li (Gordon) Tai < > > > > tzuli...@apache.org> > > > > > wrote: > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > Please review and vote on the release candidate #1 for the > version > > > > 3.0.0 > > > > > of > > > > > > Apache Flink Stateful Functions, as follows: > > > > > > [ ] +1, Approve the release > > > > > > [ ] -1, Do not approve the release (please provide specific > > comments) > > > > > > > > > > > > **Testing Guideline** > > > > > > > > > > > > You can find here [1] a page in the project wiki on instructions > > for > > > > > > testing. > > > > > > To cast a vote, it is not necessary to perform all listed checks, > > > > > > but please mention which checks you have performed when voting. > > > > > > > > > > > > **Release Overview** > > > > > > > > > > > > As an overview, the release consists of the following: > > > > > > a) Stateful Functions canonical source distribution, to be > deployed > > > to > > > > > the > > > > > > release repository at dist.apache.org > > > > > > b) Stateful Functions Python SDK distributions to be deployed to > > PyPI > > > > > > c) Maven artifacts to be deployed to the Maven Central Repository > > > > > > d) New Dockerfiles for the release > > > > > > > > > > > > **Staging Areas to Review** > > > > > > > > > > > > The staging areas containing the above mentioned artifacts are as > > > > > follows, > > > > > > for your review: > > > > > > * All artifacts for a) and b) can be found in the corresponding > dev > > > > > > repository at dist.apache.org [2] > > > > > > * All artifacts for c) can be found at the Apache Nexus > Repository > > > [3] > > > > > > > > > > > > All artifacts are signed with the key > > > > > > 1C1E2394D3194E1944613488F320986D35C33D6A [4] > > > > > > > > > > > > Other links for your review: > > > > > > * JIRA release notes [5] > > > > > > * source code ta
Re: Zigzag shape in TM JVM used memory
Hi, Piotr Thanks for replying. I asked this because such a pattern might imply memory oversubscription. For example, I tuned down the memory of one app from heap 2.63GB to 367MB and the job still runs fine: before: https://drive.google.com/file/d/1o8k9Vv3yb5gXITi4GvmlXMteQcRfmOhr/view?usp=sharing after: https://drive.google.com/file/d/1wNTHBT8aSJaAmL1rVY8jUkdp-G5znnMN/view?usp=sharing What's the best practice for tuning Flink job memory? 1. What’s a good start point users should try first? 2. How to make progress? e.g. flink application Foo currently encountered error OOM: java heap space. Where to move next? simply bump up taskmananger.memory? or just increase heap? 3. What’s the final state? Job running fine and ensuring XYZ headroom in each memory component? Best Lu On Tue, Apr 6, 2021 at 12:26 AM Piotr Nowojski wrote: > Hi, > > this should be posted on the user mailing list not the dev. > > Apart from that, this looks like normal/standard behaviour of JVM, and has > very little to do with Flink. Garbage Collector (GC) is kicking in when > memory usage is approaching some threshold: > https://www.google.com/search?q=jvm+heap+memory+usage&tbm=isch > > Piotrek > > > pon., 5 kwi 2021 o 22:54 Lu Niu napisał(a): > > > Hi, > > > > we need to update our email system then :) . Here are the links: > > > > > > > https://drive.google.com/file/d/1lZ5_P8_NqsN1JeLzutGj4DxkyWJN75mR/view?usp=sharing > > > > > > > https://drive.google.com/file/d/1J6c6rQJwtDp1moAGlvQyLQXTqcuG4HjL/view?usp=sharing > > > > > > > https://drive.google.com/file/d/1-R2KzsABC471AEjkF5qTm5O3V47cpbBV/view?usp=sharing > > > > All are DataStream job. > > > > Best > > Lu > > > > On Sun, Apr 4, 2021 at 9:17 PM Yun Gao wrote: > > > > > > > > Hi Lu, > > > > > > The image seems not be able to shown due to the mail server limitation, > > > could you upload it somewhere and paste the link here ? > > > > > > Logically, I think zigzag usually due to there are some small object > get > > > created and eliminated soon in the heap. Are you running a SQL job or a > > > DataStream job ? > > > > > > Best, > > > Yun > > > > > > -- > > > Sender:Lu Niu > > > Date:2021/04/05 12:06:24 > > > Recipient:dev@flink.apache.org > > > Theme:Zigzag shape in TM JVM used memory > > > > > > Hi, Flink dev > > > > > > We observed that the TM JVM used memory metric shows zigzag shape among > > > lots of our applications, although these applications are quite > different > > > in business logic. The upper bound is close to the max heap size. Is > this > > > expected in flink application? Or does flink internally > > > aggressively pre-allocate memory? > > > > > > app1 > > > [image: Screen Shot 2021-04-04 at 8.46.45 PM.png] > > > app2 > > > [image: Screen Shot 2021-04-04 at 8.45.35 PM.png] > > > app3 > > > [image: Screen Shot 2021-04-04 at 8.43.53 PM.png] > > > > > > Best > > > Lu > > > > > > > > >
Re: [DISCUSS] Backport FLIP-27 Kafka source connector fixes with API change to release-1.12.
Thanks for the comment, Till and Thomas. As far as I know there are some users who have just upgraded their Flink version from 1.8 / 1.9 to Flink 1.12 and might not upgrade the version in 6 months or more. There are also some organizations that have the strategy of not running the latest version of a project, but only the second latest version with bug fixes. So those users may still benefit from the backport. However, arguably the old Kafka source is there anyways in 1.12, so they should not be blocked on having the new source. I am leaning towards backporting the fixes mainly because this way we might have more users migrating to the new Source and provide feedback. It will take some time for the users to pick up 1.13, especially for the users running Flink at large scale. So backporting the fixes to 1.12 would help get the new source to be used sooner. Thanks, Jiangjie (Becket) Qin On Thu, Apr 8, 2021 at 12:40 AM Thomas Weise wrote: > Hi, > > Thanks for fixing the new KafkaSource issues. > > I'm interested in using these fixes with 1.12 for experimental purposes. > > +1 for backporting. 1.12 is the current stable release and users who would > like to try the FLIP-27 sources are likely to use that release. > > Thomas > > On Wed, Apr 7, 2021 at 2:50 AM Till Rohrmann wrote: > > > Hi Becket, > > > > If I remember correctly, then we deliberately not documented the Kafka > > connector in the 1.12 release. Hence, from this point there should be no > > need to backport any fixes because users are not aware of this feature. > > > > On the other hand this also means that we should be able to break > anything > > we want to. Consequently, backporting these fixes should be possible. > > > > The question would probably be whether we want to ship new features with > a > > bug fix release. Do we know of any users who want to use the new Kafka > > source, are using the 1.12 version and cannot upgrade to 1.13 once it is > > released? If this is the case, then this could be an argument for > shipping > > this feature with a bug fix release. If not, then we could save some work > > by not backporting it. > > > > Cheers, > > Till > > > > On Wed, Apr 7, 2021 at 10:43 AM Becket Qin wrote: > > > > > Hi folks, > > > > > > I'd like to start a discussion thread about backporting some FLIP-27 > > Kafka > > > source connector fixes to release-1.12. These fixes include some API > > > changes and thus needs a public discussion. > > > > > > The tickets in question are following: > > > https://issues.apache.org/jira/browse/FLINK-20379 > > > https://issues.apache.org/jira/browse/FLINK-20114 > > > https://issues.apache.org/jira/browse/FLINK-21817 > > > > > > Without these fixes, the FLIP-27 Kafka source in release-1.12 is not > > really > > > usable, and the API changes only affect the Kafka Source. So it seems > > > breaking the API in this case is still worthwhile. > > > > > > It would be good to see what others think. > > > > > > Thanks, > > > > > > Jiangjie (Becket) Qin > > > > > >
[jira] [Created] (FLINK-22145) CheckStyle for scala not work
MaChengLong created FLINK-22145: --- Summary: CheckStyle for scala not work Key: FLINK-22145 URL: https://issues.apache.org/jira/browse/FLINK-22145 Project: Flink Issue Type: Improvement Components: API / Scala Reporter: MaChengLong I followed this doc [https://ci.apache.org/projects/flink/flink-docs-master/zh/docs/flinkdev/ide_setup/] to setup scala code style format but when i format exists scala code with code->reformat code some original scala code style was broken,it seems CheckStyle for scala( tools/maven/scalastyle-config.xml was placed to .idea/) not work -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22146) Migrate StateBackend related Chinese docs to latest state backend
Yun Tang created FLINK-22146: Summary: Migrate StateBackend related Chinese docs to latest state backend Key: FLINK-22146 URL: https://issues.apache.org/jira/browse/FLINK-22146 Project: Flink Issue Type: Sub-task Components: Documentation, Runtime / State Backends Reporter: Yun Tang Fix For: 1.13.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [VOTE] Apache Flink Stateful Functions 3.0.0, release candidate #1
@Robert Metzger I assume the com/google/protobuf classfile you found is this one: https://github.com/apache/flink-statefun/blob/master/statefun-shaded/statefun-protobuf-shaded/src/main/java/org/apache/flink/statefun/sdk/shaded/com/google/protobuf/MoreByteStrings.java This actually isn't a class pulled from a Protobuf dependency - it's code developed under StateFun. The package com/google/protobuf was required because the class exists essentially as a workaround to access some package-private protected methods on Protobuf. I believe that in this case, a NOTICE acknowledgement is not required as we actually own that piece of code. Let me know what you think and if this clears things up! Cheers, Gordon On Thu, Apr 8, 2021 at 4:00 AM Robert Metzger wrote: > This jar contains a com/google/protobuf classfile, which is not declared in > any NOTICE file (and doesn't ship the license file of protobuf): > > https://repository.apache.org/content/repositories/orgapacheflink-1415/org/apache/flink/statefun-flink-common/3.0.0/statefun-flink-common-3.0.0.jar > > I fear that this could be a blocker for the release? > > Otherwise, I did the following check: > > - src distribution looks fine: No binaries, js related files are declared > (the copyright in the NOTICE file could be updated to 2021, but that's not > a blocker) > > > On Fri, Apr 2, 2021 at 8:29 AM Yu Li wrote: > > > +1 (binding) > > > > Checked sums and signatures: OK > > Checked RAT and end-to-end tests: OK > > Checked version in pom/README/setup.py files: OK > > Checked release notes: OK > > Checked docker PR: OK > > > > Thanks for driving this release, Gordon! > > > > Best Regards, > > Yu > > > > > > On Fri, 2 Apr 2021 at 09:22, Seth Wiesman wrote: > > > > > +1 (non-binding) > > > > > > - Built from source and executed end to end tests > > > - Checked licenses and signatures > > > - Deployed remote Java SDK to gke cluster > > > - Took savepoint and statefully rescaled > > > > > > Seth > > > > > > On Thu, Apr 1, 2021 at 9:05 AM Konstantin Knauf > > wrote: > > > > > > > +1 (non-binding) > > > > > > > > - mvn clean install -Prun-e2e-tests (java 8) from source > > > > - python3 -m unittest tests > > > > - spin up Statefun Cluster on EKS with an image built from the > > > Dockerfiles > > > > of [1] > > > > - run Python & Java Greeter example on AWS Lambda > > > > - read through documentation (opened [2] to fix some tpoys) > > > > > > > > [1] https://github.com/apache/flink-statefun-docker/pull/13 > > > > [2] https://github.com/apache/flink-statefun/pull/219 > > > > > > > > On Thu, Apr 1, 2021 at 6:46 AM Tzu-Li (Gordon) Tai < > > tzuli...@apache.org> > > > > wrote: > > > > > > > > > +1 (binding) > > > > > > > > > > - verified signatures and hashes > > > > > - NOTICE and LICENSE files in statefun-flink-distribution, > > > > > statefun-protobuf-shaded, and statefun-sdk-java looks sane > > > > > - maven clean install -Prun-e2e-tests (java 8) from source > > > > > - ran all examples and tutorials in > apache/flink-statefun-playground > > > with > > > > > the new artifacts > > > > > - Ran my SDK verifier utility [1] against the new Java and Python > > SDKs. > > > > > > > > > > Cheers, > > > > > Gordon > > > > > > > > > > [1] https://github.com/tzulitai/statefun-sdk-verifier > > > > > > > > > > On Wed, Mar 31, 2021 at 8:50 PM Igal Shilman > > > wrote: > > > > > > > > > > > Thanks Gordon for managing the release! > > > > > > > > > > > > +1 (non binding) from my side: > > > > > > > > > > > > Here are the results of my testing: > > > > > > - verified the signatures > > > > > > - verified that the source distribution doesn't contain any > binary > > > > files > > > > > > - ran mvn clean install -Prun-e2e-tests with java8 > > > > > > - ran the smoke test that sends 100 million messages locally. > > > > > > - extended the smoke test to include the remote sdks (1 function > in > > > the > > > > > > Java SDK, 1 function in the Python SDK), and it passes. > > > > > > - deployed to kubernetes with minio as an S3 replacement. > > > > > > > > > > > > > > > > > > On Tue, Mar 30, 2021 at 12:29 PM Tzu-Li (Gordon) Tai < > > > > > tzuli...@apache.org> > > > > > > wrote: > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > Please review and vote on the release candidate #1 for the > > version > > > > > 3.0.0 > > > > > > of > > > > > > > Apache Flink Stateful Functions, as follows: > > > > > > > [ ] +1, Approve the release > > > > > > > [ ] -1, Do not approve the release (please provide specific > > > comments) > > > > > > > > > > > > > > **Testing Guideline** > > > > > > > > > > > > > > You can find here [1] a page in the project wiki on > instructions > > > for > > > > > > > testing. > > > > > > > To cast a vote, it is not necessary to perform all listed > checks, > > > > > > > but please mention which checks you have performed when voting. > > > > > > > > > > > > > > **Release Overview** > > > > > > > > > > > > > > As an overview, the release consists of the fo
Re: [VOTE] Apache Flink Stateful Functions 3.0.0, release candidate #1
@Robert Metzger Sorry, this is the correct link to the class file you are referring to (previous link I mentioned is incorrect): https://github.com/apache/flink-statefun/blob/master/statefun-flink/statefun-flink-common/src/main/java/com/google/protobuf/MoreByteStrings.java On Thu, Apr 8, 2021 at 11:17 AM Tzu-Li (Gordon) Tai wrote: > @Robert Metzger > > I assume the com/google/protobuf classfile you found is this one: > > https://github.com/apache/flink-statefun/blob/master/statefun-shaded/statefun-protobuf-shaded/src/main/java/org/apache/flink/statefun/sdk/shaded/com/google/protobuf/MoreByteStrings.java > > This actually isn't a class pulled from a Protobuf dependency - it's code > developed under StateFun. > The package com/google/protobuf was required because the class exists > essentially as a workaround to access some package-private protected > methods on Protobuf. > > I believe that in this case, a NOTICE acknowledgement is not required as > we actually own that piece of code. > > Let me know what you think and if this clears things up! > > Cheers, > Gordon > > On Thu, Apr 8, 2021 at 4:00 AM Robert Metzger wrote: > >> This jar contains a com/google/protobuf classfile, which is not declared >> in >> any NOTICE file (and doesn't ship the license file of protobuf): >> >> https://repository.apache.org/content/repositories/orgapacheflink-1415/org/apache/flink/statefun-flink-common/3.0.0/statefun-flink-common-3.0.0.jar >> >> I fear that this could be a blocker for the release? >> >> Otherwise, I did the following check: >> >> - src distribution looks fine: No binaries, js related files are declared >> (the copyright in the NOTICE file could be updated to 2021, but that's not >> a blocker) >> >> >> On Fri, Apr 2, 2021 at 8:29 AM Yu Li wrote: >> >> > +1 (binding) >> > >> > Checked sums and signatures: OK >> > Checked RAT and end-to-end tests: OK >> > Checked version in pom/README/setup.py files: OK >> > Checked release notes: OK >> > Checked docker PR: OK >> > >> > Thanks for driving this release, Gordon! >> > >> > Best Regards, >> > Yu >> > >> > >> > On Fri, 2 Apr 2021 at 09:22, Seth Wiesman wrote: >> > >> > > +1 (non-binding) >> > > >> > > - Built from source and executed end to end tests >> > > - Checked licenses and signatures >> > > - Deployed remote Java SDK to gke cluster >> > > - Took savepoint and statefully rescaled >> > > >> > > Seth >> > > >> > > On Thu, Apr 1, 2021 at 9:05 AM Konstantin Knauf >> > wrote: >> > > >> > > > +1 (non-binding) >> > > > >> > > > - mvn clean install -Prun-e2e-tests (java 8) from source >> > > > - python3 -m unittest tests >> > > > - spin up Statefun Cluster on EKS with an image built from the >> > > Dockerfiles >> > > > of [1] >> > > > - run Python & Java Greeter example on AWS Lambda >> > > > - read through documentation (opened [2] to fix some tpoys) >> > > > >> > > > [1] https://github.com/apache/flink-statefun-docker/pull/13 >> > > > [2] https://github.com/apache/flink-statefun/pull/219 >> > > > >> > > > On Thu, Apr 1, 2021 at 6:46 AM Tzu-Li (Gordon) Tai < >> > tzuli...@apache.org> >> > > > wrote: >> > > > >> > > > > +1 (binding) >> > > > > >> > > > > - verified signatures and hashes >> > > > > - NOTICE and LICENSE files in statefun-flink-distribution, >> > > > > statefun-protobuf-shaded, and statefun-sdk-java looks sane >> > > > > - maven clean install -Prun-e2e-tests (java 8) from source >> > > > > - ran all examples and tutorials in >> apache/flink-statefun-playground >> > > with >> > > > > the new artifacts >> > > > > - Ran my SDK verifier utility [1] against the new Java and Python >> > SDKs. >> > > > > >> > > > > Cheers, >> > > > > Gordon >> > > > > >> > > > > [1] https://github.com/tzulitai/statefun-sdk-verifier >> > > > > >> > > > > On Wed, Mar 31, 2021 at 8:50 PM Igal Shilman >> > > wrote: >> > > > > >> > > > > > Thanks Gordon for managing the release! >> > > > > > >> > > > > > +1 (non binding) from my side: >> > > > > > >> > > > > > Here are the results of my testing: >> > > > > > - verified the signatures >> > > > > > - verified that the source distribution doesn't contain any >> binary >> > > > files >> > > > > > - ran mvn clean install -Prun-e2e-tests with java8 >> > > > > > - ran the smoke test that sends 100 million messages locally. >> > > > > > - extended the smoke test to include the remote sdks (1 >> function in >> > > the >> > > > > > Java SDK, 1 function in the Python SDK), and it passes. >> > > > > > - deployed to kubernetes with minio as an S3 replacement. >> > > > > > >> > > > > > >> > > > > > On Tue, Mar 30, 2021 at 12:29 PM Tzu-Li (Gordon) Tai < >> > > > > tzuli...@apache.org> >> > > > > > wrote: >> > > > > > >> > > > > > > Hi everyone, >> > > > > > > >> > > > > > > Please review and vote on the release candidate #1 for the >> > version >> > > > > 3.0.0 >> > > > > > of >> > > > > > > Apache Flink Stateful Functions, as follows: >> > > > > > > [ ] +1, Approve the release >> > > > > > > [ ] -1, Do not approve the release (please pro
[jira] [Created] (FLINK-22147) Refactor Partition Discovery Logic in KafkaSourceEnumerator
Qingsheng Ren created FLINK-22147:
-
Summary: Refactor Partition Discovery Logic in
KafkaSourceEnumerator
Key: FLINK-22147
URL: https://issues.apache.org/jira/browse/FLINK-22147
Project: Flink
Issue Type: Improvement
Components: Connectors / Kafka
Affects Versions: 1.13.0
Reporter: Qingsheng Ren
Currently the logic of partition discovery is: the worker thread checks if
there's new partitions and initialize new splits if so, then coordinator thread
marks these splits as pending and try to make assignments.
Under current design, the worker thread needs to keep an internal data
structure tracking already discovered partitions, which is duplicated with
pending splits + assigned partitions tracked by coordinator thread. Usually
this kind of double-bookkeeping is fragile.
Another issue is that the worker thread always fetches descriptions of ALL
topics at partition discovery, which will comes to a problem working with a
giant Kafka clusters with millions of topics/partitions.
In order to fix issues above, a refactor is needed for the partition discovery
logic in Kafka enumerator. Basically the logic can be changed to:
# The worker thread fetches descriptions of subscribed topics/partitions, then
hands over to coordinator thread
# The coordinator thread filters out already discovered partitions (pending +
assigned partitions), then invokes worker thread with {{callAsync}} to fetch
offsets for new partitions
# The worker thread fetches offsets and creates splits for new partitions,
then hands over new splits to coordinator thread
# The coordinator thread marks these splits as pending and try to make
assignment.
Discussion of this issue can be found in
[https://github.com/apache/flink/pull/15461] .
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [VOTE] Apache Flink Stateful Functions 3.0.0, release candidate #1
I see. Thanks a lot for clarifying. I then vote +1 (binding) on this release. Thanks a lot for driving this! On Thu, Apr 8, 2021 at 5:19 AM Tzu-Li (Gordon) Tai wrote: > @Robert Metzger > > Sorry, this is the correct link to the class file you are referring to > (previous link I mentioned is incorrect): > > https://github.com/apache/flink-statefun/blob/master/statefun-flink/statefun-flink-common/src/main/java/com/google/protobuf/MoreByteStrings.java > > On Thu, Apr 8, 2021 at 11:17 AM Tzu-Li (Gordon) Tai > wrote: > >> @Robert Metzger >> >> I assume the com/google/protobuf classfile you found is this one: >> >> https://github.com/apache/flink-statefun/blob/master/statefun-shaded/statefun-protobuf-shaded/src/main/java/org/apache/flink/statefun/sdk/shaded/com/google/protobuf/MoreByteStrings.java >> >> This actually isn't a class pulled from a Protobuf dependency - it's code >> developed under StateFun. >> The package com/google/protobuf was required because the class exists >> essentially as a workaround to access some package-private protected >> methods on Protobuf. >> >> I believe that in this case, a NOTICE acknowledgement is not required as >> we actually own that piece of code. >> >> Let me know what you think and if this clears things up! >> >> Cheers, >> Gordon >> >> On Thu, Apr 8, 2021 at 4:00 AM Robert Metzger >> wrote: >> >>> This jar contains a com/google/protobuf classfile, which is not declared >>> in >>> any NOTICE file (and doesn't ship the license file of protobuf): >>> >>> https://repository.apache.org/content/repositories/orgapacheflink-1415/org/apache/flink/statefun-flink-common/3.0.0/statefun-flink-common-3.0.0.jar >>> >>> I fear that this could be a blocker for the release? >>> >>> Otherwise, I did the following check: >>> >>> - src distribution looks fine: No binaries, js related files are declared >>> (the copyright in the NOTICE file could be updated to 2021, but that's >>> not >>> a blocker) >>> >>> >>> On Fri, Apr 2, 2021 at 8:29 AM Yu Li wrote: >>> >>> > +1 (binding) >>> > >>> > Checked sums and signatures: OK >>> > Checked RAT and end-to-end tests: OK >>> > Checked version in pom/README/setup.py files: OK >>> > Checked release notes: OK >>> > Checked docker PR: OK >>> > >>> > Thanks for driving this release, Gordon! >>> > >>> > Best Regards, >>> > Yu >>> > >>> > >>> > On Fri, 2 Apr 2021 at 09:22, Seth Wiesman wrote: >>> > >>> > > +1 (non-binding) >>> > > >>> > > - Built from source and executed end to end tests >>> > > - Checked licenses and signatures >>> > > - Deployed remote Java SDK to gke cluster >>> > > - Took savepoint and statefully rescaled >>> > > >>> > > Seth >>> > > >>> > > On Thu, Apr 1, 2021 at 9:05 AM Konstantin Knauf >>> > wrote: >>> > > >>> > > > +1 (non-binding) >>> > > > >>> > > > - mvn clean install -Prun-e2e-tests (java 8) from source >>> > > > - python3 -m unittest tests >>> > > > - spin up Statefun Cluster on EKS with an image built from the >>> > > Dockerfiles >>> > > > of [1] >>> > > > - run Python & Java Greeter example on AWS Lambda >>> > > > - read through documentation (opened [2] to fix some tpoys) >>> > > > >>> > > > [1] https://github.com/apache/flink-statefun-docker/pull/13 >>> > > > [2] https://github.com/apache/flink-statefun/pull/219 >>> > > > >>> > > > On Thu, Apr 1, 2021 at 6:46 AM Tzu-Li (Gordon) Tai < >>> > tzuli...@apache.org> >>> > > > wrote: >>> > > > >>> > > > > +1 (binding) >>> > > > > >>> > > > > - verified signatures and hashes >>> > > > > - NOTICE and LICENSE files in statefun-flink-distribution, >>> > > > > statefun-protobuf-shaded, and statefun-sdk-java looks sane >>> > > > > - maven clean install -Prun-e2e-tests (java 8) from source >>> > > > > - ran all examples and tutorials in >>> apache/flink-statefun-playground >>> > > with >>> > > > > the new artifacts >>> > > > > - Ran my SDK verifier utility [1] against the new Java and Python >>> > SDKs. >>> > > > > >>> > > > > Cheers, >>> > > > > Gordon >>> > > > > >>> > > > > [1] https://github.com/tzulitai/statefun-sdk-verifier >>> > > > > >>> > > > > On Wed, Mar 31, 2021 at 8:50 PM Igal Shilman >> > >>> > > wrote: >>> > > > > >>> > > > > > Thanks Gordon for managing the release! >>> > > > > > >>> > > > > > +1 (non binding) from my side: >>> > > > > > >>> > > > > > Here are the results of my testing: >>> > > > > > - verified the signatures >>> > > > > > - verified that the source distribution doesn't contain any >>> binary >>> > > > files >>> > > > > > - ran mvn clean install -Prun-e2e-tests with java8 >>> > > > > > - ran the smoke test that sends 100 million messages locally. >>> > > > > > - extended the smoke test to include the remote sdks (1 >>> function in >>> > > the >>> > > > > > Java SDK, 1 function in the Python SDK), and it passes. >>> > > > > > - deployed to kubernetes with minio as an S3 replacement. >>> > > > > > >>> > > > > > >>> > > > > > On Tue, Mar 30, 2021 at 12:29 PM Tzu-Li (Gordon) Tai < >>> > > > > tzuli...@apache.org> >>> > > > > > wrote: >>> > > > >
Re: [DISCUSS] Backport FLIP-27 Kafka source connector fixes with API change to release-1.12.
Hi Becket, did you need to change anything to the source interface itself? Wouldn't it be possible for users to simply use the 1.13 connector with their Flink 1.12 deployment? I think the late-upgrade argument can be made for any feature, but I also see that the Kafka connector is of high interest. I'd second Till's question if there is an issue for users that start with the current Kafka source (+bugfixes) to later upgrade to 1.13 Kafka source with API changes. Best, Arvid On Thu, Apr 8, 2021 at 2:54 AM Becket Qin wrote: > Thanks for the comment, Till and Thomas. > > As far as I know there are some users who have just upgraded their Flink > version from 1.8 / 1.9 to Flink 1.12 and might not upgrade the version in 6 > months or more. There are also some organizations that have the strategy of > not running the latest version of a project, but only the second latest > version with bug fixes. So those users may still benefit from the backport. > However, arguably the old Kafka source is there anyways in 1.12, so they > should not be blocked on having the new source. > > I am leaning towards backporting the fixes mainly because this way we might > have more users migrating to the new Source and provide feedback. It will > take some time for the users to pick up 1.13, especially for the users > running Flink at large scale. So backporting the fixes to 1.12 would help > get the new source to be used sooner. > > Thanks, > > Jiangjie (Becket) Qin > > On Thu, Apr 8, 2021 at 12:40 AM Thomas Weise wrote: > > > Hi, > > > > Thanks for fixing the new KafkaSource issues. > > > > I'm interested in using these fixes with 1.12 for experimental purposes. > > > > +1 for backporting. 1.12 is the current stable release and users who > would > > like to try the FLIP-27 sources are likely to use that release. > > > > Thomas > > > > On Wed, Apr 7, 2021 at 2:50 AM Till Rohrmann > wrote: > > > > > Hi Becket, > > > > > > If I remember correctly, then we deliberately not documented the Kafka > > > connector in the 1.12 release. Hence, from this point there should be > no > > > need to backport any fixes because users are not aware of this feature. > > > > > > On the other hand this also means that we should be able to break > > anything > > > we want to. Consequently, backporting these fixes should be possible. > > > > > > The question would probably be whether we want to ship new features > with > > a > > > bug fix release. Do we know of any users who want to use the new Kafka > > > source, are using the 1.12 version and cannot upgrade to 1.13 once it > is > > > released? If this is the case, then this could be an argument for > > shipping > > > this feature with a bug fix release. If not, then we could save some > work > > > by not backporting it. > > > > > > Cheers, > > > Till > > > > > > On Wed, Apr 7, 2021 at 10:43 AM Becket Qin > wrote: > > > > > > > Hi folks, > > > > > > > > I'd like to start a discussion thread about backporting some FLIP-27 > > > Kafka > > > > source connector fixes to release-1.12. These fixes include some API > > > > changes and thus needs a public discussion. > > > > > > > > The tickets in question are following: > > > > https://issues.apache.org/jira/browse/FLINK-20379 > > > > https://issues.apache.org/jira/browse/FLINK-20114 > > > > https://issues.apache.org/jira/browse/FLINK-21817 > > > > > > > > Without these fixes, the FLIP-27 Kafka source in release-1.12 is not > > > really > > > > usable, and the API changes only affect the Kafka Source. So it seems > > > > breaking the API in this case is still worthwhile. > > > > > > > > It would be good to see what others think. > > > > > > > > Thanks, > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > >
[jira] [Created] (FLINK-22148) Planner rules should use RexCall#equsls to check whether two rexCalls are equivalent
Shuo Cheng created FLINK-22148:
--
Summary: Planner rules should use RexCall#equsls to check whether
two rexCalls are equivalent
Key: FLINK-22148
URL: https://issues.apache.org/jira/browse/FLINK-22148
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner
Affects Versions: 1.12.2
Reporter: Shuo Cheng
Fix For: 1.13.0
Reproduce the bug by add the following test to `SemiAntiJoinTest`
{code:java}
// code placeholder
@Test
def testNotSimplifyJoinConditionWithSameDigest(): Unit = {
val sqlQuery =
"""
|SELECT a
|FROM l
|WHERE c NOT IN (
|SELECT f FROM r WHERE f = c)
|""".stripMargin
util.verifyRelPlan(sqlQuery)
}
{code}
CannotPlanException will be thrown, this is because Calcite planner will
normalize a RexCall in the `equals` method (from 1.24), while in Flink planer
rules, we still use toString to check two RexCalls are equivalent.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [VOTE] Apache Flink Stateful Functions 3.0.0, release candidate #1
Thanks for voting and testing everyone! We have a total of 6 +1 votes, 3 of which are binding: - Igal Shilman - Gordon Tai (binding) - Konstantin Knauf - Seth Wiesman - Yu Li (binding) - Robert Metzger (binding) I'll proceed now with finalizing the release of StateFun 3.0.0. The official announcement will likely happen next week, as we're finishing up with the announcement blog post which would probably also take a few days to be reviewed. Thanks, Gordon On Thu, Apr 8, 2021 at 1:50 PM Robert Metzger wrote: > I see. Thanks a lot for clarifying. > > I then vote > > +1 (binding) > > on this release. Thanks a lot for driving this! > > > On Thu, Apr 8, 2021 at 5:19 AM Tzu-Li (Gordon) Tai > wrote: > >> @Robert Metzger >> >> Sorry, this is the correct link to the class file you are referring to >> (previous link I mentioned is incorrect): >> >> https://github.com/apache/flink-statefun/blob/master/statefun-flink/statefun-flink-common/src/main/java/com/google/protobuf/MoreByteStrings.java >> >> On Thu, Apr 8, 2021 at 11:17 AM Tzu-Li (Gordon) Tai >> wrote: >> >>> @Robert Metzger >>> >>> I assume the com/google/protobuf classfile you found is this one: >>> >>> https://github.com/apache/flink-statefun/blob/master/statefun-shaded/statefun-protobuf-shaded/src/main/java/org/apache/flink/statefun/sdk/shaded/com/google/protobuf/MoreByteStrings.java >>> >>> This actually isn't a class pulled from a Protobuf dependency - it's >>> code developed under StateFun. >>> The package com/google/protobuf was required because the class exists >>> essentially as a workaround to access some package-private protected >>> methods on Protobuf. >>> >>> I believe that in this case, a NOTICE acknowledgement is not required as >>> we actually own that piece of code. >>> >>> Let me know what you think and if this clears things up! >>> >>> Cheers, >>> Gordon >>> >>> On Thu, Apr 8, 2021 at 4:00 AM Robert Metzger >>> wrote: >>> This jar contains a com/google/protobuf classfile, which is not declared in any NOTICE file (and doesn't ship the license file of protobuf): https://repository.apache.org/content/repositories/orgapacheflink-1415/org/apache/flink/statefun-flink-common/3.0.0/statefun-flink-common-3.0.0.jar I fear that this could be a blocker for the release? Otherwise, I did the following check: - src distribution looks fine: No binaries, js related files are declared (the copyright in the NOTICE file could be updated to 2021, but that's not a blocker) On Fri, Apr 2, 2021 at 8:29 AM Yu Li wrote: > +1 (binding) > > Checked sums and signatures: OK > Checked RAT and end-to-end tests: OK > Checked version in pom/README/setup.py files: OK > Checked release notes: OK > Checked docker PR: OK > > Thanks for driving this release, Gordon! > > Best Regards, > Yu > > > On Fri, 2 Apr 2021 at 09:22, Seth Wiesman wrote: > > > +1 (non-binding) > > > > - Built from source and executed end to end tests > > - Checked licenses and signatures > > - Deployed remote Java SDK to gke cluster > > - Took savepoint and statefully rescaled > > > > Seth > > > > On Thu, Apr 1, 2021 at 9:05 AM Konstantin Knauf > wrote: > > > > > +1 (non-binding) > > > > > > - mvn clean install -Prun-e2e-tests (java 8) from source > > > - python3 -m unittest tests > > > - spin up Statefun Cluster on EKS with an image built from the > > Dockerfiles > > > of [1] > > > - run Python & Java Greeter example on AWS Lambda > > > - read through documentation (opened [2] to fix some tpoys) > > > > > > [1] https://github.com/apache/flink-statefun-docker/pull/13 > > > [2] https://github.com/apache/flink-statefun/pull/219 > > > > > > On Thu, Apr 1, 2021 at 6:46 AM Tzu-Li (Gordon) Tai < > tzuli...@apache.org> > > > wrote: > > > > > > > +1 (binding) > > > > > > > > - verified signatures and hashes > > > > - NOTICE and LICENSE files in statefun-flink-distribution, > > > > statefun-protobuf-shaded, and statefun-sdk-java looks sane > > > > - maven clean install -Prun-e2e-tests (java 8) from source > > > > - ran all examples and tutorials in apache/flink-statefun-playground > > with > > > > the new artifacts > > > > - Ran my SDK verifier utility [1] against the new Java and Python > SDKs. > > > > > > > > Cheers, > > > > Gordon > > > > > > > > [1] https://github.com/tzulitai/statefun-sdk-verifier > > > > > > > > On Wed, Mar 31, 2021 at 8:50 PM Igal Shilman < i...@ververica.com> > > wrote: > > > > > > > > > Thanks Gordon for managing the release! > > > > > > > > > > +1 (non binding) from my side: > > > > > > > > > > Here are the results of my
