[jira] [Created] (FLINK-24662) PyFlink sphinx check failed with "node class 'meta' is already registered, its visitors will be overridden"
Dian Fu created FLINK-24662:
---
Summary: PyFlink sphinx check failed with "node class 'meta' is
already registered, its visitors will be overridden"
Key: FLINK-24662
URL: https://issues.apache.org/jira/browse/FLINK-24662
Project: Flink
Issue Type: Bug
Components: API / Python, Tests
Affects Versions: 1.14.0, 1.13.0, 1.15.0
Reporter: Dian Fu
Assignee: Dian Fu
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=25481=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=8d78fe4f-d658-5c70-12f8-4921589024c3]
{code}
==mypy checks... [SUCCESS]===
Oct 26 22:08:34 rm -rf _build/*
Oct 26 22:08:34 /__w/1/s/flink-python/dev/.conda/bin/sphinx-build -b html -d
_build/doctrees -a -W . _build/html
Oct 26 22:08:34 Running Sphinx v2.4.4
Oct 26 22:08:34
Oct 26 22:08:34 Warning, treated as error:
Oct 26 22:08:34 node class 'meta' is already registered, its visitors will be
overridden
Oct 26 22:08:34 Makefile:76: recipe for target 'html' failed
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-24661) ConfigOption add isSecret method to judge sensitive options
Ada Wong created FLINK-24661: Summary: ConfigOption add isSecret method to judge sensitive options Key: FLINK-24661 URL: https://issues.apache.org/jira/browse/FLINK-24661 Project: Flink Issue Type: Improvement Components: API / Core Affects Versions: 1.13.3 Reporter: Ada Wong Related ticket https://issues.apache.org/jira/browse/FLINK-24381 [~chesnay] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24660) Allow setting KafkaSubscriber in KafkaSourceBuilder
Mason Chen created FLINK-24660: -- Summary: Allow setting KafkaSubscriber in KafkaSourceBuilder Key: FLINK-24660 URL: https://issues.apache.org/jira/browse/FLINK-24660 Project: Flink Issue Type: Improvement Components: Connectors / Kafka Affects Versions: 1.13.3, 1.14.0 Reporter: Mason Chen Some users may have a different mechanism for subscribing the set of topics/partitions. The builder can allow user custom implementations of KafkaSubscriber -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24659) Two active miniCluster in RemoteBenchmarkBase
Anton Kalashnikov created FLINK-24659: - Summary: Two active miniCluster in RemoteBenchmarkBase Key: FLINK-24659 URL: https://issues.apache.org/jira/browse/FLINK-24659 Project: Flink Issue Type: Bug Components: Benchmarks, Runtime / Network Affects Versions: 1.14.0 Reporter: Anton Kalashnikov It seems that all children of RemoteBenchmarkBase work incorrectly since they configure the environment for miniCluster from FlinkEnvironmentContext but in reality, they use miniCluster from RemoteBenchmarkBase. So it definitely we should remove one of them. I think we can get rid of RemoteBenchmarkBase#miniCluster and use FlinkEnvironmentContext#miniCluster everywhere. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24658) Debug logs for buffer size calculation
Anton Kalashnikov created FLINK-24658: - Summary: Debug logs for buffer size calculation Key: FLINK-24658 URL: https://issues.apache.org/jira/browse/FLINK-24658 Project: Flink Issue Type: Improvement Components: Runtime / Network Affects Versions: 1.14.0 Reporter: Anton Kalashnikov Since the buffer debloater recalculates buffer size based on several different parameters. It makes sense to add debug logging to print all of them in case of necessary debugging. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24657) Add metric of the total real size of input/output buffers queue
Anton Kalashnikov created FLINK-24657: - Summary: Add metric of the total real size of input/output buffers queue Key: FLINK-24657 URL: https://issues.apache.org/jira/browse/FLINK-24657 Project: Flink Issue Type: Improvement Components: Runtime / Network Affects Versions: 1.14.0 Reporter: Anton Kalashnikov Right now we have the metric of the length of input/output buffers queue but since buffer debloater has been introduced this metric is not always helpful because the real size of each buffer can be different. So it is an idea to add a new metric that shows the total size of buffers in the queue. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Flink sql client usability improvements
Hi Sergey, Welcome contributions! You can read the FLIP introduction [1] first. I think FLIP-163 is a good example for you which is a SQL Client improvement we made in 1.13. [1]: https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals [2]: https://cwiki.apache.org/confluence/display/FLINK/FLIP-163%3A+SQL+Client+Improvements Best, Jark On Tue, 26 Oct 2021 at 20:09, Sergey Nuyanzin wrote: > Hello Flink community > > currently I work a bit with some regarding Flink sql client usability > issues [1] > > I noticed that there could be a lot more usability improvement, that's why > first I want to ask if it makes sense for the community to raise such a > FLIP. And yes, I would like to contribute here. > > I'm asking because I'm not quite familiar with FLIP procedure > > Some options that I have in mind > - Prompting on what is missed (bracket, end of multiline comment end etc.) > - Syntax highlighting > - Improved completion > - Some more jline oob features > - something else > > [1] https://issues.apache.org/jira/browse/FLINK-24592 > -- > Best regards, > Sergey >
[jira] [Created] (FLINK-24656) Add user document for Window Deduplication
JING ZHANG created FLINK-24656: -- Summary: Add user document for Window Deduplication Key: FLINK-24656 URL: https://issues.apache.org/jira/browse/FLINK-24656 Project: Flink Issue Type: Sub-task Components: Documentation Affects Versions: 1.15.0 Reporter: JING ZHANG -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [NOTICE] flink-streaming-java no longer depends on Scala and lost it's suffix
This is an awesome accomplishment, many thanks! Ideally we also manage to complete https://issues.apache.org/jira/browse/FLINK-24427 for 1.15 to get Flink in a much better shape for our Scala users. On Tue, 26 Oct 2021 at 12:34, Arvid Heise wrote: > Awesome. Thank you very much for all the hard work! > > On Tue, Oct 26, 2021 at 1:06 AM Chesnay Schepler > wrote: > > > This time with proper formatting... > > > > flink-batch-sql-test > > flink-cep > > flink-cli-test > > flink-clients > > flink-connector-elasticsearch-base > > flink-connector-elasticsearch5 > > flink-connector-elasticsearch6 > > flink-connector-elasticsearch7 > > flink-connector-gcp-pubsub > > flink-connector-hbase-1.4 > > flink-connector-hbase-2.2 > > flink-connector-hbase-base > > flink-connector-jdbc > > flink-connector-kafka > > flink-connector-kinesis > > flink-connector-nifi > > flink-connector-pulsar > > flink-connector-rabbitmq > > flink-connector-testing > > flink-connector-twitter > > flink-connector-wikiedits > > flink-container > > flink-distributed-cache-via-blob-test > > flink-dstl-dfs > > flink-gelly > > flink-hadoop-bulk > > flink-kubernetes > > flink-parent-child-classloading-test-lib-package > > flink-parent-child-classloading-test-program > > flink-queryable-state-test > > flink-runtime-web > > flink-scala > > flink-sql-connector-elasticsearch6 > > flink-sql-connector-elasticsearch7 > > flink-sql-connector-hbase-1.4 > > flink-sql-connector-hbase-2.2 > > flink-sql-connector-kafka > > flink-sql-connector-kinesis > > flink-sql-connector-rabbitmq > > flink-state-processor-api > > flink-statebackend-rocksdb > > flink-streaming-java > > flink-streaming-kafka-test > > flink-streaming-kafka-test-base > > flink-streaming-kinesis-test > > flink-table-api-java-bridge > > flink-test-utils > > flink-walkthrough-common > > flink-yarn > > > > > > On 26/10/2021 01:04, Chesnay Schepler wrote: > > > Hello all, > > > > > > I just wanted to inform everyone that I just merged > > > https://issues.apache.org/jira/browse/FLINK-24018, removing the > > > transitive Scala dependencies from flink-streaming-java. This also > > > means that the module lost it's Scala suffix, along with a lot of > > > other modules. > > > > > > Please keep this mind this for a few days when adding Flink > > > dependencies or new modules; it is quite likely that something has > > > changed w.r.t. the Scala suffixes. > > > > > > For completeness sake, these are the module that lost the suffix: > > > > > > |flink-batch-sql-test flink-cep flink-cli-test flink-clients > > > flink-connector-elasticsearch-base flink-connector-elasticsearch5 > > > flink-connector-elasticsearch6 flink-connector-elasticsearch7 > > > flink-connector-gcp-pubsub flink-connector-hbase-1.4 > > > flink-connector-hbase-2.2 flink-connector-hbase-base > > > flink-connector-jdbc flink-connector-kafka flink-connector-kinesis > > > flink-connector-nifi flink-connector-pulsar flink-connector-rabbitmq > > > flink-connector-testing flink-connector-twitter > > > flink-connector-wikiedits flink-container > > > flink-distributed-cache-via-blob-test flink-dstl-dfs flink-gelly > > > flink-hadoop-bulk flink-kubernetes > > > flink-parent-child-classloading-test-lib-package > > > flink-parent-child-classloading-test-program > > > flink-queryable-state-test flink-runtime-web flink-scala > > > flink-sql-connector-elasticsearch6 flink-sql-connector-elasticsearch7 > > > flink-sql-connector-hbase-1.4 flink-sql-connector-hbase-2.2 > > > flink-sql-connector-kafka flink-sql-connector-kinesis > > > flink-sql-connector-rabbitmq flink-state-processor-api > > > flink-statebackend-rocksdb flink-streaming-java > > > flink-streaming-kafka-test flink-streaming-kafka-test-base > > > flink-streaming-kinesis-test flink-table-api-java-bridge > > > flink-test-utils flink-walkthrough-common flink-yarn| > > > > > > > >
Re: Shading in flink-table-blink and upgrade compatibility issue
Hi Thomas, thanks for your feedback. The error that you are experiencing is definitely a bug in 1.13.3 and the missing method should be reintroduced in the next patch version to make code compiled against older patch versions run again. Regarding the discussion points: I agree that flink-table-blink uber jar should not contain a dependency to flink-connector-base. Even filesystem connectors should be optional and put in a dedicated module that is not in /lib by default. With having the flink-table-blink uber jar in /lib we would like to improve the SQL experience as this API is as important as the DataStream API nowadays. But the depenencies should be minimal nevertheless. Regards, Timo On 23.10.21 00:59, Thomas Weise wrote: Hi, As part of upgrading to Flink 1.13.3 from 1.13.2 we run into the following problem with KafkaSource (Flink distribution is 1.13.2 and the application was built with 1.13.3): java.lang.NoSuchMethodError: 'void org.apache.flink.connector.base.source.reader.fetcher.SingleThreadFetcherManager.(org.apache.flink.connector.base.source.reader.synchronization.FutureCompletingBlockingQueue, java.util.function.Supplier, java.util.function.Consumer)' at org.apache.flink.connector.kafka.source.reader.fetcher.KafkaSourceFetcherManager.(KafkaSourceFetcherManager.java:67) at org.apache.flink.connector.kafka.source.KafkaSource.createReader(KafkaSource.java:160) at org.apache.flink.connector.kafka.source.KafkaSource.createReader(KafkaSource.java:127) It turns out that flink-table-blink_2.12-1.13.2.jar contains flink-connector-base and because that jar is under lib the 1.13.2 connector base gets picked up instead of the one bundled in the application jar. (The constructor in org.apache.flink.connector.base.source.reader.fetcher.SingleThreadFetcherManager was added in 1.13.3.) There are a few points I would like to discuss: 1) Version compatibility: A *patch* version should ideally not introduce such a change, it should be forward and backward compatible. Hopefully this will be the case after 1.14 with stable source API. 2) flink-table-blink - if it is meant to be self contained and usable as a library - should not leak its shaded dependencies. It contains FileSource and other deps from Flink, can those be relocated? 3) Do we need flink-table-blink under lib? Can it be bundled with the application instead? It would be great if the dependencies under lib are strictly Flink core. Thanks, Thomas
Flink sql client usability improvements
Hello Flink community currently I work a bit with some regarding Flink sql client usability issues [1] I noticed that there could be a lot more usability improvement, that's why first I want to ask if it makes sense for the community to raise such a FLIP. And yes, I would like to contribute here. I'm asking because I'm not quite familiar with FLIP procedure Some options that I have in mind - Prompting on what is missed (bracket, end of multiline comment end etc.) - Syntax highlighting - Improved completion - Some more jline oob features - something else [1] https://issues.apache.org/jira/browse/FLINK-24592 -- Best regards, Sergey
[jira] [Created] (FLINK-24655) Support checkpoint for jobs using iteration
Yun Gao created FLINK-24655: --- Summary: Support checkpoint for jobs using iteration Key: FLINK-24655 URL: https://issues.apache.org/jira/browse/FLINK-24655 Project: Flink Issue Type: Sub-task Components: Library / Machine Learning Reporter: Yun Gao Assignee: Yun Gao Fix For: 0.1.0 Supports checkpoints for jobs using iteration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24654) NPE on RetractableTopNFunction when some records were cleared by state ttl
lincoln lee created FLINK-24654:
---
Summary: NPE on RetractableTopNFunction when some records were
cleared by state ttl
Key: FLINK-24654
URL: https://issues.apache.org/jira/browse/FLINK-24654
Project: Flink
Issue Type: Bug
Components: Table SQL / Runtime
Affects Versions: 1.13.3, 1.14.0
Reporter: lincoln lee
Fix For: 1.15.0
NullPointerException will occurr when some records were cleared by the state
ttl.
stack:
{quote}java.lang.NullPointerExceptionjava.lang.NullPointerException at
org.apache.flink.table.runtime.operators.rank.RetractableTopNFunction.retractRecordWithRowNumber(RetractableTopNFunction.java:379)
at
org.apache.flink.table.runtime.operators.rank.RetractableTopNFunction.processElement(RetractableTopNFunction.java:175)
at
org.apache.flink.table.runtime.operators.rank.RetractableTopNFunction.processElement(RetractableTopNFunction.java:54)
at
org.apache.flink.streaming.api.operators.KeyedProcessOperator.processElement(KeyedProcessOperator.java:83)
at
org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness.processElement(OneInputStreamOperatorTestHarness.java:209)
at
org.apache.flink.table.planner.runtime.harness.RankHarnessTest.testRetractRankWithRowNumber(RankHarnessTest.scala:111)
{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-24653) Support per-round operators inside the iteration.
Yun Gao created FLINK-24653: --- Summary: Support per-round operators inside the iteration. Key: FLINK-24653 URL: https://issues.apache.org/jira/browse/FLINK-24653 Project: Flink Issue Type: Sub-task Components: Library / Machine Learning Reporter: Yun Gao Assignee: Yun Gao Fix For: 0.1.0 Supports using the per-round operators inside the iteration. These operators would be recreated for each round. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24652) Add operator wrapper for per-round operators
Yun Gao created FLINK-24652: --- Summary: Add operator wrapper for per-round operators Key: FLINK-24652 URL: https://issues.apache.org/jira/browse/FLINK-24652 Project: Flink Issue Type: Sub-task Components: Library / Machine Learning Reporter: Yun Gao Assignee: Yun Gao Fix For: 0.1.0 After https://issues.apache.org/jira/browse/FLINK-24646 , we future add per-round operator wrappers, namely the user operators would be re-created for each round. This allows us to reuse the existing operators outside of the iteration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24651) Support bounded all-round iteration
Yun Gao created FLINK-24651: --- Summary: Support bounded all-round iteration Key: FLINK-24651 URL: https://issues.apache.org/jira/browse/FLINK-24651 Project: Flink Issue Type: Sub-task Components: Library / Machine Learning Reporter: Yun Gao Assignee: Yun Gao Fix For: 0.1.0 Supports the iteration on the bounded streams with all the operators executed with an all-round lifecycle, namely the operators would not be recreated inside the iteration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24650) Support unbounded iteration.
Yun Gao created FLINK-24650: --- Summary: Support unbounded iteration. Key: FLINK-24650 URL: https://issues.apache.org/jira/browse/FLINK-24650 Project: Flink Issue Type: Sub-task Components: Library / Machine Learning Reporter: Yun Gao Assignee: Yun Gao Fix For: 0.1.0 Supports the unbounded iteration inside the flink-ml library. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24649) Add DraftExecutionEnvironment to support wrapping operators during compile time
Yun Gao created FLINK-24649: --- Summary: Add DraftExecutionEnvironment to support wrapping operators during compile time Key: FLINK-24649 URL: https://issues.apache.org/jira/browse/FLINK-24649 Project: Flink Issue Type: Sub-task Components: Library / Machine Learning Reporter: Yun Gao Assignee: Yun Gao Fix For: 0.1.0 To add wrappers for each operator, we'll have to first make users construct the subgraph of the iteration body in a separate env, and then copy to the actual env with the operators wrapped. We'll introduce a DraftExecutionEnvironment for this purpose. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24648) Ceil, floor for some timeunit return wrong results or fail with CompileException
Sergey Nuyanzin created FLINK-24648:
---
Summary: Ceil, floor for some timeunit return wrong results or
fail with CompileException
Key: FLINK-24648
URL: https://issues.apache.org/jira/browse/FLINK-24648
Project: Flink
Issue Type: Bug
Reporter: Sergey Nuyanzin
There are issues
1. for {{TIMESTAMP WITHOUT TIMEZONE}} and {{DATE}} it returns wrong result for
queries
{code:sql}
select ceil(timstamp'2020-26-10 12:12:12' to decade);
select ceil(timstamp'2020-26-10 12:12:12' to century);
select ceil(timstamp'2020-26-10 12:12:12' to , millennium);
{code}
same for {{FLOOR}} and {{DATE}}
2. for {{TIMESTAMP WITH TIMEZONE}} it throws exception
{noformat}
[ERROR] Could not execute SQL statement. Reason:
org.codehaus.commons.compiler.CompileException: Line 57, Column 0: No
applicable constructor/method found for actual parameters
"org.apache.flink.table.data.TimestampData,
org.apache.flink.table.data.TimestampData"; candidates are: "public static int
org.apache.calcite.runtime.SqlFunctions.ceil(int, java.math.BigDecimal)",
"public static java.math.BigDecimal
org.apache.calcite.runtime.SqlFunctions.ceil(java.math.BigDecimal, int)",
"public static java.math.BigDecimal
org.apache.calcite.runtime.SqlFunctions.ceil(java.math.BigDecimal,
java.math.BigDecimal)", "public static short
org.apache.calcite.runtime.SqlFunctions.ceil(short, short)", "public static
java.math.BigDecimal
org.apache.calcite.runtime.SqlFunctions.ceil(java.math.BigDecimal)", "public
static double org.apache.calcite.runtime.SqlFunctions.ceil(double)", "public
static float org.apache.calcite.runtime.SqlFunctions.ceil(float)", "public
static byte org.apache.calcite.runtime.SqlFunctions.ceil(byte, byte)", "public
static long org.apache.calcite.runtime.SqlFunctions.ceil(long, long)", "public
static int org.apache.calcite.runtime.SqlFunctions.ceil(int, int)"
at
org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:12211)
at
org.codehaus.janino.UnitCompiler.findMostSpecificIInvocable(UnitCompiler.java:9263)
at org.codehaus.janino.UnitCompiler.findIMethod(UnitCompiler.java:9123)
at org.codehaus.janino.UnitCompiler.findIMethod(UnitCompiler.java:9025)
at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:5062)
at org.codehaus.janino.UnitCompiler.access$9100(UnitCompiler.java:215)
at
org.codehaus.janino.UnitCompiler$16.visitMethodInvocation(UnitCompiler.java:4423)
at
org.codehaus.janino.UnitCompiler$16.visitMethodInvocation(UnitCompiler.java:4396)
at org.codehaus.janino.Java$MethodInvocation.accept(Java.java:5073)
at org.codehaus.janino.UnitCompiler.compileGet(UnitCompiler.java:4396)
at
org.codehaus.janino.UnitCompiler.compileGetValue(UnitCompiler.java:5662)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3792)
at org.codehaus.janino.UnitCompiler.access$6100(UnitCompiler.java:215)
at
org.codehaus.janino.UnitCompiler$13.visitAssignment(UnitCompiler.java:3754)
at
org.codehaus.janino.UnitCompiler$13.visitAssignment(UnitCompiler.java:3734)
at org.codehaus.janino.Java$Assignment.accept(Java.java:4477)
at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:3734)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2360)
at org.codehaus.janino.UnitCompiler.access$1800(UnitCompiler.java:215)
at
org.codehaus.janino.UnitCompiler$6.visitExpressionStatement(UnitCompiler.java:1494)
at
org.codehaus.janino.UnitCompiler$6.visitExpressionStatement(UnitCompiler.java:1487)
at org.codehaus.janino.Java$ExpressionStatement.accept(Java.java:2874)
at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1487)
at
org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1567)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:1553)
at org.codehaus.janino.UnitCompiler.access$1700(UnitCompiler.java:215)
at org.codehaus.janino.UnitCompiler$6.visitBlock(UnitCompiler.java:1493)
at org.codehaus.janino.UnitCompiler$6.visitBlock(UnitCompiler.java:1487)
at org.codehaus.janino.Java$Block.accept(Java.java:2779)
at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1487)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2476)
at org.codehaus.janino.UnitCompiler.access$1900(UnitCompiler.java:215)
at
org.codehaus.janino.UnitCompiler$6.visitIfStatement(UnitCompiler.java:1495)
at
org.codehaus.janino.UnitCompiler$6.visitIfStatement(UnitCompiler.java:1487)
at org.codehaus.janino.Java$IfStatement.accept(Java.java:2950)
at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1487)
at
org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1567)
Re: [NOTICE] flink-streaming-java no longer depends on Scala and lost it's suffix
Awesome. Thank you very much for all the hard work! On Tue, Oct 26, 2021 at 1:06 AM Chesnay Schepler wrote: > This time with proper formatting... > > flink-batch-sql-test > flink-cep > flink-cli-test > flink-clients > flink-connector-elasticsearch-base > flink-connector-elasticsearch5 > flink-connector-elasticsearch6 > flink-connector-elasticsearch7 > flink-connector-gcp-pubsub > flink-connector-hbase-1.4 > flink-connector-hbase-2.2 > flink-connector-hbase-base > flink-connector-jdbc > flink-connector-kafka > flink-connector-kinesis > flink-connector-nifi > flink-connector-pulsar > flink-connector-rabbitmq > flink-connector-testing > flink-connector-twitter > flink-connector-wikiedits > flink-container > flink-distributed-cache-via-blob-test > flink-dstl-dfs > flink-gelly > flink-hadoop-bulk > flink-kubernetes > flink-parent-child-classloading-test-lib-package > flink-parent-child-classloading-test-program > flink-queryable-state-test > flink-runtime-web > flink-scala > flink-sql-connector-elasticsearch6 > flink-sql-connector-elasticsearch7 > flink-sql-connector-hbase-1.4 > flink-sql-connector-hbase-2.2 > flink-sql-connector-kafka > flink-sql-connector-kinesis > flink-sql-connector-rabbitmq > flink-state-processor-api > flink-statebackend-rocksdb > flink-streaming-java > flink-streaming-kafka-test > flink-streaming-kafka-test-base > flink-streaming-kinesis-test > flink-table-api-java-bridge > flink-test-utils > flink-walkthrough-common > flink-yarn > > > On 26/10/2021 01:04, Chesnay Schepler wrote: > > Hello all, > > > > I just wanted to inform everyone that I just merged > > https://issues.apache.org/jira/browse/FLINK-24018, removing the > > transitive Scala dependencies from flink-streaming-java. This also > > means that the module lost it's Scala suffix, along with a lot of > > other modules. > > > > Please keep this mind this for a few days when adding Flink > > dependencies or new modules; it is quite likely that something has > > changed w.r.t. the Scala suffixes. > > > > For completeness sake, these are the module that lost the suffix: > > > > |flink-batch-sql-test flink-cep flink-cli-test flink-clients > > flink-connector-elasticsearch-base flink-connector-elasticsearch5 > > flink-connector-elasticsearch6 flink-connector-elasticsearch7 > > flink-connector-gcp-pubsub flink-connector-hbase-1.4 > > flink-connector-hbase-2.2 flink-connector-hbase-base > > flink-connector-jdbc flink-connector-kafka flink-connector-kinesis > > flink-connector-nifi flink-connector-pulsar flink-connector-rabbitmq > > flink-connector-testing flink-connector-twitter > > flink-connector-wikiedits flink-container > > flink-distributed-cache-via-blob-test flink-dstl-dfs flink-gelly > > flink-hadoop-bulk flink-kubernetes > > flink-parent-child-classloading-test-lib-package > > flink-parent-child-classloading-test-program > > flink-queryable-state-test flink-runtime-web flink-scala > > flink-sql-connector-elasticsearch6 flink-sql-connector-elasticsearch7 > > flink-sql-connector-hbase-1.4 flink-sql-connector-hbase-2.2 > > flink-sql-connector-kafka flink-sql-connector-kinesis > > flink-sql-connector-rabbitmq flink-state-processor-api > > flink-statebackend-rocksdb flink-streaming-java > > flink-streaming-kafka-test flink-streaming-kafka-test-base > > flink-streaming-kinesis-test flink-table-api-java-bridge > > flink-test-utils flink-walkthrough-common flink-yarn| > > > >
[jira] [Created] (FLINK-24647) ClusterUncaughtExceptionHandler does not log the exception
Fabian Paul created FLINK-24647: --- Summary: ClusterUncaughtExceptionHandler does not log the exception Key: FLINK-24647 URL: https://issues.apache.org/jira/browse/FLINK-24647 Project: Flink Issue Type: Bug Components: Runtime / Coordination Affects Versions: 1.14.0, 1.15.0 Reporter: Fabian Paul If an uncaught exception occurs and the uncaught exception handler is configured to log it swallows the exception stack trace. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24646) Add operator wrapper for all-round iterations
Yun Gao created FLINK-24646: --- Summary: Add operator wrapper for all-round iterations Key: FLINK-24646 URL: https://issues.apache.org/jira/browse/FLINK-24646 Project: Flink Issue Type: Sub-task Components: Library / Machine Learning Reporter: Yun Gao Assignee: Yun Gao Fix For: 0.1.0 Inside the iteration, we will also broadcast the special events to mark the end of rounds. To process these events, all the operators inside the iteration is wrapped with a specialized wrapper operator. There are two kinds of wrappers: the first wrapper would not recreate the users' operator for each round, and the second one would. This issue would implement the first kind. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24645) Implements the feedback mechanism and the controller operators
Yun Gao created FLINK-24645: --- Summary: Implements the feedback mechanism and the controller operators Key: FLINK-24645 URL: https://issues.apache.org/jira/browse/FLINK-24645 Project: Flink Issue Type: Sub-task Components: Library / Machine Learning Reporter: Yun Gao Assignee: Yun Gao Fix For: 0.1.0 To iterates record, we need specialized mechanism to feedback the records from the tail of the iteration body back to the head of the iteration body. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24644) Introduce Add broadcast output to broadcast events to all the downstream tasks
Yun Gao created FLINK-24644: --- Summary: Introduce Add broadcast output to broadcast events to all the downstream tasks Key: FLINK-24644 URL: https://issues.apache.org/jira/browse/FLINK-24644 Project: Flink Issue Type: Sub-task Components: Library / Machine Learning Reporter: Yun Gao Assignee: Yun Gao Fix For: 0.1.0 Iteration requires to broadcast the epoch-watermark inside iteration no matter what partitioner is used. Currently it is not directly supported by the DataStream API, thus we need use reflection to use some low-level operations to implement this functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24643) Introduce the flink-ml-iteration module and iteration api
Yun Gao created FLINK-24643: --- Summary: Introduce the flink-ml-iteration module and iteration api Key: FLINK-24643 URL: https://issues.apache.org/jira/browse/FLINK-24643 Project: Flink Issue Type: Sub-task Reporter: Yun Gao Assignee: Yun Gao To startup, we need to create the flink-ml-iteration module inside the flink-ml project, and also add the api as listed in FLIP-176 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24642) Implements unified iteration to support algorithms in flink-ml
Yun Gao created FLINK-24642: --- Summary: Implements unified iteration to support algorithms in flink-ml Key: FLINK-24642 URL: https://issues.apache.org/jira/browse/FLINK-24642 Project: Flink Issue Type: New Feature Components: Library / Machine Learning Reporter: Yun Gao Assignee: Yun Gao Fix For: 0.1.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24641) Flink SQL FileSystem read hdfs Path, when Path's file name change, the sql will not find the file.
Hadi created FLINK-24641:
Summary: Flink SQL FileSystem read hdfs Path, when Path's file
name change, the sql will not find the file.
Key: FLINK-24641
URL: https://issues.apache.org/jira/browse/FLINK-24641
Project: Flink
Issue Type: Bug
Components: Table SQL / Client
Affects Versions: 1.13.2
Reporter: Hadi
Hello, big bro.
when I use the Flink SQL to read HDFS Files like this:
{code:java}
CREATE TABLE `cfg_city`(
`provincecode` int, `city_id` int, `city_name` string)
WITH (
'connector'='filesystem',
'path'='viewfs://path1/path2/path3/cfg_city',
'format' = 'csv',
'csv.field-delimiter' = ',',
'csv.ignore-parse-errors' = 'true'
)
;
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[RESULT][VOTE] FLIP-176: Unified Iteration to Support Algorithms (Flink ML)
Hi all, I'm glad to annouce that we have unanimously approved this FLIP. There are 4 approved votings, 3 of which are binding: Guowei Ma (binding) Till Rohrmann (binding) Yunfeng Zhou (non-binding) Becket Qin (binding) Very thanks everyone for the dicussion and votings! Best, Yun --Original Mail -- Sender:Becket Qin Send Date:Mon Oct 25 18:05:16 2021 Recipients:dev Subject:Re: [VOTE] FLIP-176: Unified Iteration to Support Algorithms (Flink ML) +1 (binding) Thanks for the FLIP. Jiangjie (Becket) Qin On Mon, Oct 25, 2021 at 4:45 PM Yunfeng Zhou wrote: > Sorry that I misunderstood the usage of "binding". I am not a Flink > committer so my vote should be a non-binding one. > > Best, > Yunfeng > > On Mon, Oct 25, 2021 at 4:33 PM Yunfeng Zhou > wrote: > > > Excellent work to support iteration for Flink. > > > > +1 (binding) > > > > Best regards, > > Yunfeng > > > > On Sat, Oct 23, 2021 at 12:27 AM Till Rohrmann > > wrote: > > > >> Thanks for creating this FLIP. > >> > >> +1 (binding) > >> > >> Cheers, > >> Till > >> > >> > >> On Fri, Oct 22, 2021 at 6:29 AM Guowei Ma wrote: > >> > >> > +1 (binding) > >> > > >> > Best, > >> > Guowei > >> > > >> > > >> > On Thu, Oct 21, 2021 at 3:58 PM Yun Gao > > >> > wrote: > >> > > >> > > > >> > > Hi all, > >> > > > >> > > We would like to start the vote for FLIP-176: Unified Iteration to > >> > Support > >> > > Algorithms (Flink ML) [1]. > >> > > This FLIP was discussed in this thread [2][3]. The FLIP-176 targets > at > >> > > implementing the iteration > >> > > API in flink-ml to support the implementation of the algorithms. > >> > > > >> > > The vote will be open for at least 72 hours till 26th Oct morning, > >> > > including the weekend. Very thanks! > >> > > > >> > > Best, > >> > > Yun > >> > > > >> > > [1] https://cwiki.apache.org/confluence/x/hAEBCw > >> > > [2] > >> > > > >> > > >> > https://lists.apache.org/thread.html/r72e87a71b14baac3d7d16268346f5fc7c65f1de989e00b4ab2aab9ab%40%3Cdev.flink.apache.org%3E > >> > > [3] > >> > > > >> > > >> > https://lists.apache.org/thread.html/r63914a616de05a91dbe8e1a3208eb2b7c7c840c5c366bbd224483754%40%3Cdev.flink.apache.org%3E > >> > > >> > > >
Re: [DISCUSS] Creating an external connector repository
Hi folks, I think some questions came up and I'd like to address the question of the timing. Could you clarify what release cadence you're thinking of? There's quite > a big range that fits "more frequent than Flink" (per-commit, daily, > weekly, bi-weekly, monthly, even bi-monthly). The short answer is: as often as needed: - If there is a CVE in a dependency and we need to bump it - release immediately. - If there is a new feature merged, release soonish. We may collect a few successive features before a release. - If there is a bugfix, release immediately or soonish depending on the severity and if there are workarounds available. We should not limit ourselves; the whole idea of independent releases is exactly that you release as needed. There is no release planning or anything needed, you just go with a release as if it was an external artifact. (1) is the connector API already stable? > From another discussion thread [1], connector API is far from stable. > Currently, it's hard to build connectors against multiple Flink versions. > There are breaking API changes both in 1.12 -> 1.13 and 1.13 -> 1.14 and > maybe also in the future versions, because Table related APIs are still > @PublicEvolving and new Sink API is still @Experimental. > The question is: what is stable in an evolving system? We recently discovered that the old SourceFunction needed to be refined such that cancellation works correctly [1]. So that interface is in Flink since 7 years, heavily used also outside, and we still had to change the contract in a way that I'd expect any implementer to recheck their implementation. It might not be necessary to change anything and you can probably change the the code for all Flink versions but still, the interface was not stable in the closest sense. If we focus just on API changes on the unified interfaces, then we expect one more change to Sink API to support compaction. For Table API, there will most likely also be some changes in 1.15. So we could wait for 1.15. But I'm questioning if that's really necessary because we will add more functionality beyond 1.15 without breaking API. For example, we may add more unified connector metrics. If you want to use it in your connector, you have to support multiple Flink versions anyhow. So rather then focusing the discussion on "when is stuff stable", I'd rather focus on "how can we support building connectors against multiple Flink versions" and make it as painless as possible. Chesnay pointed out to use different branches for different Flink versions which sounds like a good suggestion. With a mono-repo, we can't use branches differently anyways (there is no way to have release branches per connector without chaos). In these branches, we could provide shims to simulate future features in older Flink versions such that code-wise, the source code of a specific connector may not diverge (much). For example, to register unified connector metrics, we could simulate the current approach also in some utility package of the mono-repo. I see the stable core Flink API as a prerequisite for modularity. And > for connectors it is not just the source and sink API (source being > stable as of 1.14), but everything that is required to build and > maintain a connector downstream, such as the test utilities and > infrastructure. > That is a very fair point. I'm actually surprised to see that MiniClusterWithClientResource is not public. I see it being used in all connectors, especially outside of Flink. I fear that as long as we do not have connectors outside, we will not properly annotate and maintain these utilties in a classic hen-and-egg-problem. I will outline an idea at the end. > the connectors need to be adopted and require at least one release per > Flink minor release. > However, this will make the releases of connectors slower, e.g. maintain > features for multiple branches and release multiple branches. > I think the main purpose of having an external connector repository is in > order to have "faster releases of connectors"? > > Imagine a project with a complex set of dependencies. Let's say Flink > version A plus Flink reliant dependencies released by other projects > (Flink-external connectors, Beam, Iceberg, Hudi, ..). We don't want a > situation where we bump the core Flink version to B and things fall > apart (interface changes, utilities that were useful but not public, > transitive dependencies etc.). > Yes, that's why I wanted to automate the processes more which is not that easy under ASF. Maybe we automate the source provision across supported versions and have 1 vote thread for all versions of a connector? >From the perspective of CDC connector maintainers, the biggest advantage of > maintaining it outside of the Flink project is that: > 1) we can have a more flexible and faster release cycle > 2) we can be more liberal with committership for connector maintainers > which can also attract more committers to help the release. > >
