[jira] [Created] (FLINK-31720) Pulsar connector v3.0 branch's docs are empty

2023-04-04 Thread Yuxin Tan (Jira)
Yuxin Tan created FLINK-31720:
-

 Summary: Pulsar connector v3.0 branch's docs are empty
 Key: FLINK-31720
 URL: https://issues.apache.org/jira/browse/FLINK-31720
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Pulsar
Affects Versions: pulsar-3.0.1
Reporter: Yuxin Tan


Currently, the docs of Pulsar connector v3.0 branch are empty. We should add 
the docs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration

2023-04-04 Thread yuxia
Thanks Feng for driving this FLIP. 
I have few comments:
1: The mechanism of handling catalog with the same name looks a little of 
complex to me. I think it'll be better to explain it in the java doc of these 
methods and give a brief example in this FLIP.

2: TBH, the method name `addCatalog` still looks confused to me. IIUC, this 
method is for storing catalog to CatalogStore, how about renaming it to 
`storeCatalog`? It's very personal opinion, you can decide to take it or not by 
your self.

3: For CREATE CATALOG statement, which method will be called? `registerCatalog` 
or `addCatalog`? I'm wondering whether user can add a catalog to store with SQL 
stement.


3: Is it really neccessary to provide a default implmentation for interface 
`CatalogStoreFactory`? 


4: About asynchronous registration for catalog.

> When creating a catalog with CREATE CATALOG, the asynchronous registration 
> method is used by default.
If asynchronous registration is the default behavior, it there any way that 
user can switch to synchronous registration just like before?
Will both method `addCatalog` and `registerCatalog` be asynchronous 
registration?

IIUC, in asynchronous registration, it may well that CREATE CATALOG executes 
successfully, but then the following CREATE TABLE statement will fail for the 
catalog fail to open.
I think it's a break change which should be highlighted in this FLIP, may be in 
compatibility part.


BTW, by saying asynchronous registration, I would like to expect there will be 
an executor to open or register catalog in the background, but from your 
previous comments,
"the Catalog instance will be initialized if it has not been initialized yet. 
If the initialization process fails, these statements will not be executed 
successfully."
It looks more like lazy initialization for catalog than asynchronous 
registration, right?


Best regards,
Yuxia

- 原始邮件 -
发件人: "Feng Jin" 
收件人: "dev" 
发送时间: 星期一, 2023年 4 月 03日 下午 3:27:45
主题: Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and 
asynchronous registration

Hi everyone, Thank you all for your interest in this DISCUSS.

@Shammon
> How to handle a catalog with the same name that exists for both of them?

I believe this is a crucial point. Based on my current personal
understanding, the Map catalogs will serve as a cache
for instantiated catalogs and have the highest priority.

There are three methods that can affect the Map catalogs:

1. registerCatalog(String catalogName, Catalog catalog)

This method puts the catalog instance into the Map catalogs.

2. unregisterCatalog(String catalogName)

This method removes the catalog instance corresponding to catalogName
from the Map catalogs.

3. getCatalog(String catalogName)

This method first retrieves the corresponding catalog instance from
the Map catalogs. If the catalog does not exist, it
retrieves the corresponding configuration from the CatalogStore,
initializes it, and puts the initialized Catalog instance into the
Map catalogs.

The following two methods only modify the configuration in the CatalogStore:

1. addCatalog(String catalogName, Map properties)

This method saves the properties to the catalogStore and checks
whether there is a catalogName with the same name.

2. removeCatalog(String catalogName)
This method removes the specified configuration of the specified
catalogName in the catalogStore.

The following are possible conflict scenarios:

1. When the corresponding catalogName already exists in the
CatalogStore but not in the Map, the
registerCatalog(String catalogName, Catalog catalog) method can
succeed and be directly saved to the Map catalogs.

2. When the corresponding catalogName already exists in both the
CatalogStore and the Map, the registerCatalog(String
catalogName, Catalog catalog) method will fail.

3. When the corresponding catalogName already exists in the
Map, the addCatalog(String catalogName, Map properties) method can directly save the properties to the
catalogStore, but the getCatalog(String catalogName) method will not
use the new properties for initialization because the corresponding
catalog instance already exists in catalogs and will be prioritized.
Therefore, using the unregisterCatalog(String catalogName) method to
remove the instance corresponding to the original catalogName is
necessary.



> I think it will confuse users that `registerCatalog(String 
> catalogName,Catalog catalog)` in the `Map catalogs` and 
> `registerCatalog(String catalogName, Map properties)

This could potentially lead to confusion. I suggest changing the
method name, perhaps to addCatalog(String catalogName, Map properties), as previously mentioned


@Hang
> add `registerCatalog(String catalogName,Catalog catalog,
boolean lazyInit)` method

Since a catalog is already an instance, adding the "lazyInit"
parameter to the registerCatalog(String catalogName, Catalog catalog)
method may not necessarily result in lazy initialization.

> Do we need to think about

[jira] [Created] (FLINK-31721) Move JobStatusHook to flink-core module

2023-04-04 Thread tartarus (Jira)
tartarus created FLINK-31721:


 Summary: Move JobStatusHook to flink-core module
 Key: FLINK-31721
 URL: https://issues.apache.org/jira/browse/FLINK-31721
 Project: Flink
  Issue Type: Sub-task
  Components: API / Core
Reporter: tartarus


Flink Sql needs to use JobStatusHook mechanism to implement atomic CTAS 
semantics, but the Table part module can't access flink-runtime module, so we 
need to move JobStatusHook to flink-core module



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration

2023-04-04 Thread Shammon FY
Hi Feng

I think if there is a `registerCatalog` method in `CatalogManager`, it will
confuse users whether a method named `addCatalog` or `storeCatalog` is
added.

And as you mentioned, the memory catalog is a `cache`, I think the concept
of `cache` should not be exposed to users.

I found there is a pre-discussion [1] for this FLIP. Please correct me if
I'm wrong, IIUC, the conclusion of that discussion is to use
`CatalogManager` as an interface and implement it for different stores such
as memory, file and external system.

I think there is a gap between the current FLIP design and that conclusion.
What about the proposal of the discussion in thread [1] ?


[1] https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70


Best,
Shammon FY


On Tue, Apr 4, 2023 at 3:41 PM yuxia  wrote:

> Thanks Feng for driving this FLIP.
> I have few comments:
> 1: The mechanism of handling catalog with the same name looks a little of
> complex to me. I think it'll be better to explain it in the java doc of
> these methods and give a brief example in this FLIP.
>
> 2: TBH, the method name `addCatalog` still looks confused to me. IIUC,
> this method is for storing catalog to CatalogStore, how about renaming it
> to `storeCatalog`? It's very personal opinion, you can decide to take it or
> not by your self.
>
> 3: For CREATE CATALOG statement, which method will be called?
> `registerCatalog` or `addCatalog`? I'm wondering whether user can add a
> catalog to store with SQL stement.
>
>
> 3: Is it really neccessary to provide a default implmentation for
> interface `CatalogStoreFactory`?
>
>
> 4: About asynchronous registration for catalog.
>
> > When creating a catalog with CREATE CATALOG, the asynchronous
> registration method is used by default.
> If asynchronous registration is the default behavior, it there any way
> that user can switch to synchronous registration just like before?
> Will both method `addCatalog` and `registerCatalog` be asynchronous
> registration?
>
> IIUC, in asynchronous registration, it may well that CREATE CATALOG
> executes successfully, but then the following CREATE TABLE statement will
> fail for the catalog fail to open.
> I think it's a break change which should be highlighted in this FLIP, may
> be in compatibility part.
>
>
> BTW, by saying asynchronous registration, I would like to expect there
> will be an executor to open or register catalog in the background, but from
> your previous comments,
> "the Catalog instance will be initialized if it has not been initialized
> yet. If the initialization process fails, these statements will not be
> executed successfully."
> It looks more like lazy initialization for catalog than asynchronous
> registration, right?
>
>
> Best regards,
> Yuxia
>
> - 原始邮件 -
> 发件人: "Feng Jin" 
> 收件人: "dev" 
> 发送时间: 星期一, 2023年 4 月 03日 下午 3:27:45
> 主题: Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration
> and asynchronous registration
>
> Hi everyone, Thank you all for your interest in this DISCUSS.
>
> @Shammon
> > How to handle a catalog with the same name that exists for both of them?
>
> I believe this is a crucial point. Based on my current personal
> understanding, the Map catalogs will serve as a cache
> for instantiated catalogs and have the highest priority.
>
> There are three methods that can affect the Map catalogs:
>
> 1. registerCatalog(String catalogName, Catalog catalog)
>
> This method puts the catalog instance into the Map
> catalogs.
>
> 2. unregisterCatalog(String catalogName)
>
> This method removes the catalog instance corresponding to catalogName
> from the Map catalogs.
>
> 3. getCatalog(String catalogName)
>
> This method first retrieves the corresponding catalog instance from
> the Map catalogs. If the catalog does not exist, it
> retrieves the corresponding configuration from the CatalogStore,
> initializes it, and puts the initialized Catalog instance into the
> Map catalogs.
>
> The following two methods only modify the configuration in the
> CatalogStore:
>
> 1. addCatalog(String catalogName, Map properties)
>
> This method saves the properties to the catalogStore and checks
> whether there is a catalogName with the same name.
>
> 2. removeCatalog(String catalogName)
> This method removes the specified configuration of the specified
> catalogName in the catalogStore.
>
> The following are possible conflict scenarios:
>
> 1. When the corresponding catalogName already exists in the
> CatalogStore but not in the Map, the
> registerCatalog(String catalogName, Catalog catalog) method can
> succeed and be directly saved to the Map catalogs.
>
> 2. When the corresponding catalogName already exists in both the
> CatalogStore and the Map, the registerCatalog(String
> catalogName, Catalog catalog) method will fail.
>
> 3. When the corresponding catalogName already exists in the
> Map, the addCatalog(String catalogName, Map String> properties) method can directly save the properties to the
> catalogStore, but the get

[jira] [Created] (FLINK-31722) Remove dependency on flink-shaded

2023-04-04 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-31722:
--

 Summary: Remove dependency on flink-shaded
 Key: FLINK-31722
 URL: https://issues.apache.org/jira/browse/FLINK-31722
 Project: Flink
  Issue Type: Technical Debt
  Components: Connectors / Cassandra
Reporter: Martijn Visser


The Cassandra connector relies on flink-shaded and uses Flinks' shaded Guava. 
With the externalization of connector, connectors shouldn't rely on 
Flink-Shaded but instead shade dependencies such as this one themselves



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (FLINK-31723) DispatcherTest#testCancellationDuringInitialization is unstable

2023-04-04 Thread Sergey Nuyanzin (Jira)
Sergey Nuyanzin created FLINK-31723:
---

 Summary: DispatcherTest#testCancellationDuringInitialization is 
unstable
 Key: FLINK-31723
 URL: https://issues.apache.org/jira/browse/FLINK-31723
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.18.0
Reporter: Sergey Nuyanzin


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=47889&view=logs&j=0e7be18f-84f2-53f0-a32d-4a5e4a174679&t=7c1d86e3-35bd-5fd5-3b7c-30c126a78702&l=8487

{noformat}
Apr 04 02:29:17 [ERROR] Failures: 
Apr 04 02:29:17 [ERROR]   
DispatcherTest.testCancellationDuringInitialization:389 
Apr 04 02:29:17 Expected: is 
Apr 04 02:29:17  but: was 

{noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (FLINK-31724) SqlClientITCase.testMatchRecognize fails with "bash -c rm -rf /opt/flink/checkpoint/*" returned non-zero exit code 1

2023-04-04 Thread Sergey Nuyanzin (Jira)
Sergey Nuyanzin created FLINK-31724:
---

 Summary: SqlClientITCase.testMatchRecognize fails with "bash -c rm 
-rf /opt/flink/checkpoint/*" returned non-zero exit code 1
 Key: FLINK-31724
 URL: https://issues.apache.org/jira/browse/FLINK-31724
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Client
Affects Versions: 1.18.0
Reporter: Sergey Nuyanzin


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=47893&view=logs&j=af184cdd-c6d8-5084-0b69-7e9c67b35f7a&t=160c9ae5-96fd-516e-1c91-deb81f59292a&l=12715

{noformat}
2023-04-04T08:11:47.8737556Z Apr 04 08:11:47 Command "bash -c rm -rf 
/opt/flink/checkpoint/*" returned non-zero exit code 1. 
2023-04-04T08:11:47.8737861Z Apr 04 08:11:47 STDOUT: 
2023-04-04T08:11:47.8738297Z Apr 04 08:11:47 STDERR: rm: cannot remove 
'/opt/flink/checkpoint/e2b7cbfc940e5f066e587037f80e74af': Directory not empty
2023-04-04T08:11:47.8738611Z Apr 04 08:11:47 
2023-04-04T08:11:47.8738971Z Apr 04 08:11:47at 
org.apache.flink.connector.testframe.container.FlinkContainers.deleteJobManagerTemporaryFiles(FlinkContainers.java:471)
2023-04-04T08:11:47.8740127Z Apr 04 08:11:47at 
org.apache.flink.connector.testframe.container.FlinkContainers.stop(FlinkContainers.java:241)
2023-04-04T08:11:47.8740803Z Apr 04 08:11:47at 
SqlClientITCase.tearDown(SqlClientITCase.java:114)
2023-04-04T08:11:47.8741144Z Apr 04 08:11:47at 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2023-04-04T08:11:47.8741677Z Apr 04 08:11:47at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2023-04-04T08:11:47.8742090Z Apr 04 08:11:47at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2023-04-04T08:11:47.8742463Z Apr 04 08:11:47at 
java.lang.reflect.Method.invoke(Method.java:498)
2023-04-04T08:11:47.8742825Z Apr 04 08:11:47at 
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
2023-04-04T08:11:47.8743253Z Apr 04 08:11:47at 
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
2023-04-04T08:11:47.8743709Z Apr 04 08:11:47at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
2023-04-04T08:11:47.873Z Apr 04 08:11:47at 
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
2023-04-04T08:11:47.8744880Z Apr 04 08:11:47at 
org.junit.jupiter.engine.extension.TimeoutExtension.interceptLifecycleMethod(TimeoutExtension.java:128)
2023-04-04T08:11:47.8745318Z Apr 04 08:11:47at 
org.junit.jupiter.engine.extension.TimeoutExtension.interceptAfterEachMethod(TimeoutExtension.java:110)
2023-04-04T08:11:47.8745812Z Apr 04 08:11:47at 
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
2023-04-04T08:11:47.8746540Z Apr 04 08:11:47at 
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
2023-04-04T08:11:47.8747033Z Apr 04 08:11:47at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
2023-04-04T08:11:47.8747515Z Apr 04 08:11:47at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
2023-04-04T08:11:47.8747969Z Apr 04 08:11:47at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
2023-04-04T08:11:47.8748418Z Apr 04 08:11:47at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
2023-04-04T08:11:47.8748845Z Apr 04 08:11:47at 
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
2023-04-04T08:11:47.8749300Z Apr 04 08:11:47at 
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
2023-04-04T08:11:47.8749890Z Apr 04 08:11:47at 
org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeMethodInExtensionContext(ClassBasedTestDescriptor.java:520)
2023-04-04T08:11:47.8750393Z Apr 04 08:11:47at 
org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$synthesizeAfterEachMethodAdapter$24(ClassBasedTestDescriptor.java:510)
2023-04-04T08:11:47.8751271Z Apr 04 08:11:47at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAfterEachMethods$10(TestMethodTestDescriptor.java:243)
2023-04-04T08:11:47.8751861Z Apr 04 08:11:47at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAllAfterMethodsOrCallbacks$13(TestMethodTestDescriptor.java:276)
2023-04-04T08:11:47.8752334Z Ap

Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration

2023-04-04 Thread Feng Jin
Thank you for your reply. I am very sorry for the misunderstanding caused
by my deviation from the original discussion.

@Shammon
> I found there is a pre-discussion [1] for this FLIP
Yes, there was indeed such a discussion before.  However, in designing the
whole solution, I found that the logic of CatalogManager itself doesn't
need to change much. *We cannot only persist Catalog instances themselves*,
so exposing only registerCatalog(String catalogName, Catalog catalog) might
not be enough to save Catalogs, because in the end we still need to save
the configurations corresponding to the Catalog instances.  Therefore, I
decided to introduce the CatalogStore interface for configuration
persistence. Regarding this part, I also took inspiration from Trino's
implementation[1].


@yuxia

> 1: The mechanism of handling catalog with the same name looks a little of
complex to me.
Thank you for the suggestion. I will provide a detailed description and
code examples for this part, and add it to the FLIP.

> 2: TBH, the method name `addCatalog` still looks confused to me. IIUC,
this method is for storing catalog to CatalogStore, how about renaming it
to `storeCatalog`? It's very personal opinion, you can decide to take it or
not by your self.
StoreCatalog looks more intuitive to me. I don't see any problem with it.

> 3.For CREATE CATALOG statement, which method will be called?
`registerCatalog` or `addCatalog`? I'm wondering whether user can add a
catalog to store with SQL stement.
For CREATE CATALOG, my original design was to add it directly to the
CatalogStore, but this would disrupt the existing logic. Therefore, I think
we can do both: save the configuration to the CatalogStore and initialize
the Catalog instance at the same time

> 3. Is it really neccessary to provide a default implmentation for
interface `CatalogStoreFactory`?
I think it is necessary, otherwise we would need to introduce an additional
Map to store the configuration for lazy loading.

> 4: About asynchronous registration for catalog.
I don't think registerCatalog(String catalogName, Catalog catalog) can be
made into an asynchronous interface because Catalog is already an instance.

> It looks more like lazy initialization for catalog than asynchronous
registration, right?
Indeed, my description was inaccurate. It should be lazy registration
instead of asynchronous registration. I have already updated the title of
the FLIP.




[1].
https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java


On Tue, Apr 4, 2023 at 4:27 PM Shammon FY  wrote:

> Hi Feng
>
> I think if there is a `registerCatalog` method in `CatalogManager`, it will
> confuse users whether a method named `addCatalog` or `storeCatalog` is
> added.
>
> And as you mentioned, the memory catalog is a `cache`, I think the concept
> of `cache` should not be exposed to users.
>
> I found there is a pre-discussion [1] for this FLIP. Please correct me if
> I'm wrong, IIUC, the conclusion of that discussion is to use
> `CatalogManager` as an interface and implement it for different stores such
> as memory, file and external system.
>
> I think there is a gap between the current FLIP design and that conclusion.
> What about the proposal of the discussion in thread [1] ?
>
>
> [1] https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
>
>
> Best,
> Shammon FY
>
>
> On Tue, Apr 4, 2023 at 3:41 PM yuxia  wrote:
>
> > Thanks Feng for driving this FLIP.
> > I have few comments:
> > 1: The mechanism of handling catalog with the same name looks a little of
> > complex to me. I think it'll be better to explain it in the java doc of
> > these methods and give a brief example in this FLIP.
> >
> > 2: TBH, the method name `addCatalog` still looks confused to me. IIUC,
> > this method is for storing catalog to CatalogStore, how about renaming it
> > to `storeCatalog`? It's very personal opinion, you can decide to take it
> or
> > not by your self.
> >
> > 3: For CREATE CATALOG statement, which method will be called?
> > `registerCatalog` or `addCatalog`? I'm wondering whether user can add a
> > catalog to store with SQL stement.
> >
> >
> > 3: Is it really neccessary to provide a default implmentation for
> > interface `CatalogStoreFactory`?
> >
> >
> > 4: About asynchronous registration for catalog.
> >
> > > When creating a catalog with CREATE CATALOG, the asynchronous
> > registration method is used by default.
> > If asynchronous registration is the default behavior, it there any way
> > that user can switch to synchronous registration just like before?
> > Will both method `addCatalog` and `registerCatalog` be asynchronous
> > registration?
> >
> > IIUC, in asynchronous registration, it may well that CREATE CATALOG
> > executes successfully, but then the following CREATE TABLE statement will
> > fail for the catalog fail to open.
> > I think it's a break change which should be highlighted in this FLIP, may
> > be in compatibility part.

Re: [DISCUSS] Planning Flink 1.18

2023-04-04 Thread Jing Ge
Thanks Qingsheng for your coordination and thanks to all previous release
managers for sharing your thoughts!

Best regards,
Jing

On Sat, Apr 1, 2023 at 5:54 PM Jark Wu  wrote:

> Thank Konstantin, Qingsheng, Jing, Sergey for volunteering releases
> managers.
> Thank Qingsheng for being the release manager for the two consecutive
> releases,
> your experience in v1.17 will greatly help the 1.18 release cycle.
>
> Feature freeze in the middle of July sounds good to me.
>
> Best,
> Jark
>
> > 2023年3月30日 19:33,Martijn Visser  写道:
> >
> > Hi all,
> >
> > Thanks for stepping up as volunteers, much appreciated!
> >
> > Best regards,
> >
> > Martijn
> >
> > On Thu, Mar 30, 2023 at 5:59 AM Leonard Xu  wrote:
> >
> >> Thanks Konstantin and Qingsheng for kicking off and pushing forward the
> >> discussion.
> >>
> >> Thanks Qingsheng, Jing, Konstantin, Sergey, Yun for volunteering to
> >> release manager candidates, 4 release manager from different timezones
> make
> >> sense to me, it brings better coordination for globalization community
> >> contributors.
> >>
> >> About  the feature freeze date , I also propose the middle of July is
> >> better as we can preserve a buffer for special case just like we did in
> >> release 1.17 cycle.
> >>
> >> Btw, I’m willing to provide help  in 1.18 release cycle not as release
> >> manager too,please ping me if you need help.
> >>
> >> Best,
> >> Leonard
> >>
> >>
> >>> On Mar 30, 2023, at 11:33 AM, Qingsheng Ren  wrote:
> >>>
> >>> Hi everyone,
> >>>
> >>> I'd like to share some updates about release managers of Flink 1.18.
> >> Jing,
> >>> Konstantin, Sergey, Yun and I had a discussion offline and we reached a
> >>> consensus that having 5 release managers is a bit too much and might
> lead
> >>> to some overhead in communication and workload division. Yun would like
> >> to
> >>> provide some help in this release cycle but not as a release manager.
> >>>
> >>> Then we have 4 RM volunteers now for 1.18: Jing Ge, Konstantin Knauf,
> >>> Qingsheng Ren and Sergey Nuyanzin. Again thanks Yun and everyone for
> your
> >>> help and look forward to working with you in Flink 1.18!
> >>>
> >>> Best regards,
> >>> Qingsheng, together with Jing, Konstantin, Sergey and Yun
> >>>
> >>> On Mon, Mar 27, 2023 at 3:09 PM Matthias Pohl
> >>>  wrote:
> >>>
>  Thanks Konstantin for driving this.
>  I have no objections in regards to the feature freeze happening in
> July.
> 
>  Thanks everyone for volunteering to drive 1.18. +1 on the people who
>  offered to help
> 
>  PS: I shared learnings from the 1.17 release in the other thread [1]
> to
> >> not
>  spam this thread with 1.17-related stuff.
> 
>  [1] https://lists.apache.org/thread/1mdbpdo42df4642wtmf91b26p8v2qtbf
> 
> 
>  On Sat, Mar 25, 2023 at 4:00 AM Yun Tang  wrote:
> 
> > Thanks for kick off the discussion.
> >
> > I'm not sure whether I am a bit late here. I had experience on
> >> releasing
> > minor version of Flink-1.13.2 and also be interested in helping to
>  release
> > the next major Flink-1.18 version.
> >
> >
> > Best
> > Yun Tang
> > 
> > From: Sergey Nuyanzin 
> > Sent: Saturday, March 25, 2023 9:46
> > To: dev@flink.apache.org 
> > Cc: Jing Ge ; Konstantin Knauf <
> kna...@apache.org>
> > Subject: Re: [DISCUSS] Planning Flink 1.18
> >
> > Thanks for starting the discussion about Flink 1.18.
> > I would be interested in helping out around the release as well.
> >
> > On Sat, Mar 25, 2023 at 2:20 AM Qingsheng Ren 
> >> wrote:
> >
> >> Hi Konstantin,
> >>
> >> Thanks for kicking off 1.18! As mentioned by Jing we’d like to
> > participate
> >> in the new cycle, and hope my experience in 1.17 could help.
> >>
> >> About the feature freezing date, I’m also thinking to set it around
> >> mid
> >> July. 1.18 will be a larger release with more features to work with
> >> compared to 1.17. In 1.17 we have ~3 months for development so I
> think
> > 3.5
> >> months for 1.18 would be reasonable.
> >>
> >> Best,
> >> Qingsheng
> >>
> >> On Sat, Mar 25, 2023 at 07:37 Jing Ge 
> > wrote:
> >>
> >>> Hi Konstantin,
> >>>
> >>> Qingsheng and I would love to participate and join you as release
> >> managers
> >>> for 1.18.
> >>>
> >>> Speaking of the timeline, since we will have a big feature release
>  with
> >>> 1.18, a feature freeze in the middle of July would be better. It is
> > also
> >>> not far from your proposal.
> >>>
> >>> Best regards,
> >>> Jing
> >>>
> >>>
> >>> On Fri, Mar 24, 2023 at 7:35 PM Konstantin Knauf <
> kna...@apache.org>
> >>> wrote:
> >>>
>  Hi everyone,
> 
>  "Nach dem Spiel ist vor dem Spiel" [1] aka "There is always a next
>  release". With the announcement of Fli

[jira] [Created] (FLINK-31725) Synchronize dependency version between Flink and flink-connector-pulsar

2023-04-04 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-31725:
--

 Summary: Synchronize dependency version between Flink and 
flink-connector-pulsar
 Key: FLINK-31725
 URL: https://issues.apache.org/jira/browse/FLINK-31725
 Project: Flink
  Issue Type: Technical Debt
  Components: Connectors / Pulsar
Reporter: Martijn Visser
Assignee: Martijn Visser






--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (FLINK-31726) PyFlink module java.base does not "opens java.lang" to unnamed module

2023-04-04 Thread padavan (Jira)
padavan created FLINK-31726:
---

 Summary: PyFlink module java.base does not "opens java.lang" to 
unnamed module
 Key: FLINK-31726
 URL: https://issues.apache.org/jira/browse/FLINK-31726
 Project: Flink
  Issue Type: Bug
Reporter: padavan


I want to run simple example from Flink documentation. And after start i got 
exception:


{code:java}
Unable to make field private final byte[] java.lang.String.value accessible: 
module java.base does not "opens java.lang" to unnamed module @228575c0{code}
Installed:
{code:java}
Python 3.10.6 openjdk version "19.0.2" 2023-01-17 OpenJDK Runtime Environment 
(build 19.0.2+7-Ubuntu-0ubuntu322.04) OpenJDK 64-Bit Server VM (build 
19.0.2+7-Ubuntu-0ubuntu322.04, mixed mode, sharing){code}
Simple code from flink site:

[https://nightlies.apache.org/flink/flink-docs-master/api/python/examples/datastream/word_count.html]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (FLINK-31727) Source parallelism should match number of Kafka partitions

2023-04-04 Thread Yordan Pavlov (Jira)
Yordan Pavlov created FLINK-31727:
-

 Summary: Source parallelism should match number of Kafka partitions
 Key: FLINK-31727
 URL: https://issues.apache.org/jira/browse/FLINK-31727
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka
Affects Versions: 1.17.0
Reporter: Yordan Pavlov
 Attachments: fill-topic.sh, main.scala

We seem to have hit a problem in how Flink fires windows, the problem presents 
itself on job recovery. To simplify the problem I am attaching a simple self 
sustained Flink job which illustrates the problem. What we have inside is, a 
KafkaSource consuming a topic with 3 partitions with job parallelism of 1. 
Right after the source data is consumed we have a TimeWindow, watermarks are 
constructed based on event data. For simplicity, data in the Kafka topic is 
just integers, which are also used as watermarks. The topic looks like so:
{quote}partition 0: 0, 3, 6, 9, 12 ...

partition 1: 1, 4, 7, 10, 13 ...

partition 2: 2, 5, 8, 11, 14 ...
{quote}
What we expect, and what is the case before a restart, is for Flink to wait for 
watermarks to progress on each of the 3 partitions and only then trigger 
windows. If this is met, we have windows triggered for each consecutive number.

The problem we observe happens on restart, then Flink would start fire windows 
even though it seems to be reading data only from some of the partitions.

Please find attached program, I am also attaching a simple Bash script used to 
generate the Kafka input data.

[^main.scala]

[^fill-topic.sh]

 

We do not see the problem if we set parallelism to 3 and the job have 3 slots 
per task manager. Going through the documentation I did not see such 
requirement though, this may be hard to enforce for all jobs.

Regards



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[SUMMARY] Flink 1.18 Release Sync 4/4/2023

2023-04-04 Thread Jing Ge
Dear devs and users,

Today was the kickoff meeting for Flink 1.18 release cycle. I'd like to
share the info synced in the meeting.

Meeting schedule:

Zoom will be used with a 40 mins limit for each meeting. That should be
fine for now. We will check it again if we have an issue with the time
limit later.

Release cycle will start bi-weekly and switch to weekly after the feature
freeze.

Feature freezing date:

July 11, 2023

Retrospective of 1.17 release:

There are many valuable thoughts and suggestions from previous release
managers[1]. Some of them are summarized as following:

- [Attention] Backports/merges without PRs will break master/release
branches. Kindly remind, every Flink developer, please pay attention to
avoid doing it.
- It is encouraged to create release testing tasks in advance, label them
properly, and finish them earlier, not necessarily wait to do it at the end
of the release cycle.
- A non-votable rc0 will be released in the future for developers to
validate the release.
- Some jira tickets have been created for 1.17 release that could be used
as the starting point to build a standard release pipeline. The release
process documented on the wiki page could be turned into a list of Jira
tasks (Jira template) in the future.

Daily work divisions:

In general, every release manager will be working on all daily issues. For
some major tasks, in order to make sure there will at least always be
someone to take care of them, they have been assigned to specific release
managers[2]. If you need support in each of these areas, please don't
hesitate to contact us.

1.18 Release page:

Flink 1.18 release has been kicked off today. We'd like to invite you to
update your development plan on the release page[2].

The next release sync up meeting will be on April 18, 2023. Please feel
free to join us!

Zoom meeting:
https://us04web.zoom.us/j/79158702091?pwd=8CXPqxMzbabWkma5b0qFXI1IcLbxBh.1

Best regards,
Konstantin, Sergey, Qingsheng, and Jing

[1] https://cwiki.apache.org/confluence/display/FLINK/1.17+Release
[2] https://cwiki.apache.org/confluence/display/FLINK/1.18+Release


[jira] [Created] (FLINK-31728) Remove Scala API dependencies from batch/streaming examples

2023-04-04 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-31728:


 Summary: Remove Scala API dependencies from batch/streaming 
examples
 Key: FLINK-31728
 URL: https://issues.apache.org/jira/browse/FLINK-31728
 Project: Flink
  Issue Type: Technical Debt
  Components: Build System, Examples
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.18.0


The example modules have leftover Scala API dependencies and build 
infrastructure. Remove them, along with the scala suffix on these modules.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


what happened to the images in FLIP-24: SQL Client?

2023-04-04 Thread David Anderson
Does anyone know what happened to the diagrams that used to be in
FLIP-24: SQL Client? The last time I looked at this FLIP -- a few
weeks ago -- there were architecture diagrams for Gateway  Mode and
Embedded Mode, but now those images are missing.

David


[jira] [Created] (FLINK-31729) Unexpected UPDATE_BEFORE output record in LEFT OUTER JOIN

2023-04-04 Thread Alexander Smirnov (Jira)
Alexander Smirnov created FLINK-31729:
-

 Summary: Unexpected UPDATE_BEFORE output record in LEFT OUTER JOIN
 Key: FLINK-31729
 URL: https://issues.apache.org/jira/browse/FLINK-31729
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Runtime
Affects Versions: 1.17.0
Reporter: Alexander Smirnov
 Fix For: 1.18.0
 Attachments: image-2023-04-05-00-08-32-984.png

Currently, in streaming LEFT/RIGHT/FULL OUTER JOIN Flink SQL doesn't emit 
UPDATE_BEFORE/UPDATE_AFTER records, but instead explicitly change RowKind of 
output records to INSERT/DELETE for simplicity. However, it doesn't work as 
expected, because sometimes UPDATE_BEFORE rows can be emitted. What is more 
confusing - after UPDATE_BEFORE record there will be INSERT record (not 
UPDATE_AFTER), which can cause bugs in case when downstream operators process 
UPDATE records in a different way than INSERT/DELETE (for example, it can 
assume, that after UPDATE_BEFORE there should be UPDATE_AFTER record at some 
point of time).

How to reproduce:

Suppose we have tables "source1" and "source2":
CREATE TABLE source1(
  id int PRIMARY KEY,
  c3 bigint
) WITH (
  'connector' = 'kafka',
   ...
  'format' = 'debezium-json'
);

 

CREATE TABLE source2(
  id int PRIMARY KEY,
  c3 bigint
) WITH (
  'connector' = 'kafka',
   ...
  'format' = 'debezium-json'
);

And we execute the following query:
"select  t1.id, t1.c3,t2.id, t2.c3 from source1 t1 left join source2 t2 on 
t1.id = t2.id"

Then we insert records one by one:
source1: \{"before":null,"after":{"id":2,"c3":7121},"op":"c"}
source2: \{"before":null,"after":{"id":2,"c3":364},"op":"c"}
source1: \{"before":{"id":2,"c3":7121},"after":\{"id":2,"c3":7222},"op":"u"}
source2: \{"before":{"id":2,"c3":364},"after":\{"id":2,"c3":564},"op":"u"}

The result will be as in the following screenshot:

!image-2023-04-05-00-08-32-984.png!


Note, that after implementing ticket 
https://issues.apache.org/jira/browse/FLINK-17337 (support emitting 
UPDATE_BEFORE/UPDATE_AFTER records not only in inner join) the described error 
won't be relevant anymore.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: what happened to the images in FLIP-24: SQL Client?

2023-04-04 Thread Jing Ge
Hi David,

I just checked and both diagrams are still there. The last change of the
page was on Sep 18, 2022. Would you like to clean your cache or change to
another browser and try again?

Best regards,
Jing



On Tue, Apr 4, 2023 at 5:25 PM David Anderson  wrote:

> Does anyone know what happened to the diagrams that used to be in
> FLIP-24: SQL Client? The last time I looked at this FLIP -- a few
> weeks ago -- there were architecture diagrams for Gateway  Mode and
> Embedded Mode, but now those images are missing.
>
> David
>


[jira] [Created] (FLINK-31730) Support Ephemeral Storage in KubernetesConfigOptions

2023-04-04 Thread Zhenqiu Huang (Jira)
Zhenqiu Huang created FLINK-31730:
-

 Summary: Support Ephemeral Storage in KubernetesConfigOptions
 Key: FLINK-31730
 URL: https://issues.apache.org/jira/browse/FLINK-31730
 Project: Flink
  Issue Type: New Feature
  Components: Deployment / Kubernetes
Affects Versions: 1.18.0, 1.17.1
Reporter: Zhenqiu Huang


There is a common need to config flink main container with Ephemeral Storage 
size. It will be more user friendly to support it as a flink config.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: what happened to the images in FLIP-24: SQL Client?

2023-04-04 Thread Mate Czagany
Hi,

It does not work for me either, the images are hosted on Google Drawings
and it seems like it went private.
Maybe you have the images in your browser cache if you can see them. I see
that other FLIPs have their images uploaded to Confluence, I think that
would be a better way to store them.

Regards,
Mate

Jing Ge  ezt írta (időpont: 2023. ápr. 4., K,
19:24):

> Hi David,
>
> I just checked and both diagrams are still there. The last change of the
> page was on Sep 18, 2022. Would you like to clean your cache or change to
> another browser and try again?
>
> Best regards,
> Jing
>
>
>
> On Tue, Apr 4, 2023 at 5:25 PM David Anderson 
> wrote:
>
> > Does anyone know what happened to the diagrams that used to be in
> > FLIP-24: SQL Client? The last time I looked at this FLIP -- a few
> > weeks ago -- there were architecture diagrams for Gateway  Mode and
> > Embedded Mode, but now those images are missing.
> >
> > David
> >
>


Re: [VOTE] Release flink-connector-kafka, release candidate #1

2023-04-04 Thread Tzu-Li (Gordon) Tai
Hi all,

I've ported the critical fixes I mentioned to v3.0 and v4.0 branches of
apache/flink-connector-kafka now.

@martijnvis...@apache.org  let me know if you'd
need help with creating a new RC, if there's too much to juggle on
your end. Happy to help out.

Thanks,
Gordon

On Sun, Apr 2, 2023 at 11:21 PM Konstantin Knauf  wrote:

> +1. Thanks, Gordon!
>
> Am Mo., 3. Apr. 2023 um 06:37 Uhr schrieb Tzu-Li (Gordon) Tai <
> tzuli...@apache.org>:
>
> > Hi Martijn,
> >
> > Since this RC vote was opened, we had three critical bug fixes that was
> > merged for the Kafka connector:
> >
> >- https://issues.apache.org/jira/browse/FLINK-31363
> >- https://issues.apache.org/jira/browse/FLINK-31305
> >- https://issues.apache.org/jira/browse/FLINK-31620
> >
> > Given the severity of these issues (all of them are violations of
> > exactly-once semantics), and the fact that they are currently not
> included
> > yet in any released version, do you think it makes sense to cancel this
> RC
> > in favor of a new one that includes these?
> > Since this RC vote has been stale for quite some time already, it doesn't
> > seem like we're throwing away too much effort that has already been done
> if
> > we start a new RC with these critical fixes included.
> >
> > What do you think?
> >
> > Thanks,
> > Gordon
> >
> > On Thu, Feb 9, 2023 at 3:26 PM Tzu-Li (Gordon) Tai 
> > wrote:
> >
> > > +1 (binding)
> > >
> > > - Verified legals (license headers and root LICENSE / NOTICE file).
> > AFAICT
> > > no dependencies require explicit acknowledgement in the NOTICE files.
> > > - No binaries in staging area
> > > - Built source with tests
> > > - Verified signatures and hashes
> > > - Web PR changes LGTM
> > >
> > > Thanks Martijn!
> > >
> > > Cheers,
> > > Gordon
> > >
> > > On Mon, Feb 6, 2023 at 6:12 PM Mason Chen 
> > wrote:
> > >
> > >> That makes sense, thanks for the clarification!
> > >>
> > >> Best,
> > >> Mason
> > >>
> > >> On Wed, Feb 1, 2023 at 7:16 AM Martijn Visser <
> martijnvis...@apache.org
> > >
> > >> wrote:
> > >>
> > >> > Hi Mason,
> > >> >
> > >> > Thanks, [4] is indeed a copy-paste error and you've made the right
> > >> > assumption that
> > >> >
> > >> >
> > >>
> >
> https://repository.apache.org/content/repositories/orgapacheflink-1582/org/apache/flink/
> > >> > is the correct maven central link.
> > >> >
> > >> > I think we should use FLINK-30052 to move the Kafka connector code
> > from
> > >> the
> > >> > 1.17 release also over the Kafka connector repo (especially since
> > >> there's
> > >> > now a v3.0 branch for the Kafka connector, so it can be merged in
> > main).
> > >> > When those commits have been merged, we can make a next Kafka
> > connector
> > >> > release (which is equivalent to the 1.17 release, which can only be
> > done
> > >> > when 1.17 is done because of the split level watermark alignment)
> and
> > >> then
> > >> > FLINK-30859 can be finished.
> > >> >
> > >> > Best regards,
> > >> >
> > >> > Martijn
> > >> >
> > >> > Op wo 1 feb. 2023 om 09:16 schreef Mason Chen <
> mas.chen6...@gmail.com
> > >:
> > >> >
> > >> > > +1 (non-binding)
> > >> > >
> > >> > > * Verified hashes and signatures
> > >> > > * Verified no binaries
> > >> > > * Verified LICENSE and NOTICE files
> > >> > > * Verified poms point to 3.0.0-1.16
> > >> > > * Reviewed web PR
> > >> > > * Built from source
> > >> > > * Verified git tag
> > >> > >
> > >> > > I think [4] your is a copy-paste error and I did all the
> > verification
> > >> > > assuming that
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> https://repository.apache.org/content/repositories/orgapacheflink-1582/org/apache/flink/
> > >> > > is the correct maven central link.
> > >> > >
> > >> > > Regarding the release notes, should we close
> > >> > > https://issues.apache.org/jira/browse/FLINK-30052 and link it
> > there?
> > >> > I've
> > >> > > created https://issues.apache.org/jira/browse/FLINK-30859 to
> remove
> > >> the
> > >> > > existing code from the master branch.
> > >> > >
> > >> > > Best,
> > >> > > Mason
> > >> > >
> > >> > > On Tue, Jan 31, 2023 at 6:23 AM Martijn Visser <
> > >> martijnvis...@apache.org
> > >> > >
> > >> > > wrote:
> > >> > >
> > >> > > > Hi everyone,
> > >> > > > Please review and vote on the release candidate #1 for
> > >> > > > flink-connector-kafka version 3.0.0, as follows:
> > >> > > > [ ] +1, Approve the release
> > >> > > > [ ] -1, Do not approve the release (please provide specific
> > >> comments)
> > >> > > >
> > >> > > > Note: this is the same code as the Kafka connector for the Flink
> > >> 1.16
> > >> > > > release.
> > >> > > >
> > >> > > > The complete staging area is available for your review, which
> > >> includes:
> > >> > > > * JIRA release notes [1],
> > >> > > > * the official Apache source release to be deployed to
> > >> dist.apache.org
> > >> > > > [2],
> > >> > > > which are signed with the key with fingerprint
> > >> > > > A5F3BCE4CBE993573EC5966A65321B8382B219AF [3],
> > >> > > > * all artifact

Re: what happened to the images in FLIP-24: SQL Client?

2023-04-04 Thread Jing Ge
Hi Mate,

Thanks for the hint. I just updated the page and checked with a fresh,
different browser. It should work now. Would you like to try again?

Best regards,
Jing

On Tue, Apr 4, 2023 at 8:17 PM Mate Czagany  wrote:

> Hi,
>
> It does not work for me either, the images are hosted on Google Drawings
> and it seems like it went private.
> Maybe you have the images in your browser cache if you can see them. I see
> that other FLIPs have their images uploaded to Confluence, I think that
> would be a better way to store them.
>
> Regards,
> Mate
>
> Jing Ge  ezt írta (időpont: 2023. ápr. 4., K,
> 19:24):
>
> > Hi David,
> >
> > I just checked and both diagrams are still there. The last change of the
> > page was on Sep 18, 2022. Would you like to clean your cache or change to
> > another browser and try again?
> >
> > Best regards,
> > Jing
> >
> >
> >
> > On Tue, Apr 4, 2023 at 5:25 PM David Anderson 
> > wrote:
> >
> > > Does anyone know what happened to the diagrams that used to be in
> > > FLIP-24: SQL Client? The last time I looked at this FLIP -- a few
> > > weeks ago -- there were architecture diagrams for Gateway  Mode and
> > > Embedded Mode, but now those images are missing.
> > >
> > > David
> > >
> >
>


[jira] [Created] (FLINK-31731) No suitable constructor found for DebeziumAvroSerializationSchema

2023-04-04 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-31731:
--

 Summary: No suitable constructor found for 
DebeziumAvroSerializationSchema
 Key: FLINK-31731
 URL: https://issues.apache.org/jira/browse/FLINK-31731
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka
Affects Versions: 1.18.0, kafka-4.0.0
Reporter: Martijn Visser


{code:java}
Error:  Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.8.0:testCompile 
(default-testCompile) on project flink-connector-kafka: Compilation failure
Error:  
/home/runner/work/flink-connector-kafka/flink-connector-kafka/flink-connector-kafka/src/test/java/org/apache/flink/streaming/connectors/kafka/table/KafkaDynamicTableFactoryTest.java:[939,16]
 no suitable constructor found for 
DebeziumAvroSerializationSchema(org.apache.flink.table.types.logical.RowType,java.lang.String,java.lang.String,)
Error:  constructor 
org.apache.flink.formats.avro.registry.confluent.debezium.DebeziumAvroSerializationSchema.DebeziumAvroSerializationSchema(org.apache.flink.table.types.logical.RowType,java.lang.String,java.lang.String,java.lang.String,java.util.Map)
 is not applicable
Error:(actual and formal argument lists differ in length)
Error:  constructor 
org.apache.flink.formats.avro.registry.confluent.debezium.DebeziumAvroSerializationSchema.DebeziumAvroSerializationSchema(org.apache.flink.formats.avro.AvroRowDataSerializationSchema)
 is not applicable
Error:(actual and formal argument lists differ in length)
Error:  -> [Help 1]
Error:  
Error:  To see the full stack trace of the errors, re-run Maven with the -e 
switch.
Error:  Re-run Maven using the -X switch to enable full debug logging.
Error:  
Error:  For more information about the errors and possible solutions, please 
read the following articles:
Error:  [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
Error:  
Error:  After correcting the problems, you can resume the build with the command
Error:mvn  -rf :flink-connector-kafka
Error: Process completed with exit code 1.
{code}

https://github.com/apache/flink-connector-kafka/actions/runs/4610715024/jobs/8149513647#step:13:153



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: what happened to the images in FLIP-24: SQL Client?

2023-04-04 Thread Mate Czagany
Hi,

Thank you, it works perfectly now for me.

Regards,
Mate

Jing Ge  ezt írta (időpont: 2023. ápr. 4., K,
20:32):

> Hi Mate,
>
> Thanks for the hint. I just updated the page and checked with a fresh,
> different browser. It should work now. Would you like to try again?
>
> Best regards,
> Jing
>
> On Tue, Apr 4, 2023 at 8:17 PM Mate Czagany  wrote:
>
> > Hi,
> >
> > It does not work for me either, the images are hosted on Google Drawings
> > and it seems like it went private.
> > Maybe you have the images in your browser cache if you can see them. I
> see
> > that other FLIPs have their images uploaded to Confluence, I think that
> > would be a better way to store them.
> >
> > Regards,
> > Mate
> >
> > Jing Ge  ezt írta (időpont: 2023. ápr. 4.,
> K,
> > 19:24):
> >
> > > Hi David,
> > >
> > > I just checked and both diagrams are still there. The last change of
> the
> > > page was on Sep 18, 2022. Would you like to clean your cache or change
> to
> > > another browser and try again?
> > >
> > > Best regards,
> > > Jing
> > >
> > >
> > >
> > > On Tue, Apr 4, 2023 at 5:25 PM David Anderson 
> > > wrote:
> > >
> > > > Does anyone know what happened to the diagrams that used to be in
> > > > FLIP-24: SQL Client? The last time I looked at this FLIP -- a few
> > > > weeks ago -- there were architecture diagrams for Gateway  Mode and
> > > > Embedded Mode, but now those images are missing.
> > > >
> > > > David
> > > >
> > >
> >
>


[DISCUSS] EXACTLY_ONCE delivery semantics for upsert-kafka connector

2023-04-04 Thread Alexander Sorokoumov
Hello Flink community,

I would like to discuss if it is worth adding EXACTLY_ONCE delivery
semantics to upsert-kafka connector. According to upsert-kafka docs[1] and
ReducingUpsertSink javadoc[2], the connector is correct even with duplicate
records under AT_LEAST_ONCE because the records are idempotent, and the
read path de-duplicates them. However, there are at least 2 reasons to
configure the connector with EXACTLY_ONCE:

1. There might be other non-Flink topic consumers that would rather not
have duplicated records.
2. Multiple upsert-kafka producers might cause keys to roll back to
previous values. Consider a scenario where 2 producing jobs A and B write
to the same topic with AT_LEAST_ONCE, and a consuming job reads from the
topic. Both producers write unique, monotonically increasing sequences to
the same key. Job A writes x=a1,a2,a3,a4,a5… Job B writes
x=b1,b2,b3,b4,b5, With this setup, we can have the following sequence:

   1. Job A produces x=a5.
   2. Job B produces x=b5.
   3. Job A produces the duplicate write x=5.

The consuming job would observe x going to a5, then to b5, then back a5.
EXACTLY_ONCE would prevent this behavior.

I created https://issues.apache.org/jira/browse/FLINK-31408 and a WIP patch
to add EXACTLY_ONCE to upsert-kafka, but would like to know what the
community thinks about it before moving forward with it.

Thanks,
Alexander

1.
https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/upsert-kafka/#consistency-guarantees
2.
https://github.com/apache/flink-connector-kafka/blob/40cf9994dd847c13602acf1f90895cf9f89b2ce6/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/table/ReducingUpsertSink.java#L31-L37


[jira] [Created] (FLINK-31732) flink-ml-uber module should include statefun as a dependency

2023-04-04 Thread Zhipeng Zhang (Jira)
Zhipeng Zhang created FLINK-31732:
-

 Summary: flink-ml-uber module should include statefun as a 
dependency
 Key: FLINK-31732
 URL: https://issues.apache.org/jira/browse/FLINK-31732
 Project: Flink
  Issue Type: Improvement
Reporter: Zhipeng Zhang






--
This message was sent by Atlassian Jira
(v8.20.10#820010)