[jira] [Created] (FLINK-30246) Registering the same metric object with different name and group will conflict in metric reporter

[Qingsheng Ren (Jira)]

Qingsheng Ren created FLINK-30246:
-

 Summary: Registering the same metric object with different name 
and group will conflict in metric reporter
 Key: FLINK-30246
 URL: https://issues.apache.org/jira/browse/FLINK-30246
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Metrics
Affects Versions: 1.17.0
Reporter: Qingsheng Ren


Metrics are organized by maps in AbstractReporter and MetricQueryService, which 
uses metric object as key and metric name as value. This kind of implementation 
is problematic when the same metric object is registered multiple times, either 
with different names or under different metric groups, which will override each 
other and only one metric is kept finally.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-30245) NPE thrown when filtering decimal(18, 4) values after calling DecimalDataUtils.subtract method

[Wei Zhong (Jira)]

Wei Zhong created FLINK-30245:
-

 Summary: NPE thrown when filtering decimal(18, 4) values after 
calling DecimalDataUtils.subtract method
 Key: FLINK-30245
 URL: https://issues.apache.org/jira/browse/FLINK-30245
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Runtime
Affects Versions: 1.13.6, 1.17.0
Reporter: Wei Zhong
 Attachments: image-2022-11-30-15-11-03-706.png

Reproduce code:
{code:java}
TableEnvironment tableEnv = 
TableEnvironment.create(EnvironmentSettings.newInstance().build());

tableEnv.executeSql("create table datagen_source1 (disburse_amount int) 
with ('connector' = 'datagen')");

tableEnv.executeSql("create table print_sink (disburse_amount 
Decimal(18,4)) with ('connector' = 'print')");

tableEnv.executeSql("create view mid as select cast(disburse_amount as 
Decimal(18,4)) - cast(disburse_amount as Decimal(18,4)) as disburse_amount from 
datagen_source1");

tableEnv.executeSql("insert into print_sink select * from mid where 
disburse_amount > 0 ").await();
{code}
Excpetion:
{code:java}
Exception in thread "main" java.util.concurrent.ExecutionException: 
org.apache.flink.table.api.TableException: Failed to wait job finish
at 
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at 
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
at 
org.apache.flink.table.api.internal.TableResultImpl.awaitInternal(TableResultImpl.java:118)
at 
org.apache.flink.table.api.internal.TableResultImpl.await(TableResultImpl.java:81)
at com.shopee.flink.BugExample2.main(BugExample2.java:21)
Caused by: org.apache.flink.table.api.TableException: Failed to wait job finish
at 
org.apache.flink.table.api.internal.InsertResultProvider.hasNext(InsertResultProvider.java:85)
at 
org.apache.flink.table.api.internal.InsertResultProvider.isFirstRowReady(InsertResultProvider.java:71)
at 
org.apache.flink.table.api.internal.TableResultImpl.lambda$awaitInternal$1(TableResultImpl.java:105)
at 
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: 
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at 
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at 
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
at 
org.apache.flink.table.api.internal.InsertResultProvider.hasNext(InsertResultProvider.java:83)
... 6 more
Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution 
failed.
at 
org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
at 
org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:141)
at 
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
at 
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
at 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at 
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
at 
org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$1(AkkaInvocationHandler.java:267)
at 
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at 
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at 
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
at 
org.apache.flink.util.concurrent.FutureUtils.doForward(FutureUtils.java:1277)
at 
org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$null$1(ClassLoadingUtils.java:93)
at 
org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)
at 
org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$2(ClassLoadingUtils.java:92)
at 
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at 
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at 
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
at 

[jira] [Created] (FLINK-30244) When task using udf/udtf with jni, flink on k8s session the old

[AlexHu (Jira)]

AlexHu created FLINK-30244:
--

 Summary: When task using udf/udtf with jni, flink on k8s session 
the old
 Key: FLINK-30244
 URL: https://issues.apache.org/jira/browse/FLINK-30244
 Project: Flink
  Issue Type: Bug
Reporter: AlexHu






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-30243) Remove duplicate commas

[Shilun Fan (Jira)]

Shilun Fan created FLINK-30243:
--

 Summary: Remove duplicate commas
 Key: FLINK-30243
 URL: https://issues.apache.org/jira/browse/FLINK-30243
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Planner
Affects Versions: 1.16.0
Reporter: Shilun Fan


When reading the code, I found that there are duplicate commas in the comments, 
I submit pr for improvement.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-269: Properly Handling the Processing Timers on Job Termination

[Yun Gao]

Hi Dawid,
Thanks for the comments!
As a whole I'm also open to the API and I also prefer to use simple 
but flexible interfaces, but it still looks there are some problem to 
just let users to implement the termination actions. 
Let's take the WindowOperator as an example. As seen in [1],
in the timer processing logic it needs to acquire the key / namespace
information bound to the timer (which is only supported by the 
InternalTimerService).
Thus if we want users to implement the same logic on termination, we either let 
users 
to trigger the timer handler directly or we also allows users to access these 
piece of 
information. If we go with the later direction, we might need to provide 
interfaces like
interface PendingTimerProcessor {
void onTimer(Timer timer) {
 doHandleTimer(timer);
 }
}
class Timer {
 long getTimestamp();
 KEY getKey();
 NAMESPACE getNamespace();
}
Then we'll have the issue that since we need the interface to handle both of 
cases of 
InternalTimerSerivce and raw ProcessTimeService, the later do not have key and 
namespace information attached, and its also be a bit inconsistency for users 
to have to set 
the KEY and NAMESPACE types.
Besides, it looks to me that if we want to implement behaviors like waiting 
for, it might
be not simply reuse the time handler time, then it requires every operator 
authors to 
re-implement such waiting logics. 
> Moreover it still have the downside that if you call back to the `onTimer` 
> method after 
> `trigger` you have access to the Context which lets you register new timers.
I think we could simply drop the timers registered during we start processing 
the pending timers
on termination. Logically there should be no new data after termination.
> I think I am not convinced to these arguments. First of all I'm afraid there 
> is no clear distinction 
> in that area what is runtime and what is not. I always found 
> `AbstracStreamOperator(*)` actually part 
> of runtime or Flink's internals and thus I don't find `InternalTimerService` 
> a utility, but a vital part 
> of the system. Let's be honest it is impossible to implement an operator 
> without extending from 
> `AbstractStreamOperator*`.What would be the problem with having a proper 
> implementation in 
> `InternalTimerService`? Can't we do it like this?:
I think the original paragraph is only explanation to that the interface is 
harder to support if we 
allows the users to implement the arbitrary logic. But since now we are at the 
page with the callback 
option, users could always be allowed to implement arbitrary logic no matter we 
support timer.trigger()
 or not, thus I think now there is no divergence on this point. I also believe 
in we'll finally have some logic 
similar to the proposed one that drain all the times and process it. 
Best,
Yun Gao
[1] 
https://github.com/apache/flink/blob/a7fdab8b23cddf568fa32ee7eb804d7c3eb23a35/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/windowing/WindowOperator.java#L488
 

--
From:Dawid Wysakowicz 
Send Time:2022 Nov. 28 (Mon.) 23:33
To:dev 
Subject:Re: [DISCUSS] FLIP-269: Properly Handling the Processing Timers on Job 
Termination
Do we really need to have separate methods for triggering/waiting/cancelling. 
To me it sounds rather counterintuitive. Why can't users just execute whatever 
they want in the handler itself instead of additional back and forth with the 
system? Moreover it still have the downside that if you call back to the 
`onTimer` method after `trigger` you have access to the Context which lets you 
register new timers.
I find following approach much simpler:
 void onTimer(...) {
 doHandleTimer(timestamp);
 }
 void processPendingTimer(...) {
 // trigger
 doHandleTimer(timestamp);
 // for cancel, simply do nothing...
 }
Sorry I might not make it very clear here. I think the difficulty with 
supported setting the currentKey is a special issue for the callback options 
(no matter what the interface is) since it allows users to execute logic other 
than the one registered with the timers. The complexity comes from that 
currently we have two level of TimerServices: The ProcessingTimerService (there 
is no key) and InternalTimerService (with key). Currently only 
ProcessingTimerService is exposed to the runtime and InternalTimerService is 
much more a utility to implement the operator. Then with the current code, the 
runtime could only access to ProcessingTimerService on termination. 
 I think I am not convinced to these arguments. First of all I'm afraid there 
is no clear distinction in that area what is runtime and what is not. I always 
found `AbstracStreamOperator(*)` actually part of runtime or Flink's internals 
and thus I don't find 

[jira] [Created] (FLINK-30242) Push localHashAggregate pass the union node

[Aitozi (Jira)]

Aitozi created FLINK-30242:
--

 Summary: Push localHashAggregate pass the union node
 Key: FLINK-30242
 URL: https://issues.apache.org/jira/browse/FLINK-30242
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Planner
Reporter: Aitozi


The local hash aggregate after union will have an extra shuffle stage. We can 
swap it with the union node so the local hash aggregate can chain with the 
mapper stage saving the unnecessary shuffle, especially in the batch job.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] FLIP-271: Autoscaling

[Mason Chen]

+1 (non-binding)

On Tue, Nov 29, 2022 at 11:55 AM Ferenc Csaky 
wrote:

> +1 (non-binding)
>
>
>
>
> --- Original Message ---
> On Tuesday, November 29th, 2022 at 15:39, Márton Balassi <
> balassi.mar...@gmail.com> wrote:
>
>
> >
> >
> > +1 (binding)
> >
> > On Tue, Nov 29, 2022 at 6:13 AM Chenya Zhang chenyazhangche...@gmail.com
> >
> > wrote:
> >
> > > +1 (non-binding)
> > >
> > > On Sun, Nov 27, 2022 at 5:49 PM Jiangang Liu liujiangangp...@gmail.com
> > > wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > Best,
> > > > Jiangang Liu
> > > >
> > > > Thomas Weise t...@apache.org 于2022年11月28日周一 06:23写道：
> > > >
> > > > > +1 (binding)
> > > > >
> > > > > On Sat, Nov 26, 2022 at 8:11 AM Zheng Yu Chen jam.gz...@gmail.com
> > > > > wrote:
> > > > >
> > > > > > +1(no-binding)
> > > > > >
> > > > > > Maximilian Michels m...@apache.org 于 2022年11月24日周四 上午12:25写道：
> > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > I'd like to start a vote for FLIP-271 [1] which we previously
> > > > > > > discussed
> > > > > > > on
> > > > > > > the dev mailing list [2].
> > > > > > >
> > > > > > > I'm planning to keep the vote open for at least until Tuesday,
> Nov
> > > > > > > 29.
> > > > > > >
> > > > > > > -Max
> > > > > > >
> > > > > > > [1]
> > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-271%3A+Autoscaling
> > >
> > > > > > > [2]
> > > > > > >
> https://lists.apache.org/thread/pvfb3fw99mj8r1x8zzyxgvk4dcppwssz
>

[jira] [Created] (FLINK-30241) Flink ML Iteration ConcurrentModificationException

[Yunfeng Zhou (Jira)]

Yunfeng Zhou created FLINK-30241:


 Summary: Flink ML Iteration ConcurrentModificationException
 Key: FLINK-30241
 URL: https://issues.apache.org/jira/browse/FLINK-30241
 Project: Flink
  Issue Type: Bug
  Components: Library / Machine Learning
Affects Versions: ml-2.1.0
Reporter: Yunfeng Zhou


https://github.com/jiangxin369/flink-ml/actions/runs/3577811156/jobs/6017233847


{code}
___ LinearRegressionTest.test_get_model_data ___

self = 

def test_get_model_data(self):
regression = LinearRegression().set_weight_col('weight')
model = regression.fit(self.input_data_table)
model_data = self.t_env.to_data_stream(
>   model.get_model_data()[0]).execute_and_collect().next()

pyflink/ml/lib/regression/tests/test_linearregression.py:124: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/site-packages/pyflink/datastream/data_stream.py:1760:
 in next
if not self._j_closeable_iterator.hasNext():
/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/site-packages/py4j/java_gateway.py:1322:
 in __call__
answer, self.gateway_client, self.target_id, self.name)
/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/site-packages/pyflink/util/exceptions.py:146:
 in deco
return f(*a, **kw)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

answer = 'xro12236'
gateway_client = 
target_id = 'o12139', name = 'hasNext'

def get_return_value(answer, gateway_client, target_id=None, name=None):
"""Converts an answer received from the Java gateway into a Python 
object.

For example, string representation of integers are converted to Python
integer, string representation of objects are converted to JavaObject
instances, etc.

:param answer: the string returned by the Java gateway
:param gateway_client: the gateway client used to communicate with the 
Java
Gateway. Only necessary if the answer is a reference (e.g., object,
list, map)
:param target_id: the name of the object from which the answer comes 
from
(e.g., *object1* in `object1.hello()`). Optional.
:param name: the name of the member from which the answer comes from
(e.g., *hello* in `object1.hello()`). Optional.
"""
if is_error(answer)[0]:
if len(answer) > 1:
type = answer[1]
value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
if answer[1] == REFERENCE_TYPE:
raise Py4JJavaError(
"An error occurred while calling {0}{1}{2}.\n".
>   format(target_id, ".", name), value)
E   py4j.protocol.Py4JJavaError: An error occurred while 
calling o12139.hasNext.
E   : java.lang.RuntimeException: Failed to fetch next result
E   at 
org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:109)
E   at 
org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
E   at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown 
Source)
E   at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
E   at java.lang.reflect.Method.invoke(Method.java:498)
E   at 
org.apache.flink.api.python.shaded.py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
E   at 
org.apache.flink.api.python.shaded.py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
E   at 
org.apache.flink.api.python.shaded.py4j.Gateway.invoke(Gateway.java:282)
E   at 
org.apache.flink.api.python.shaded.py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
E   at 
org.apache.flink.api.python.shaded.py4j.commands.CallCommand.execute(CallCommand.java:79)
E   at 
org.apache.flink.api.python.shaded.py4j.GatewayConnection.run(GatewayConnection.java:238)
E   at java.lang.Thread.run(Thread.java:750)
E   Caused by: java.io.IOException: Failed to fetch job 
execution result
E   at 
org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.getAccumulatorResults(CollectResultFetcher.java:184)
E   at 
org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:121)
E   at 
org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:106)
E 

Re: [VOTE] Release flink-connector-dynamodb v3.0.0, release candidate #0

[Martijn Visser]

Hi Danny,

+1 (binding)

- Validated hashes
- Verified signature
- Verified that no binaries exist in the source archive
- Build the source with Maven
- Verified licenses
- Verified web PR

Best regards,

Martijn

On Tue, Nov 29, 2022 at 8:37 PM Martijn Visser  wrote:

> Hi Danny,
>
> +1 (binding)
>
> - Validated hashes
> - Verified signature
> - Verified that no binaries exist in the source archive
> - Build the source with Maven
> - Verified licenses
> - Verified web PR
>
> Best regards,
>
> Martijn
>
> On Tue, Nov 29, 2022 at 12:57 PM Hamdy, Ahmed 
> wrote:
>
>> +1 (non-binding)
>>
>> On 29/11/2022, 08:27, "Teoh, Hong"  wrote:
>>
>> CAUTION: This email originated from outside of the organization. Do
>> not click links or open attachments unless you can confirm the sender and
>> know the content is safe.
>>
>>
>>
>> +1 (non-binding)
>>
>> * Hashes and Signatures look good
>> * All required files on dist.apache.org
>> * Tag is present in Github
>> * Verified source archive does not contain any binary files
>> * Source archive builds using maven
>> * Started packaged example SQL job using SQL client. Verified that it
>> writes successfully to the sink table.
>> * Verified sink metrics look ok.
>>
>>
>> Cheers,
>> Hong
>>
>> On 28/11/2022, 16:44, "Danny Cranmer" 
>> wrote:
>>
>> CAUTION: This email originated from outside of the organization.
>> Do not click links or open attachments unless you can confirm the sender
>> and know the content is safe.
>>
>>
>>
>> Hi everyone,
>> Please review and vote on the release candidate #0 for the
>> version 3.0.0 as
>> follows:
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific
>> comments)
>>
>>
>> The complete staging area is available for your review, which
>> includes:
>> * JIRA release notes [1],
>> * the official Apache source release to be deployed to
>> dist.apache.org [2],
>> which are signed with the key with fingerprint 125FD8DB [3],
>> * all artifacts to be deployed to the Maven Central Repository
>> [4],
>> * source code tag v3.0.0-rc0 [5],
>> * website pull request listing the new release [6].
>>
>> The vote will be open for at least 72 hours (Thursday 1st
>> December 17:00
>> UTC). It is adopted by majority approval, with at least 3 PMC
>> affirmative
>> votes.
>>
>> Please note, this is a new connector and the first release.
>>
>> Thanks,
>> Danny
>>
>> [1]
>>
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352277
>> [2]
>>
>> https://dist.apache.org/repos/dist/dev/flink/flink-connector-aws-3.0.0-rc0
>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
>> [4]
>> https://repository.apache.org/content/repositories/orgapacheflink-1552/
>> [5]
>> https://github.com/apache/flink-connector-aws/releases/tag/v3.0.0-rc0
>> [6] https://github.com/apache/flink-web/pull/588
>>
>>
>>

Re: [VOTE] FLIP-271: Autoscaling

[Ferenc Csaky]

+1 (non-binding)




--- Original Message ---
On Tuesday, November 29th, 2022 at 15:39, Márton Balassi 
 wrote:


> 
> 
> +1 (binding)
> 
> On Tue, Nov 29, 2022 at 6:13 AM Chenya Zhang chenyazhangche...@gmail.com
> 
> wrote:
> 
> > +1 (non-binding)
> > 
> > On Sun, Nov 27, 2022 at 5:49 PM Jiangang Liu liujiangangp...@gmail.com
> > wrote:
> > 
> > > +1 (non-binding)
> > > 
> > > Best,
> > > Jiangang Liu
> > > 
> > > Thomas Weise t...@apache.org 于2022年11月28日周一 06:23写道：
> > > 
> > > > +1 (binding)
> > > > 
> > > > On Sat, Nov 26, 2022 at 8:11 AM Zheng Yu Chen jam.gz...@gmail.com
> > > > wrote:
> > > > 
> > > > > +1(no-binding)
> > > > > 
> > > > > Maximilian Michels m...@apache.org 于 2022年11月24日周四 上午12:25写道：
> > > > > 
> > > > > > Hi everyone,
> > > > > > 
> > > > > > I'd like to start a vote for FLIP-271 [1] which we previously
> > > > > > discussed
> > > > > > on
> > > > > > the dev mailing list [2].
> > > > > > 
> > > > > > I'm planning to keep the vote open for at least until Tuesday, Nov
> > > > > > 29.
> > > > > > 
> > > > > > -Max
> > > > > > 
> > > > > > [1]
> > 
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-271%3A+Autoscaling
> > 
> > > > > > [2]
> > > > > > https://lists.apache.org/thread/pvfb3fw99mj8r1x8zzyxgvk4dcppwssz

Re: [VOTE] Release flink-connector-cassandra, release candidate #1

[Ryan Skraba]

Hello!  +1 (non-binding)

1.  Verified signature and SHA512 checksum in svn
2.  Verified that the tag exists on the source git repository and is
(mostly) identical to the source tar release
  - Some .-prefixed files like .idea/  and .gitignore in the repo under the
tag that are not in the tar (but .editorconfig is).
3. Verified that the original source in
apache/flink/flink-connectors/flink-connector-cassandra branch is (mostly)
identical to the externalized repository
  - A minor change to an ITCase constant to avoid a dependency, no non-test
changes.
4. Built the source with Maven
5. Verified staged artifacts and their SHA1 checksum (using
--strict-checksums)

All my best and thanks for this work!  Ryan


On Tue, Nov 29, 2022 at 3:44 PM Chesnay Schepler  wrote:

> The usage of the artifact shortcode was somewhat intentional; the
> connector_artifact shortcode does not support scala suffixes, which the
> cassandra connector unfortunately needs
>
> On 29/11/2022 15:16, Danny Cranmer wrote:
> > +1 (binding)
> >
> > - Verified signature/hashes
> > - Build the source with Maven (tests pass)
> > - Verified NOTICE files
> > - Verified that no binaries exist in the source archive
> > - Verified artifacts in repository.apache.org are as expected
> > - Verified tag exists on github
> > - Approved web PR
> >
> > I noticed the incorrect short code has been used in the docs [1], however
> > this can be addressed as a follow-up.
> >
> > Thanks,
> >
> > [1]
> >
> https://github.com/apache/flink-connector-cassandra/blob/main/docs/content/docs/connectors/datastream/cassandra.md
> >
> > On Tue, Nov 29, 2022 at 1:48 PM Martijn Visser  >
> > wrote:
> >
> >> +1 (binding)
> >>
> >> - Validated hashes
> >> - Verified signature
> >> - Verified that no binaries exist in the source archive
> >> - Build the source with Maven
> >> - Verified licenses
> >> - Verified web PR
> >>
> >> On Tue, Nov 29, 2022 at 11:50 AM Chesnay Schepler 
> >> wrote:
> >>
> >>>   > the link to a tag in the initial message is wrong
> >>>
> >>> Whoops, small bug in the message generator.
> >>>
> >>>   > Did we ever consider having the `flink-version` always as part of
> the
> >>> connector version?
> >>>
> >>> Not specifically, no.
> >>> I think it makes sense though that if you take the "3.0.0" source that
> >>> you get "3.0.0" artifacts (and making such a suffix work for SNAPSHOT
> >>> artifacts is a bit tricky because it's suddenly an infix).
> >>> In general the current setup is less complex.
> >>> I can see upsides though, like not having to modify the poms when
> >>> staging the jars.
> >>> But medium-term we want to get rid of these suffixes anyway.
> >>>
> >>> On 29/11/2022 10:36, Dawid Wysakowicz wrote:
>  +1 (binding)
> 
>  - Downloaded artifacts
>  - Checked hash and signature
>  - No binaries in source archive found
>  - Verified NOTICE files
>  - Built from source code
>  - Verified that no SNAPSHOT versions exist, all versions point to
> 3.0.0
> >>> in
>  POM files
>  - Tag is OK
>  - Reviewed the Web PR
> 
>  Few notes:
> 
>  - the link to a tag in the initial message is wrong. It should've been
> 
> >>
> https://github.com/apache/flink-connector-cassandra/releases/tag/v3.0.0-rc1
>  - I was a bit surprised that executing `mvn -DskipTests package`
>  produces artifacts without the Flink version suffix. Did we ever
>  consider having the `flink-version` always as part of the connector
>  version?
> 
>  Best,
> 
>  Dawid
> 
>  On 25/11/2022 10:31, Chesnay Schepler wrote:
> > Hi everyone,
> > Please review and vote on the release candidate #1 for the version
> > 3.0.0, as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> >
> > The complete staging area is available for your review, which
> >> includes:
> > * JIRA release notes [1],
> > * the official Apache source release to be deployed to
> > dist.apache.org [2], which are signed with the key with fingerprint
> > C2EED7B111D464BA [3],
> > * all artifacts to be deployed to the Maven Central Repository [4],
> > * source code tag v3.0.0-1 [5],
> > * website pull request listing the new release [6].
> >
> > The vote will be open for at least 72 hours. It is adopted by
> > majority approval, with at least 3 PMC affirmative votes.
> >
> > This is the first externalized released of the Cassandra connector
> > and functionally identical to 1.16.0.
> >
> >
> > Thanks,
> > Chesnay
> >
> > [1] https://issues.apache.org/jira/projects/FLINK/versions/12352593
> > [2]
> >
> >>
> https://dist.apache.org/repos/dist/dev/flink/flink-connector-cassandra-3.0.0-rc1
> > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > [4]
> >
> >> https://repository.apache.org/content/repositories/orgapacheflink-1549/

[jira] [Created] (FLINK-30240) Fix Doc error pointing to TableKind#MANAGED

[Samrat Deb (Jira)]

Samrat Deb created FLINK-30240:
--

 Summary: Fix Doc error pointing to TableKind#MANAGED 
 Key: FLINK-30240
 URL: https://issues.apache.org/jira/browse/FLINK-30240
 Project: Flink
  Issue Type: Bug
Reporter: Samrat Deb


TableKind#MANAGED is referenced in 
[Catalog.java|https://github.com/apache/flink/blob/master/flink-table/flink-table-common/src/main/java/org/apache/flink/table/catalog/Catalog.java#L320].
 
but TableKind is an enum consisting of only 2 types 
`TABLE` and `VIEW`  
[here|https://github.com/apache/flink/blob/master/flink-table/flink-table-common/src/main/java/org/apache/flink/table/catalog/CatalogBaseTable.java#L40].
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-30239) The flame graph doesn't work due to groupExecutionsByLocation has bug

[Rui Fan (Jira)]

Rui Fan created FLINK-30239:
---

 Summary: The flame graph doesn't work due to 
groupExecutionsByLocation has bug
 Key: FLINK-30239
 URL: https://issues.apache.org/jira/browse/FLINK-30239
 Project: Flink
  Issue Type: Bug
  Components: Runtime / REST, Runtime / Web Frontend
Affects Versions: 1.16.0, 1.17.0
Reporter: Rui Fan
 Fix For: 1.17.0, 1.16.1
 Attachments: image-2022-11-30-00-10-48-940.png, 
image-2022-11-30-00-11-09-728.png, image-2022-11-30-00-14-11-355.png

The flame graph cannot be generated forever when multiple tasks in the same TM. 
It's caused by FLINK-26074

 
h1. Root cause:

A Set cannot be converted to an ImmutableSet during the aggregation of 
ExecutionAttemptIDs. It will cause only the first ExecutionAttemptID of the TM 
to be added to the set, the second ExecutionAttemptID will fail.

 

!image-2022-11-30-00-14-11-355.png!

 

 

 

!image-2022-11-30-00-11-09-728.png!

 

Exception Info: 

!image-2022-11-30-00-10-48-940.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Release flink-connector-cassandra, release candidate #1

[Chesnay Schepler]

The usage of the artifact shortcode was somewhat intentional; the 
connector_artifact shortcode does not support scala suffixes, which the 
cassandra connector unfortunately needs


On 29/11/2022 15:16, Danny Cranmer wrote:

+1 (binding)

- Verified signature/hashes
- Build the source with Maven (tests pass)
- Verified NOTICE files
- Verified that no binaries exist in the source archive
- Verified artifacts in repository.apache.org are as expected
- Verified tag exists on github
- Approved web PR

I noticed the incorrect short code has been used in the docs [1], however
this can be addressed as a follow-up.

Thanks,

[1]
https://github.com/apache/flink-connector-cassandra/blob/main/docs/content/docs/connectors/datastream/cassandra.md

On Tue, Nov 29, 2022 at 1:48 PM Martijn Visser 
wrote:


+1 (binding)

- Validated hashes
- Verified signature
- Verified that no binaries exist in the source archive
- Build the source with Maven
- Verified licenses
- Verified web PR

On Tue, Nov 29, 2022 at 11:50 AM Chesnay Schepler 
wrote:


  > the link to a tag in the initial message is wrong

Whoops, small bug in the message generator.

  > Did we ever consider having the `flink-version` always as part of the
connector version?

Not specifically, no.
I think it makes sense though that if you take the "3.0.0" source that
you get "3.0.0" artifacts (and making such a suffix work for SNAPSHOT
artifacts is a bit tricky because it's suddenly an infix).
In general the current setup is less complex.
I can see upsides though, like not having to modify the poms when
staging the jars.
But medium-term we want to get rid of these suffixes anyway.

On 29/11/2022 10:36, Dawid Wysakowicz wrote:

+1 (binding)

- Downloaded artifacts
- Checked hash and signature
- No binaries in source archive found
- Verified NOTICE files
- Built from source code
- Verified that no SNAPSHOT versions exist, all versions point to 3.0.0

in

POM files
- Tag is OK
- Reviewed the Web PR

Few notes:

- the link to a tag in the initial message is wrong. It should've been


https://github.com/apache/flink-connector-cassandra/releases/tag/v3.0.0-rc1

- I was a bit surprised that executing `mvn -DskipTests package`
produces artifacts without the Flink version suffix. Did we ever
consider having the `flink-version` always as part of the connector
version?

Best,

Dawid

On 25/11/2022 10:31, Chesnay Schepler wrote:

Hi everyone,
Please review and vote on the release candidate #1 for the version
3.0.0, as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)


The complete staging area is available for your review, which

includes:

* JIRA release notes [1],
* the official Apache source release to be deployed to
dist.apache.org [2], which are signed with the key with fingerprint
C2EED7B111D464BA [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag v3.0.0-1 [5],
* website pull request listing the new release [6].

The vote will be open for at least 72 hours. It is adopted by
majority approval, with at least 3 PMC affirmative votes.

This is the first externalized released of the Cassandra connector
and functionally identical to 1.16.0.


Thanks,
Chesnay

[1] https://issues.apache.org/jira/projects/FLINK/versions/12352593
[2]


https://dist.apache.org/repos/dist/dev/flink/flink-connector-cassandra-3.0.0-rc1

[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4]


https://repository.apache.org/content/repositories/orgapacheflink-1549/

[5]


https://github.com/apache/flink-connector-cassandra/releases/tag/v3.0.0-1

[6] https://github.com/apache/flink-web/pull/587

Re: [VOTE] FLIP-271: Autoscaling

[Márton Balassi]

+1 (binding)

On Tue, Nov 29, 2022 at 6:13 AM Chenya Zhang 
wrote:

> +1 (non-binding)
>
> On Sun, Nov 27, 2022 at 5:49 PM Jiangang Liu 
> wrote:
>
> > +1 (non-binding)
> >
> > Best,
> > Jiangang Liu
> >
> > Thomas Weise  于2022年11月28日周一 06:23写道：
> >
> > > +1 (binding)
> > >
> > >
> > > On Sat, Nov 26, 2022 at 8:11 AM Zheng Yu Chen 
> > wrote:
> > >
> > > > +1(no-binding)
> > > >
> > > > Maximilian Michels  于 2022年11月24日周四 上午12:25写道：
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > I'd like to start a vote for FLIP-271 [1] which we previously
> > discussed
> > > > on
> > > > > the dev mailing list [2].
> > > > >
> > > > > I'm planning to keep the vote open for at least until Tuesday, Nov
> > 29.
> > > > >
> > > > > -Max
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-271%3A+Autoscaling
> > > > > [2]
> https://lists.apache.org/thread/pvfb3fw99mj8r1x8zzyxgvk4dcppwssz
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] OLM Bundles for Flink Kubernetes Operator

[Őrhidi Mátyás]

Hi folks,

I'm going to shepherd the upcoming 1.3.0 release. I'll also try to find
some time to review the OLM integration proposal, and see how much extra
effort it would be.

Best,
Matyas

On Wed, Nov 23, 2022 at 5:45 PM Yang Wang  wrote:

> Improving the visibility of Flink Kubernetes Operator is great. And I agree
> OLM could help with this.
>
> I just hope this will not make the whole release process too complicated.
> Of course, if we want to integrate the OLM into the official release, it
> should be tested by the users easily.
>
> Best,
> Yang
>
> Gyula Fóra  于2022年11月24日周四 00:29写道：
>
> > Ted, Jim:
> >
> > When we create the RC bundle (jars, sources, helm chart) we execute the
> > following steps:
> >  1. Push the RC tag to git -> this will generate and publish an image
> with
> > the RC git commit tag to ghcr.io
> >  2. We bake into the helm chart the RC tag as the image tag
> >  3. We create the source and helm bundle, then publish it
> >
> > In step 3 we also need to add the OLM bundle creation and we can bake in
> > the same ghcr.io image tag.
> >
> > Gyula
> >
> > On Wed, Nov 23, 2022 at 7:13 AM Jim Busche  wrote:
> >
> > > I'm curious how the RC automation works now - is it fully automatic?
> For
> > > example, a RC Debian image gets created, something like:
> > > ghcr.io/apache/flink-kubernetes-operator:95128bf<
> > > http://ghcr.io/apache/flink-kubernetes-operator:95128bf> and is pushed
> > to
> > > ghcr.io … then that's used in the rc helm chart?
> > >
> > > If that's all automated, then that rc candidate operator image value
> must
> > > be stored as a variable, and could be utilized to build the OLM bundle
> as
> > > well with the same rc operator image.  Then the bundle and catalog
> could
> > be
> > > pushed to ghcr.io for testing.
> > >
> > >
> > >
> > > If it's not automated, then in the manual steps, there could a few
> steps
> > > added to set the rc operator image value prior to running the bundle
> > > creation, then manually pushing the bundle and catalog to ghcr.io for
> > > testing.
> > >
> > >
> > > Thanks, Jim
> > > --
> > > James Busche | Sr. Software Engineer, Watson AI and Data Open
> Technology
> > |
> > > 408-460-0737 | jbus...@us.ibm.com
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > From: Hao t Chang 
> > > Date: Tuesday, November 22, 2022 at 2:55 PM
> > > To: dev@flink.apache.org 
> > > Subject: [EXTERNAL] [DISCUSS] OLM Bundles for Flink Kubernetes Operator
> > > Hi Gyula,
> > >
> > > Agree, we should include the bundle file and let community inspect them
> > in
> > > the staging repo. In addition, people can do a few things to test the
> > > bundle files.
> > > 1.Run CI test suits (
> > > https://github.com/tedhtchang/olm#run-ci-test-suits-before-creating-pr
> )
> > > with the bundle files directly.
> > > 2.Deploy operator with OLM (requires the bundle image in a
> > > registry)
> > > 3.Test operators upgrade from the previous version with
> > > OLM(requires both bundle and catalog image in a registry)
> > >
> > > For 2 and 3, it’s better to build a bundle and catalog images as part
> of
> > > the staging. For example, during the next release(1.3.0-rc1),
> temporally
> > > push the 2 images to
> > > ghcr.io/apache/flink-kubernetes-operator-bundle:1.3.0-rc1 and
> > > ghcr.io/apache/flink-kubernetes-opeator-catalog:1.3.0-rc1. Then,
> > > community can test 2. and 3. easily with the following commands:
> > > # Deploy the catalog src in default ns
> > > cat < > > apiVersion: operators.coreos.com/v1alpha1
> > > kind: CatalogSource
> > > metadata:
> > >   name: olm-flink-operator-catalog
> > >   namespace: default
> > > spec:
> > >   sourceType: grpc
> > >   image: ghcr.io/apache/flink-kubernetes-opeator-catalog:1.3.0-rc1
> > > EOF
> > >
> > > # Deploy operator from the catalog
> > > Cat < > > apiVersion: operators.coreos.com/v1alpha2
> > > kind: OperatorGroup
> > > metadata:
> > >   name: default-og
> > >   namespace: default
> > > spec:
> > >   targetNamespaces:
> > >   - default
> > > ---
> > > apiVersion: operators.coreos.com/v1alpha1
> > > kind: Subscription
> > > metadata:
> > >   name: flink-kubernetes-operator
> > >   namespace: default
> > > spec:
> > >   channel: alpha
> > >   name: flink-kubernetes-operator
> > >   source: olm-flink-operator-catalog
> > >   sourceNamespace: default
> > >   # For testing upgrade from previous version
> > >   # installPlanApproval: Automatic # Manual
> > >   # startingCSV: flink-kubernetes-operator.v1.2.0
> > > EOF
> > >
> >
>

Re: Kubernetes Operator 1.3.0 release

[Márton Balassi]

Hi all,

Thanks for volunteering, Matyas and +1 for the 1.3 operator release.
The modified timeline suggested by Gyula sounds good to me.

On Tue, Nov 29, 2022 at 7:04 AM Őrhidi Mátyás 
wrote:

> Thanks Gyula! Sounds good.
>
> On Mon, Nov 28, 2022 at 9:59 PM Gyula Fóra  wrote:
>
> > Hi Matyas!
> >
> > +1 for the 1.3.0 release.
> >
> > Thank you for the proposal and for volunteering as a release manager.
> >
> > Given that it's already Nov 29 here, I suggest moving the feature freeze
> > to the end of this week (Dec 2) and release cut to mid next week (Dec 7)
> to
> > give us some flexibility in merging the currently pending PRs.
> >
> > Thanks,
> > Gyula
> >
> > On Tue, Nov 29, 2022 at 6:44 AM Őrhidi Mátyás 
> > wrote:
> >
> >> Hi Devs,
> >>
> >> Our planned milestones for the upcoming operator 1.3.0 release are the
> >> following:
> >> Feature Freeze:   Nov 28
> >> Release Cut:Dec 5
> >> Release Date:  Dec 12
> >>
> >> There are a few JIRAs that still need to be addressed, but most of them
> >> are work in progress, so it seems we can most probably meet these
> deadlines.
> >>
> >> I volunteer as the release manager.
> >>
> >> Cheers,
> >> Matyas
> >>
> >
>

Re: [VOTE] Release flink-connector-cassandra, release candidate #1

[Danny Cranmer]

+1 (binding)

- Verified signature/hashes
- Build the source with Maven (tests pass)
- Verified NOTICE files
- Verified that no binaries exist in the source archive
- Verified artifacts in repository.apache.org are as expected
- Verified tag exists on github
- Approved web PR

I noticed the incorrect short code has been used in the docs [1], however
this can be addressed as a follow-up.

Thanks,

[1]
https://github.com/apache/flink-connector-cassandra/blob/main/docs/content/docs/connectors/datastream/cassandra.md

On Tue, Nov 29, 2022 at 1:48 PM Martijn Visser 
wrote:

> +1 (binding)
>
> - Validated hashes
> - Verified signature
> - Verified that no binaries exist in the source archive
> - Build the source with Maven
> - Verified licenses
> - Verified web PR
>
> On Tue, Nov 29, 2022 at 11:50 AM Chesnay Schepler 
> wrote:
>
> >  > the link to a tag in the initial message is wrong
> >
> > Whoops, small bug in the message generator.
> >
> >  > Did we ever consider having the `flink-version` always as part of the
> > connector version?
> >
> > Not specifically, no.
> > I think it makes sense though that if you take the "3.0.0" source that
> > you get "3.0.0" artifacts (and making such a suffix work for SNAPSHOT
> > artifacts is a bit tricky because it's suddenly an infix).
> > In general the current setup is less complex.
> > I can see upsides though, like not having to modify the poms when
> > staging the jars.
> > But medium-term we want to get rid of these suffixes anyway.
> >
> > On 29/11/2022 10:36, Dawid Wysakowicz wrote:
> > >
> > > +1 (binding)
> > >
> > > - Downloaded artifacts
> > > - Checked hash and signature
> > > - No binaries in source archive found
> > > - Verified NOTICE files
> > > - Built from source code
> > > - Verified that no SNAPSHOT versions exist, all versions point to 3.0.0
> > in
> > > POM files
> > > - Tag is OK
> > > - Reviewed the Web PR
> > >
> > > Few notes:
> > >
> > > - the link to a tag in the initial message is wrong. It should've been
> > >
> >
> https://github.com/apache/flink-connector-cassandra/releases/tag/v3.0.0-rc1
> > > - I was a bit surprised that executing `mvn -DskipTests package`
> > > produces artifacts without the Flink version suffix. Did we ever
> > > consider having the `flink-version` always as part of the connector
> > > version?
> > >
> > > Best,
> > >
> > > Dawid
> > >
> > > On 25/11/2022 10:31, Chesnay Schepler wrote:
> > >> Hi everyone,
> > >> Please review and vote on the release candidate #1 for the version
> > >> 3.0.0, as follows:
> > >> [ ] +1, Approve the release
> > >> [ ] -1, Do not approve the release (please provide specific comments)
> > >>
> > >>
> > >> The complete staging area is available for your review, which
> includes:
> > >> * JIRA release notes [1],
> > >> * the official Apache source release to be deployed to
> > >> dist.apache.org [2], which are signed with the key with fingerprint
> > >> C2EED7B111D464BA [3],
> > >> * all artifacts to be deployed to the Maven Central Repository [4],
> > >> * source code tag v3.0.0-1 [5],
> > >> * website pull request listing the new release [6].
> > >>
> > >> The vote will be open for at least 72 hours. It is adopted by
> > >> majority approval, with at least 3 PMC affirmative votes.
> > >>
> > >> This is the first externalized released of the Cassandra connector
> > >> and functionally identical to 1.16.0.
> > >>
> > >>
> > >> Thanks,
> > >> Chesnay
> > >>
> > >> [1] https://issues.apache.org/jira/projects/FLINK/versions/12352593
> > >> [2]
> > >>
> >
> https://dist.apache.org/repos/dist/dev/flink/flink-connector-cassandra-3.0.0-rc1
> > >> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > >> [4]
> > >>
> https://repository.apache.org/content/repositories/orgapacheflink-1549/
> > >> [5]
> > >>
> >
> https://github.com/apache/flink-connector-cassandra/releases/tag/v3.0.0-1
> > >> [6] https://github.com/apache/flink-web/pull/587
> > >>
> >
>

Re: [VOTE] Release flink-connector-cassandra, release candidate #1

[Martijn Visser]

+1 (binding)

- Validated hashes
- Verified signature
- Verified that no binaries exist in the source archive
- Build the source with Maven
- Verified licenses
- Verified web PR

On Tue, Nov 29, 2022 at 11:50 AM Chesnay Schepler 
wrote:

>  > the link to a tag in the initial message is wrong
>
> Whoops, small bug in the message generator.
>
>  > Did we ever consider having the `flink-version` always as part of the
> connector version?
>
> Not specifically, no.
> I think it makes sense though that if you take the "3.0.0" source that
> you get "3.0.0" artifacts (and making such a suffix work for SNAPSHOT
> artifacts is a bit tricky because it's suddenly an infix).
> In general the current setup is less complex.
> I can see upsides though, like not having to modify the poms when
> staging the jars.
> But medium-term we want to get rid of these suffixes anyway.
>
> On 29/11/2022 10:36, Dawid Wysakowicz wrote:
> >
> > +1 (binding)
> >
> > - Downloaded artifacts
> > - Checked hash and signature
> > - No binaries in source archive found
> > - Verified NOTICE files
> > - Built from source code
> > - Verified that no SNAPSHOT versions exist, all versions point to 3.0.0
> in
> > POM files
> > - Tag is OK
> > - Reviewed the Web PR
> >
> > Few notes:
> >
> > - the link to a tag in the initial message is wrong. It should've been
> >
> https://github.com/apache/flink-connector-cassandra/releases/tag/v3.0.0-rc1
> > - I was a bit surprised that executing `mvn -DskipTests package`
> > produces artifacts without the Flink version suffix. Did we ever
> > consider having the `flink-version` always as part of the connector
> > version?
> >
> > Best,
> >
> > Dawid
> >
> > On 25/11/2022 10:31, Chesnay Schepler wrote:
> >> Hi everyone,
> >> Please review and vote on the release candidate #1 for the version
> >> 3.0.0, as follows:
> >> [ ] +1, Approve the release
> >> [ ] -1, Do not approve the release (please provide specific comments)
> >>
> >>
> >> The complete staging area is available for your review, which includes:
> >> * JIRA release notes [1],
> >> * the official Apache source release to be deployed to
> >> dist.apache.org [2], which are signed with the key with fingerprint
> >> C2EED7B111D464BA [3],
> >> * all artifacts to be deployed to the Maven Central Repository [4],
> >> * source code tag v3.0.0-1 [5],
> >> * website pull request listing the new release [6].
> >>
> >> The vote will be open for at least 72 hours. It is adopted by
> >> majority approval, with at least 3 PMC affirmative votes.
> >>
> >> This is the first externalized released of the Cassandra connector
> >> and functionally identical to 1.16.0.
> >>
> >>
> >> Thanks,
> >> Chesnay
> >>
> >> [1] https://issues.apache.org/jira/projects/FLINK/versions/12352593
> >> [2]
> >>
> https://dist.apache.org/repos/dist/dev/flink/flink-connector-cassandra-3.0.0-rc1
> >> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> >> [4]
> >> https://repository.apache.org/content/repositories/orgapacheflink-1549/
> >> [5]
> >>
> https://github.com/apache/flink-connector-cassandra/releases/tag/v3.0.0-1
> >> [6] https://github.com/apache/flink-web/pull/587
> >>
>

Re: [jira] [Created] (FLINK-30238) Unified Sink committer does not clean up state on final savepoint

[Fabian Paul]

Hi folks,

I did some initial investigation, and the problem seems twofold.

If no post-commit topology is used, we do not run into a problem where
we could lose data but since we do not clean up the state correctly,
we will hit this [1] when trying to stop the pipeline with a savepoint
after we have started it from a savepoint.
AFAICT all two-phase commit sinks are affected Kafka, File etc.

For sinks using the post-commit topology, the same applies.
Additionally, we might never do the commit from the post-commit
topology resulting in lost data.

Best,
Fabian

[1] 
https://github.com/apache/flink/blob/ed46cb2fd64f1cb306ae5b7654d2b4d64ab69f22/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittableManagerImpl.java#L83

Re: [jira] [Created] (FLINK-30238) Unified Sink committer does not clean up state on final savepoint

[Martijn Visser]

Let's loop in Fabian to clarify. I'm not sure if this only occurs when
using a post-commit topology (like compaction), but he can definitely
clarify :)

On Tue, Nov 29, 2022 at 2:19 PM Galen Warren 
wrote:

> This seems scary -- am I interpreting it correctly to mean that unified
> FileSink doesn't work properly with jobs that need to be
> stopped-with-savepoints and restarted?
>
> Should one use the deprecated StreamingFileSink until this is resolved?
>
> On Tue, Nov 29, 2022 at 6:02 AM Fabian Paul (Jira) 
> wrote:
>
> > Fabian Paul created FLINK-30238:
> > ---
> >
> >  Summary: Unified Sink committer does not clean up state on
> > final savepoint
> >  Key: FLINK-30238
> >  URL: https://issues.apache.org/jira/browse/FLINK-30238
> >  Project: Flink
> >   Issue Type: Bug
> >   Components: Connectors / Common
> > Affects Versions: 1.15.3, 1.17.0, 1.16.1
> > Reporter: Fabian Paul
> >
> >
> > During stop-with-savepoint the committer only commits the pending
> > committables on notifyCheckpointComplete.
> >
> > This has several downsides.
> >  * Last committableSummary has checkpoint id LONG.MAX and is never
> cleared
> > from the state leading to that stop-with-savepoint does not work when the
> > pipeline recovers from a savepoint
> >  * While the committables are committed during stop-with-savepoint they
> > are not forwarded to post-commit topology, potentially losing data and
> > preventing to close open transactions.
> >
> >
> >
> > --
> > This message was sent by Atlassian Jira
> > (v8.20.10#820010)
> >
>

Re: [jira] [Created] (FLINK-30238) Unified Sink committer does not clean up state on final savepoint

[Galen Warren]

This seems scary -- am I interpreting it correctly to mean that unified
FileSink doesn't work properly with jobs that need to be
stopped-with-savepoints and restarted?

Should one use the deprecated StreamingFileSink until this is resolved?

On Tue, Nov 29, 2022 at 6:02 AM Fabian Paul (Jira)  wrote:

> Fabian Paul created FLINK-30238:
> ---
>
>  Summary: Unified Sink committer does not clean up state on
> final savepoint
>  Key: FLINK-30238
>  URL: https://issues.apache.org/jira/browse/FLINK-30238
>  Project: Flink
>   Issue Type: Bug
>   Components: Connectors / Common
> Affects Versions: 1.15.3, 1.17.0, 1.16.1
> Reporter: Fabian Paul
>
>
> During stop-with-savepoint the committer only commits the pending
> committables on notifyCheckpointComplete.
>
> This has several downsides.
>  * Last committableSummary has checkpoint id LONG.MAX and is never cleared
> from the state leading to that stop-with-savepoint does not work when the
> pipeline recovers from a savepoint
>  * While the committables are committed during stop-with-savepoint they
> are not forwarded to post-commit topology, potentially losing data and
> preventing to close open transactions.
>
>
>
> --
> This message was sent by Atlassian Jira
> (v8.20.10#820010)
>

Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

[Yanfei Lei]

Hi Martijn,

Thanks for bringing this up.

In the past two months, this channel has helped us find many benchmark fail
issues, like FLINK-29883
[1],
FLINK-29886 [2],
FLINK-30015 [3] and
FLINK-30181 [4]. I also
have tried investigating several of the frequently reported regressions and
replied under the notification in slack channel(copy them here):

   1. serializerHeavyString
   
:
   It is unstable for a long time, see [5]
   https://issues.apache.org/jira/browse/FLINK-27165 for possible reasons.
   2. Regressions are detected by a simple script which may have false
   positives and false negatives, especially for benchmarks with small
   absolute values, small value changes cause large percentage changes. see
   [6] for details.

 Maybe slidingWindow
(value~=600),
stateBackends.ROCKS

(value~=260) and serializerHeavyString
(value~=170)
are
not true regressions.

   1. For deployAllTasks.STREAMING
   
,
   this benchmark result is how much time it takes to deploy job, the less
   value the better performance, see [7] for details. FLINK-27571
   [8] would fix this
   problem.


As mentioned before, regressions are detected by a simple script that is
less stable, FLINK-29825 [9]
is created to improve the benchmark's stability. I planned to invite more
volunteers to monitor it after the checking of regression became more
stable, but I've been stuck with something else lately, sorry for the late
response.  Any suggestions on handling benchmark regressions/fails are
welcome.

[1] https://issues.apache.org/jira/browse/FLINK-29883

[2] https://issues.apache.org/jira/browse/FLINK-29886

[3] https://issues.apache.org/jira/browse/FLINK-30015

[4] https://issues.apache.org/jira/browse/FLINK-30181

[5] https://issues.apache.org/jira/browse/FLINK-27165

[6]
https://github.com/apache/flink-benchmarks/blob/master/regression_report.py#L132-L136

[7]
https://github.com/apache/flink-benchmarks/blob/master/src/main/java/org/apache/flink/scheduler/benchmark/deploying/DeployingTasksInStreamingJobBenchmarkExecutor.java#L58

[8] https://issues.apache.org/jira/browse/FLINK-27571

[9] https://issues.apache.org/jira/browse/FLINK-29825


Best,

Yanfei

Martijn Visser  于2022年11月29日周二 15:54写道：

> Hi,
>
> Is there any update to be expected on the benchmark? I see results of the
> benchmark being posted to Slack, but it appears that it's not being
> monitored and no follow-up actions are being taken. I think it's currently
> lacking a process on how to interpret the results and what action should
> be taken and by whom.
>
> Best regards,
>
> Martijn
>
> On Thu, Nov 3, 2022 at 12:22 PM Jing Ge  wrote:
>
> > Thanks yanfei for driving this!
> >
> > Looking forward to further discussion w.r.t. the workflow.
> >
> > Best regards,
> > Jing
> >
> > On Mon, Oct 31, 2022 at 6:04 PM Mason Chen 
> wrote:
> >
> > > +1, thanks for driving this!
> > >
> > > On a side note, can we also ensure that a performance summary report
> for
> > > Flink major version upgrades is in release notes, once this
> > infrastructure
> > > becomes mature? From the user perspective, it would be nice to know
> what
> > > the expected (or unexpected) regressions in a major version upgrade
> are.
> > > I've seen the community do something like this before (e.g. the major
> > > rocksdb version bump in 1.14?) and it was quite valuable to know that
> > > upfront!
> > >
> > > Best,
> > > Mason
> > >
> > > On Fri, Oct 28, 2022 at 1:46 AM weijie guo 
> > > wrote:
> > >
> > > > Thanks Yanfei for driving this.
> > > >
> > > > It allows us to easily find the problem of performance regression.
> > > > Especially recently, I have made some improvements to the scheduling
> > > > related parts, your work is very important to ensure that these
> changes
> > > do
> > > > not cause some unexpected problems.
> > > >
> > > > Best regards,
> > > >
> > > > Weijie
> > > >
> > > >
> > > > Congxian Qiu  于2022年10月28日周五 16:03写道：
> > > >
> > > > > Thanks for driving this and making the performance monitoring
> public,
> > > > this
> > > > > can make us know and resolve the performance problem quickly.
> > > > >
> > > > > Looking forward to the workflow and detailed descriptions fo
> > > > > flink-dev-benchmarks.
> > > > >
> > > > > Best,
> > > > > Congxian
> > > 

[SUMMARY] Flink 1.17 Release Sync 11/29/2022

[Qingsheng Ren]

Hi devs and users,

I'd like to share some highlights from the release sync on 11/29/2022.

1. @Contributors please update your progress on the release 1.17 wiki page
[1] before the sync meeting so that everyone could track it.

2. We have new CI stability tickets and owners should have been pinged on
JIRA or offline. Please take a look at your inbox.

3. Externalizing connectors is progressing well. We could expect most
connectors being externalized in the 1.17 release cycle.

4. It’ll be great to have some input for monitoring the performance test.
Currently there’s a slack channel reporting benchmark results periodically
as discussed in the dev ML [2], and we’re working on formalizing the work
on regression testing.

There are 7 weeks before the feature freeze on 1/17/2023. The next release
meeting will be on December 13th, 2022. Feel free to join us if you
are interested!

Google Meet: https://meet.google.com/wcx-fjbt-hhz
Dial-in: https://tel.meet/wcx-fjbt-hhz?pin=1940846765126

Best regards,
Leonard, Martijn, Matthias and Qingsheng

[1] https://cwiki.apache.org/confluence/display/FLINK/1.17+Release
[2] https://lists.apache.org/thread/zok62sx4m50c79htfp18ymq5vmtgbgxj

Re: [VOTE] Release flink-connector-dynamodb v3.0.0, release candidate #0

[Hamdy, Ahmed]

+1 (non-binding)

On 29/11/2022, 08:27, "Teoh, Hong"  wrote:

CAUTION: This email originated from outside of the organization. Do not 
click links or open attachments unless you can confirm the sender and know the 
content is safe.



+1 (non-binding)

* Hashes and Signatures look good
* All required files on dist.apache.org
* Tag is present in Github
* Verified source archive does not contain any binary files
* Source archive builds using maven
* Started packaged example SQL job using SQL client. Verified that it 
writes successfully to the sink table.
* Verified sink metrics look ok.


Cheers,
Hong

On 28/11/2022, 16:44, "Danny Cranmer"  wrote:

CAUTION: This email originated from outside of the organization. Do not 
click links or open attachments unless you can confirm the sender and know the 
content is safe.



Hi everyone,
Please review and vote on the release candidate #0 for the version 
3.0.0 as
follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)


The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release to be deployed to dist.apache.org 
[2],
which are signed with the key with fingerprint 125FD8DB [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag v3.0.0-rc0 [5],
* website pull request listing the new release [6].

The vote will be open for at least 72 hours (Thursday 1st December 17:00
UTC). It is adopted by majority approval, with at least 3 PMC 
affirmative
votes.

Please note, this is a new connector and the first release.

Thanks,
Danny

[1]

https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352277
[2]

https://dist.apache.org/repos/dist/dev/flink/flink-connector-aws-3.0.0-rc0
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4] 
https://repository.apache.org/content/repositories/orgapacheflink-1552/
[5] 
https://github.com/apache/flink-connector-aws/releases/tag/v3.0.0-rc0
[6] https://github.com/apache/flink-web/pull/588

[jira] [Created] (FLINK-30238) Unified Sink committer does not clean up state on final savepoint

[Fabian Paul (Jira)]

Fabian Paul created FLINK-30238:
---

 Summary: Unified Sink committer does not clean up state on final 
savepoint
 Key: FLINK-30238
 URL: https://issues.apache.org/jira/browse/FLINK-30238
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Common
Affects Versions: 1.15.3, 1.17.0, 1.16.1
Reporter: Fabian Paul


During stop-with-savepoint the committer only commits the pending committables 
on notifyCheckpointComplete.

This has several downsides.
 * Last committableSummary has checkpoint id LONG.MAX and is never cleared from 
the state leading to that stop-with-savepoint does not work when the pipeline 
recovers from a savepoint 
 * While the committables are committed during stop-with-savepoint they are not 
forwarded to post-commit topology, potentially losing data and preventing to 
close open transactions.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Release flink-connector-cassandra, release candidate #1

[Chesnay Schepler]

> the link to a tag in the initial message is wrong

Whoops, small bug in the message generator.

> Did we ever consider having the `flink-version` always as part of the 
connector version?


Not specifically, no.
I think it makes sense though that if you take the "3.0.0" source that 
you get "3.0.0" artifacts (and making such a suffix work for SNAPSHOT 
artifacts is a bit tricky because it's suddenly an infix).

In general the current setup is less complex.
I can see upsides though, like not having to modify the poms when 
staging the jars.

But medium-term we want to get rid of these suffixes anyway.

On 29/11/2022 10:36, Dawid Wysakowicz wrote:


+1 (binding)

- Downloaded artifacts
- Checked hash and signature
- No binaries in source archive found
- Verified NOTICE files
- Built from source code
- Verified that no SNAPSHOT versions exist, all versions point to 3.0.0 in
POM files
- Tag is OK
- Reviewed the Web PR

Few notes:

- the link to a tag in the initial message is wrong. It should've been 
https://github.com/apache/flink-connector-cassandra/releases/tag/v3.0.0-rc1
- I was a bit surprised that executing `mvn -DskipTests package` 
produces artifacts without the Flink version suffix. Did we ever 
consider having the `flink-version` always as part of the connector 
version?


Best,

Dawid

On 25/11/2022 10:31, Chesnay Schepler wrote:

Hi everyone,
Please review and vote on the release candidate #1 for the version 
3.0.0, as follows:

[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)


The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release to be deployed to 
dist.apache.org [2], which are signed with the key with fingerprint 
C2EED7B111D464BA [3],

* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag v3.0.0-1 [5],
* website pull request listing the new release [6].

The vote will be open for at least 72 hours. It is adopted by 
majority approval, with at least 3 PMC affirmative votes.


This is the first externalized released of the Cassandra connector 
and functionally identical to 1.16.0.



Thanks,
Chesnay

[1] https://issues.apache.org/jira/projects/FLINK/versions/12352593
[2] 
https://dist.apache.org/repos/dist/dev/flink/flink-connector-cassandra-3.0.0-rc1

[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4] 
https://repository.apache.org/content/repositories/orgapacheflink-1549/
[5] 
https://github.com/apache/flink-connector-cassandra/releases/tag/v3.0.0-1

[6] https://github.com/apache/flink-web/pull/587

Re: [VOTE] Release flink-connector-cassandra, release candidate #1

[Dawid Wysakowicz]

+1 (binding)

- Downloaded artifacts
- Checked hash and signature
- No binaries in source archive found
- Verified NOTICE files
- Built from source code
- Verified that no SNAPSHOT versions exist, all versions point to 3.0.0 in
POM files
- Tag is OK
- Reviewed the Web PR

Few notes:

- the link to a tag in the initial message is wrong. It should've been 
https://github.com/apache/flink-connector-cassandra/releases/tag/v3.0.0-rc1
- I was a bit surprised that executing `mvn -DskipTests package` 
produces artifacts without the Flink version suffix. Did we ever 
consider having the `flink-version` always as part of the connector version?


Best,

Dawid

On 25/11/2022 10:31, Chesnay Schepler wrote:

Hi everyone,
Please review and vote on the release candidate #1 for the version 
3.0.0, as follows:

[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)


The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release to be deployed to dist.apache.org 
[2], which are signed with the key with fingerprint C2EED7B111D464BA [3],

* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag v3.0.0-1 [5],
* website pull request listing the new release [6].

The vote will be open for at least 72 hours. It is adopted by majority 
approval, with at least 3 PMC affirmative votes.


This is the first externalized released of the Cassandra connector and 
functionally identical to 1.16.0.



Thanks,
Chesnay

[1] https://issues.apache.org/jira/projects/FLINK/versions/12352593
[2] 
https://dist.apache.org/repos/dist/dev/flink/flink-connector-cassandra-3.0.0-rc1

[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4] 
https://repository.apache.org/content/repositories/orgapacheflink-1549/
[5] 
https://github.com/apache/flink-connector-cassandra/releases/tag/v3.0.0-1

[6] https://github.com/apache/flink-web/pull/587



OpenPGP_0x31D2DD10BFC15A2D.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

[jira] [Created] (FLINK-30237) Only bundle a single Zookeeper version

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-30237:


 Summary: Only bundle a single Zookeeper version
 Key: FLINK-30237
 URL: https://issues.apache.org/jira/browse/FLINK-30237
 Project: Flink
  Issue Type: Technical Debt
  Components: Runtime / Coordination
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.17.0


Way back when we added support for ZK 3.5 we started bundling 2 zookeeper 
clients, because of incompatibilities between 3.4 and 3.5. This is no longer 
required, and we could simplify things again.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Release flink-connector-dynamodb v3.0.0, release candidate #0

[Teoh, Hong]

+1 (non-binding)

* Hashes and Signatures look good
* All required files on dist.apache.org
* Tag is present in Github
* Verified source archive does not contain any binary files
* Source archive builds using maven
* Started packaged example SQL job using SQL client. Verified that it writes 
successfully to the sink table.
* Verified sink metrics look ok.


Cheers,
Hong

On 28/11/2022, 16:44, "Danny Cranmer"  wrote:

CAUTION: This email originated from outside of the organization. Do not 
click links or open attachments unless you can confirm the sender and know the 
content is safe.



Hi everyone,
Please review and vote on the release candidate #0 for the version 3.0.0 as
follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)


The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release to be deployed to dist.apache.org [2],
which are signed with the key with fingerprint 125FD8DB [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag v3.0.0-rc0 [5],
* website pull request listing the new release [6].

The vote will be open for at least 72 hours (Thursday 1st December 17:00
UTC). It is adopted by majority approval, with at least 3 PMC affirmative
votes.

Please note, this is a new connector and the first release.

Thanks,
Danny

[1]

https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352277
[2]
https://dist.apache.org/repos/dist/dev/flink/flink-connector-aws-3.0.0-rc0
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4] https://repository.apache.org/content/repositories/orgapacheflink-1552/
[5] https://github.com/apache/flink-connector-aws/releases/tag/v3.0.0-rc0
[6] https://github.com/apache/flink-web/pull/588