[GitHub] [flink] Jiayi-Liao commented on pull request #17151: [FLINK-23180] Do not initialize checkpoint base locations when checkp…

[GitBox]

Jiayi-Liao commented on pull request #17151:
URL: https://github.com/apache/flink/pull/17151#issuecomment-920601219


   > > > I think in the end there is no harm doing so, but a the very least the 
javadoc must be updated. I would even suggest renaming initializeBaseLocations 
to initializeBaseLocationsForCheckpoints.
   > > 
   > > 
   > > I think @dawidwys 's suggestion is right, we should change the javadoc 
description. Since this class is not annoated as "Public" and might not be 
extended in user classes, it's fine to rename this method (the renaming commit 
could be a separate one).
   > 
   > @dawidwys @Myasuka You're right. Will fix this soon...
   
   Hi, I've added two new commits for comment updates and renaming of method. 
Please take an another look :).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] Airblader commented on pull request #17186: [FLINK-16203][table] Support JSON_OBJECT()

[GitBox]

Airblader commented on pull request #17186:
URL: https://github.com/apache/flink/pull/17186#issuecomment-920601084


   @JingsongLi Thanks for taking care of this PR! I squashed the fixup commits 
now.
   
   > This is interesting. Can we create a JIRA to track this issue? We can try 
to fix this in Calcite side or Flink overwrite.
   
   I'm not sure I understand; I did fix this on the Flink side now in this PR 
by excluding JSON_OBJECT from being reduced. What would the JIRA issue be for?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #16798: [FLINK-23651][python] Support RabbitMQ in PyFlink

[GitBox]

flinkbot edited a comment on pull request #16798:
URL: https://github.com/apache/flink/pull/16798#issuecomment-897826601


   
   ## CI report:
   
   * a2de26625d24fb8c554a1f2d286652af75cb1234 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24176)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-24300) MultipleInputOperator is running much more slowly in TPCDS

[Xintong Song (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-24300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415878#comment-17415878
 ] 

Xintong Song commented on FLINK-24300:
--

I'm upgrading this to blocker for now. We can downgrade it if the conclusion 
turns out to be otherwise.

> MultipleInputOperator is running much more slowly in TPCDS
> --
>
> Key: FLINK-24300
> URL: https://issues.apache.org/jira/browse/FLINK-24300
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / Network
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Zhilong Hong
>Priority: Blocker
> Fix For: 1.14.0, 1.15.0
>
> Attachments: 64570e4c56955713ca599fd1d7ae7be891a314c6.png, 
> detail-of-the-job.png, e3010c16947ed8da2ecb7d89a3aa08dacecc524a.png, 
> jstack-2.txt, jstack.txt
>
>
> When we are running TPCDS with release 1.14 we find that the job with 
> {{MultipleInputOperator}} is running much more slowly than before. With a 
> binary search among the commits, we find that the issue may be introduced by 
> FLINK-23408. 
> At the commit 64570e4c56955713ca599fd1d7ae7be891a314c6, the job in TPCDS runs 
> normally, as the image below illustrates:
> !64570e4c56955713ca599fd1d7ae7be891a314c6.png|width=600!
> At the commit e3010c16947ed8da2ecb7d89a3aa08dacecc524a, the job q2.sql gets 
> stuck for a pretty long time (longer than half an hour), as the image below 
> illustrates:
> !e3010c16947ed8da2ecb7d89a3aa08dacecc524a.png|width=600!
> The detail of the job is illustrated below:
> !detail-of-the-job.png|width=600!
> The job uses a {{MultipleInputOperator}} with one normal input and two 
> chained FileSource. It has finished reading the normal input and start to 
> read the chained source. Each chained source has one source data fetcher.
> We capture the jstack of the stuck tasks and attach the file below. From the 
> [^jstack.txt] we can see the main thread is blocked on waiting for the lock, 
> and the lock is held by a source data fetcher. The source data fetcher is 
> still running but the stack keeps on {{CompletableFuture.cleanStack}}.
> This issue happens in a batch job. However, from where it get blocked, it 
> seems also affects the streaming jobs.
> For the reference, the code of TPCDS we are running is located at 
> [https://github.com/ververica/flink-sql-benchmark/tree/dev].



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-24300) MultipleInputOperator is running much more slowly in TPCDS

[Xintong Song (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-24300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xintong Song updated FLINK-24300:
-
Fix Version/s: 1.15.0
   1.14.0

> MultipleInputOperator is running much more slowly in TPCDS
> --
>
> Key: FLINK-24300
> URL: https://issues.apache.org/jira/browse/FLINK-24300
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / Network
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Zhilong Hong
>Priority: Blocker
> Fix For: 1.14.0, 1.15.0
>
> Attachments: 64570e4c56955713ca599fd1d7ae7be891a314c6.png, 
> detail-of-the-job.png, e3010c16947ed8da2ecb7d89a3aa08dacecc524a.png, 
> jstack-2.txt, jstack.txt
>
>
> When we are running TPCDS with release 1.14 we find that the job with 
> {{MultipleInputOperator}} is running much more slowly than before. With a 
> binary search among the commits, we find that the issue may be introduced by 
> FLINK-23408. 
> At the commit 64570e4c56955713ca599fd1d7ae7be891a314c6, the job in TPCDS runs 
> normally, as the image below illustrates:
> !64570e4c56955713ca599fd1d7ae7be891a314c6.png|width=600!
> At the commit e3010c16947ed8da2ecb7d89a3aa08dacecc524a, the job q2.sql gets 
> stuck for a pretty long time (longer than half an hour), as the image below 
> illustrates:
> !e3010c16947ed8da2ecb7d89a3aa08dacecc524a.png|width=600!
> The detail of the job is illustrated below:
> !detail-of-the-job.png|width=600!
> The job uses a {{MultipleInputOperator}} with one normal input and two 
> chained FileSource. It has finished reading the normal input and start to 
> read the chained source. Each chained source has one source data fetcher.
> We capture the jstack of the stuck tasks and attach the file below. From the 
> [^jstack.txt] we can see the main thread is blocked on waiting for the lock, 
> and the lock is held by a source data fetcher. The source data fetcher is 
> still running but the stack keeps on {{CompletableFuture.cleanStack}}.
> This issue happens in a batch job. However, from where it get blocked, it 
> seems also affects the streaming jobs.
> For the reference, the code of TPCDS we are running is located at 
> [https://github.com/ververica/flink-sql-benchmark/tree/dev].



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-24300) MultipleInputOperator is running much more slowly in TPCDS

[Xintong Song (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-24300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xintong Song updated FLINK-24300:
-
Priority: Blocker  (was: Major)

> MultipleInputOperator is running much more slowly in TPCDS
> --
>
> Key: FLINK-24300
> URL: https://issues.apache.org/jira/browse/FLINK-24300
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / Network
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Zhilong Hong
>Priority: Blocker
> Attachments: 64570e4c56955713ca599fd1d7ae7be891a314c6.png, 
> detail-of-the-job.png, e3010c16947ed8da2ecb7d89a3aa08dacecc524a.png, 
> jstack-2.txt, jstack.txt
>
>
> When we are running TPCDS with release 1.14 we find that the job with 
> {{MultipleInputOperator}} is running much more slowly than before. With a 
> binary search among the commits, we find that the issue may be introduced by 
> FLINK-23408. 
> At the commit 64570e4c56955713ca599fd1d7ae7be891a314c6, the job in TPCDS runs 
> normally, as the image below illustrates:
> !64570e4c56955713ca599fd1d7ae7be891a314c6.png|width=600!
> At the commit e3010c16947ed8da2ecb7d89a3aa08dacecc524a, the job q2.sql gets 
> stuck for a pretty long time (longer than half an hour), as the image below 
> illustrates:
> !e3010c16947ed8da2ecb7d89a3aa08dacecc524a.png|width=600!
> The detail of the job is illustrated below:
> !detail-of-the-job.png|width=600!
> The job uses a {{MultipleInputOperator}} with one normal input and two 
> chained FileSource. It has finished reading the normal input and start to 
> read the chained source. Each chained source has one source data fetcher.
> We capture the jstack of the stuck tasks and attach the file below. From the 
> [^jstack.txt] we can see the main thread is blocked on waiting for the lock, 
> and the lock is held by a source data fetcher. The source data fetcher is 
> still running but the stack keeps on {{CompletableFuture.cleanStack}}.
> This issue happens in a batch job. However, from where it get blocked, it 
> seems also affects the streaming jobs.
> For the reference, the code of TPCDS we are running is located at 
> [https://github.com/ververica/flink-sql-benchmark/tree/dev].



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-24197) Streaming File Sink end-to-end test fails with : "RestClientException: [File upload failed.]"

[Xintong Song (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-24197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415877#comment-17415877
 ] 

Xintong Song commented on FLINK-24197:
--

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24174=logs=91bf6583-3fb2-592f-e4d4-d79d79c3230a=cc5499f8-bdde-5157-0d76-b6528ecd808e=9866

> Streaming File Sink end-to-end test fails with : "RestClientException: [File 
> upload failed.]"
> -
>
> Key: FLINK-24197
> URL: https://issues.apache.org/jira/browse/FLINK-24197
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / FileSystem, Tests
>Affects Versions: 1.14.0
>Reporter: Dawid Wysakowicz
>Assignee: Chesnay Schepler
>Priority: Major
>  Labels: pull-request-available, test-stability
> Fix For: 1.14.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=23672=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=070ff179-953e-5bda-71fa-d6599415701c=11040
> {code}
> Caused by: org.apache.flink.util.FlinkException: Failed to execute job 
> 'StreamingFileSinkProgram'.
>   at 
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2056)
>   at 
> org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:137)
>   at 
> org.apache.flink.client.program.StreamContextEnvironment.execute(StreamContextEnvironment.java:76)
>   at 
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1917)
>   at FileSinkProgram.main(FileSinkProgram.java:105)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
>   ... 8 more
> Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed to 
> submit JobGraph.
>   at 
> org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$11(RestClusterClient.java:433)
>   at 
> java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:884)
>   at 
> java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:866)
>   at 
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>   at 
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
>   at 
> org.apache.flink.util.concurrent.FutureUtils.lambda$retryOperationWithDelay$9(FutureUtils.java:373)
>   at 
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>   at 
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>   at 
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>   at 
> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>   at 
> java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:943)
>   at 
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.flink.runtime.rest.util.RestClientException: [File 
> upload failed.]
>   at 
> org.apache.flink.runtime.rest.RestClient.parseResponse(RestClient.java:532)
>   at 
> org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$3(RestClient.java:512)
>   at 
> java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:966)
>   at 
> java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:940)
>   ... 4 more
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Closed] (FLINK-24218) Translate "UNLOAD Statements" page of "SQL" into Chinese

[Jark Wu (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-24218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jark Wu closed FLINK-24218.
---
Fix Version/s: 1.15.0
   Resolution: Fixed

Fixed in master: 647103c7111583cdb6ab626d739cdd69ec7fdd06


> Translate "UNLOAD Statements" page of "SQL" into Chinese
> 
>
> Key: FLINK-24218
> URL: https://issues.apache.org/jira/browse/FLINK-24218
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation
>Reporter: wuguihu
>Assignee: wuguihu
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 1.15.0
>
>
> [https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/table/sql/unload/]
> docs/content.zh/docs/dev/table/sql/unload.md



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] wuchong merged pull request #17222: [FLINK-24218][doc]Translate "UNLOAD Statements" page of "SQL" into Ch…

[GitBox]

wuchong merged pull request #17222:
URL: https://github.com/apache/flink/pull/17222


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17293: Update hive_functions.md

[GitBox]

flinkbot edited a comment on pull request #17293:
URL: https://github.com/apache/flink/pull/17293#issuecomment-920552580


   
   ## CI report:
   
   * 365dae20fd07c91a7c4a6d9bb713a7963ccff4fb Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24180)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17233: FLINK-23949 adapt to Flink1.12

[GitBox]

flinkbot edited a comment on pull request #17233:
URL: https://github.com/apache/flink/pull/17233#issuecomment-916715178


   
   ## CI report:
   
   * fc5f053fdd2896295254a4e18185baff3424640f Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23912)
 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24001)
 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24179)
 
   * 1babe4eb904a6c433f4d3efdab4df0ea739a261f Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24182)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17203: [FLINK-22944][state] Optimize writing state changelog

[GitBox]

flinkbot edited a comment on pull request #17203:
URL: https://github.com/apache/flink/pull/17203#issuecomment-915404842


   
   ## CI report:
   
   * 4bd2434f31574c63e93fea17893e4a38dce60f0b Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24175)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-24305) BatchPandasUDAFITTests.test_over_window_aggregate_function fails on azure

[Xintong Song (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-24305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415875#comment-17415875
 ] 

Xintong Song commented on FLINK-24305:
--

Thanks for the info, [~dianfu].

I've labeled this as a 1.14 release blocker for now. Feel free to update the 
ticket if the conclusion is otherwise.

> BatchPandasUDAFITTests.test_over_window_aggregate_function fails on azure
> -
>
> Key: FLINK-24305
> URL: https://issues.apache.org/jira/browse/FLINK-24305
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python
>Affects Versions: 1.14.0, 1.12.5, 1.13.2, 1.15.0
>Reporter: Xintong Song
>Assignee: Huang Xingbo
>Priority: Blocker
>  Labels: test-stability
> Fix For: 1.14.0, 1.12.6, 1.13.3, 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24170=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=c67e71ed-6451-5d26-8920-5a8cf9651901=23011
> {code}
> Sep 15 20:40:43 cls =  'pyflink.table.tests.test_pandas_udaf.BatchPandasUDAFITTests'>
> Sep 15 20:40:43 actual = JavaObject id=o8666
> Sep 15 20:40:43 expected = ['+I[1, 4.335, 13, 5.5, 3.0, 3.0, 4.335, 
> 8.0, 5.0, 5.0]', '+I[1, 4.335, 5, 4.335, 3.0, 3.0, 2.5, 4.3330, 
> 4.0, 2.0]', '+I[2, 2.0, 9, 2.0, 4.0, 4.0, 2.0, 2.0, 4.0, 4.0]', '+I[3, 2.0, 
> 3, 2.0, 1.0, 1.0, 2.0, 2.0, 1.0, 1.0]']
> Sep 15 20:40:43 
> Sep 15 20:40:43 @classmethod
> Sep 15 20:40:43 def assert_equals(cls, actual, expected):
> Sep 15 20:40:43 if isinstance(actual, JavaObject):
> Sep 15 20:40:43 actual_py_list = cls.to_py_list(actual)
> Sep 15 20:40:43 else:
> Sep 15 20:40:43 actual_py_list = actual
> Sep 15 20:40:43 actual_py_list.sort()
> Sep 15 20:40:43 expected.sort()
> Sep 15 20:40:43 assert len(actual_py_list) == len(expected)
> Sep 15 20:40:43 >   assert all(x == y for x, y in zip(actual_py_list, 
> expected))
> Sep 15 20:40:43 E   AssertionError: assert False
> Sep 15 20:40:43 E+  where False = all( PyFlinkTestCase.assert_equals.. at 0x7f792d98b900>)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-24305) BatchPandasUDAFITTests.test_over_window_aggregate_function fails on azure

[Xintong Song (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-24305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415874#comment-17415874
 ] 

Xintong Song commented on FLINK-24305:
--

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24174=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=c67e71ed-6451-5d26-8920-5a8cf9651901=22918

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24174=logs=821b528f-1eed-5598-a3b4-7f748b13f261=6bb545dd-772d-5d8c-f258-f5085fba3295=25895

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24174=logs=bdd9ea51-4de2-506a-d4d9-f3930e4d2355=dd50312f-73b5-56b5-c172-4d81d03e2ef1=26081

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24174=logs=bf5e383b-9fd3-5f02-ca1c-8f788e2e76d3=85189c57-d8a0-5c9c-b61d-fc05cfac62cf=24835

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24174=logs=ff2e2ea5-07e3-5521-7b04-a4fc3ad765e9=1ec6382b-bafe-5817-63ae-eda7d4be718e=23411

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24174=logs=e92ecf6d-e207-5a42-7ff7-528ff0c5b259=40fc352e-9b4c-5fd8-363f-628f24b01ec2=24234

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24174=logs=3e4dd1a2-fe2f-5e5d-a581-48087e718d53=b4612f28-e3b5-5853-8a8b-610ae894217a=24259

> BatchPandasUDAFITTests.test_over_window_aggregate_function fails on azure
> -
>
> Key: FLINK-24305
> URL: https://issues.apache.org/jira/browse/FLINK-24305
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python
>Affects Versions: 1.14.0, 1.12.5, 1.13.2, 1.15.0
>Reporter: Xintong Song
>Assignee: Huang Xingbo
>Priority: Blocker
>  Labels: test-stability
> Fix For: 1.14.0, 1.12.6, 1.13.3, 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24170=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=c67e71ed-6451-5d26-8920-5a8cf9651901=23011
> {code}
> Sep 15 20:40:43 cls =  'pyflink.table.tests.test_pandas_udaf.BatchPandasUDAFITTests'>
> Sep 15 20:40:43 actual = JavaObject id=o8666
> Sep 15 20:40:43 expected = ['+I[1, 4.335, 13, 5.5, 3.0, 3.0, 4.335, 
> 8.0, 5.0, 5.0]', '+I[1, 4.335, 5, 4.335, 3.0, 3.0, 2.5, 4.3330, 
> 4.0, 2.0]', '+I[2, 2.0, 9, 2.0, 4.0, 4.0, 2.0, 2.0, 4.0, 4.0]', '+I[3, 2.0, 
> 3, 2.0, 1.0, 1.0, 2.0, 2.0, 1.0, 1.0]']
> Sep 15 20:40:43 
> Sep 15 20:40:43 @classmethod
> Sep 15 20:40:43 def assert_equals(cls, actual, expected):
> Sep 15 20:40:43 if isinstance(actual, JavaObject):
> Sep 15 20:40:43 actual_py_list = cls.to_py_list(actual)
> Sep 15 20:40:43 else:
> Sep 15 20:40:43 actual_py_list = actual
> Sep 15 20:40:43 actual_py_list.sort()
> Sep 15 20:40:43 expected.sort()
> Sep 15 20:40:43 assert len(actual_py_list) == len(expected)
> Sep 15 20:40:43 >   assert all(x == y for x, y in zip(actual_py_list, 
> expected))
> Sep 15 20:40:43 E   AssertionError: assert False
> Sep 15 20:40:43 E+  where False = all( PyFlinkTestCase.assert_equals.. at 0x7f792d98b900>)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-24305) BatchPandasUDAFITTests.test_over_window_aggregate_function fails on azure

[Dian Fu (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-24305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415872#comment-17415872
 ] 

Dian Fu commented on FLINK-24305:
-

Thanks [~xtsong], [~hxbks2ks] is currently looking into this issue. We suspect 
that it's caused by protobuf 3.18.0 which was released today. We are still 
confirming that. If this is true, we need to limit the version of protobuf.

> BatchPandasUDAFITTests.test_over_window_aggregate_function fails on azure
> -
>
> Key: FLINK-24305
> URL: https://issues.apache.org/jira/browse/FLINK-24305
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python
>Affects Versions: 1.14.0, 1.12.5, 1.13.2, 1.15.0
>Reporter: Xintong Song
>Assignee: Huang Xingbo
>Priority: Blocker
>  Labels: test-stability
> Fix For: 1.14.0, 1.12.6, 1.13.3, 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24170=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=c67e71ed-6451-5d26-8920-5a8cf9651901=23011
> {code}
> Sep 15 20:40:43 cls =  'pyflink.table.tests.test_pandas_udaf.BatchPandasUDAFITTests'>
> Sep 15 20:40:43 actual = JavaObject id=o8666
> Sep 15 20:40:43 expected = ['+I[1, 4.335, 13, 5.5, 3.0, 3.0, 4.335, 
> 8.0, 5.0, 5.0]', '+I[1, 4.335, 5, 4.335, 3.0, 3.0, 2.5, 4.3330, 
> 4.0, 2.0]', '+I[2, 2.0, 9, 2.0, 4.0, 4.0, 2.0, 2.0, 4.0, 4.0]', '+I[3, 2.0, 
> 3, 2.0, 1.0, 1.0, 2.0, 2.0, 1.0, 1.0]']
> Sep 15 20:40:43 
> Sep 15 20:40:43 @classmethod
> Sep 15 20:40:43 def assert_equals(cls, actual, expected):
> Sep 15 20:40:43 if isinstance(actual, JavaObject):
> Sep 15 20:40:43 actual_py_list = cls.to_py_list(actual)
> Sep 15 20:40:43 else:
> Sep 15 20:40:43 actual_py_list = actual
> Sep 15 20:40:43 actual_py_list.sort()
> Sep 15 20:40:43 expected.sort()
> Sep 15 20:40:43 assert len(actual_py_list) == len(expected)
> Sep 15 20:40:43 >   assert all(x == y for x, y in zip(actual_py_list, 
> expected))
> Sep 15 20:40:43 E   AssertionError: assert False
> Sep 15 20:40:43 E+  where False = all( PyFlinkTestCase.assert_equals.. at 0x7f792d98b900>)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] RocMarshal commented on a change in pull request #17222: [FLINK-24218][doc]Translate "UNLOAD Statements" page of "SQL" into Ch…

[GitBox]

RocMarshal commented on a change in pull request #17222:
URL: https://github.com/apache/flink/pull/17222#discussion_r709784728



##
File path: docs/content/docs/dev/table/sql/unload.md
##
@@ -46,7 +46,7 @@ The following examples show how to run a UNLOAD statement in 
`TableEnvironment`.
 {{< /tab >}}
 {{< tab "Python" >}}
 
-UNLOAD statements can be executed with the `executeSql()` method of the 
`TableEnvironment`. The `executeSql()` method returns 'OK' for a successful 
LOAD operation; otherwise it will throw an exception.
+UNLOAD statements can be executed with the `execute_sql()` method of the 
`TableEnvironment`. The `execute_sql()` method returns 'OK' for a successful 
LOAD operation; otherwise it will throw an exception.

Review comment:
   Good catch.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (FLINK-24306) group by index throw SqlValidatorException

[zlzhang0122 (Jira)]

zlzhang0122 created FLINK-24306:
---

 Summary: group by index throw SqlValidatorException
 Key: FLINK-24306
 URL: https://issues.apache.org/jira/browse/FLINK-24306
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / API
Affects Versions: 1.13.1, 1.12.2
Reporter: zlzhang0122
 Fix For: 1.15.0
 Attachments: sql_exception.png

We create a table using following DDL:
{code:java}
create table if not exists datagen_source ( 
id int,
name string,
sex string,
age int,
birthday string,
proc_time as proctime()
) with (
'connector' = 'datagen',
'rows-per-second' = '1',
'fields.id.kind' = 'random',
'fields.id.min' = '1',
'fields.id.max' = '200');{code}
When we running 
{code:java}
select id, count(*) from datagen_source group by id;{code}
Everything will be fine.But if we running 
{code:java}
select id, count(*) from datagen_source group by 1;{code}
We will get a SqlValidatorException like this:
!sql_exception.png!
>From the error message we know that Flink SQL also can reconize that the '1' 
>index reprent the 'id'.Since MySql\Hive\Spark SQL\etc. all support group by 
>index, I think Flink also should support this syntax too.
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot edited a comment on pull request #17233: FLINK-23949 adapt to Flink1.12

[GitBox]

flinkbot edited a comment on pull request #17233:
URL: https://github.com/apache/flink/pull/17233#issuecomment-916715178


   
   ## CI report:
   
   * fc5f053fdd2896295254a4e18185baff3424640f Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23912)
 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24001)
 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24179)
 
   * 1babe4eb904a6c433f4d3efdab4df0ea739a261f UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17205: [FLINK-24168][table-planner] Update MATCH_ROWTIME function which could receive 0 argument or 1 argument

[GitBox]

flinkbot edited a comment on pull request #17205:
URL: https://github.com/apache/flink/pull/17205#issuecomment-915449441


   
   ## CI report:
   
   * 170096fc6b1f13b86f5ceecfad89f8bf61b16181 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24123)
 
   * c8ea9552f00c97efc8cbb3e19fa23b89b8c69772 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24181)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17293: Update hive_functions.md

[GitBox]

flinkbot edited a comment on pull request #17293:
URL: https://github.com/apache/flink/pull/17293#issuecomment-920552580


   
   ## CI report:
   
   * 365dae20fd07c91a7c4a6d9bb713a7963ccff4fb Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24180)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17205: [FLINK-24168][table-planner] Update MATCH_ROWTIME function which could receive 0 argument or 1 argument

[GitBox]

flinkbot edited a comment on pull request #17205:
URL: https://github.com/apache/flink/pull/17205#issuecomment-915449441


   
   ## CI report:
   
   * 170096fc6b1f13b86f5ceecfad89f8bf61b16181 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24123)
 
   * c8ea9552f00c97efc8cbb3e19fa23b89b8c69772 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] iyupeng commented on pull request #14894: [FLINK-20895][flink-table-planner-blink] support local aggregate push down in blink planner

[GitBox]

iyupeng commented on pull request #14894:
URL: https://github.com/apache/flink/pull/14894#issuecomment-920562189


   @sebastianliu 
   Hello Sebastian, will you continue this work for `local aggregate push down`?
   
   We have an implementation of `local aggregate push down` too. 
   
   However, it's not so completed as this pull request for the lack of test 
cases.
   
   If this pull request is up-streamed, then we could add more work like 
aggregates with arithmetical expressions:
   
   For example:
   ```
   select avg(l_extendedprice * l_discount) from lineitem group by l_suppkey;
   
   == Abstract Syntax Tree ==
   LogicalProject(EXPR$0=[$1])
   +- LogicalAggregate(group=[{0}], EXPR$0=[AVG($1)])
  +- LogicalProject(l_suppkey=[$2], $f1=[*($5, $6)])
 +- LogicalTableScan(table=[[myhive, tpch_1g_snappy, lineitem]])
   
   == Optimized Physical Plan ==
   Calc(select=[EXPR$0])
   +- HashAggregate(isMerge=[true], groupBy=[l_suppkey], select=[l_suppkey, 
Final_AVG(sum$0, count$1) AS EXPR$0])
  +- Exchange(distribution=[hash[l_suppkey]])
 +- LocalHashAggregate(groupBy=[l_suppkey], select=[l_suppkey, 
Partial_AVG($f1) AS (sum$0, count$1)])
+- Calc(select=[l_suppkey, *(l_extendedprice, l_discount) AS $f1])
   +- TableSourceScan(table=[[myhive, tpch_1g_snappy, lineitem, 
project=[l_suppkey, l_extendedprice, l_discount]]], fields=[l_suppkey, 
l_extendedprice, l_discount])
   ```
   
   Thanks.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (FLINK-24305) BatchPandasUDAFITTests.test_over_window_aggregate_function fails on azure

[Xintong Song (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-24305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xintong Song updated FLINK-24305:
-
Fix Version/s: 1.13.3
   1.14.0

> BatchPandasUDAFITTests.test_over_window_aggregate_function fails on azure
> -
>
> Key: FLINK-24305
> URL: https://issues.apache.org/jira/browse/FLINK-24305
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python
>Affects Versions: 1.14.0, 1.12.5, 1.13.2, 1.15.0
>Reporter: Xintong Song
>Assignee: Huang Xingbo
>Priority: Blocker
>  Labels: test-stability
> Fix For: 1.14.0, 1.12.6, 1.13.3, 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24170=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=c67e71ed-6451-5d26-8920-5a8cf9651901=23011
> {code}
> Sep 15 20:40:43 cls =  'pyflink.table.tests.test_pandas_udaf.BatchPandasUDAFITTests'>
> Sep 15 20:40:43 actual = JavaObject id=o8666
> Sep 15 20:40:43 expected = ['+I[1, 4.335, 13, 5.5, 3.0, 3.0, 4.335, 
> 8.0, 5.0, 5.0]', '+I[1, 4.335, 5, 4.335, 3.0, 3.0, 2.5, 4.3330, 
> 4.0, 2.0]', '+I[2, 2.0, 9, 2.0, 4.0, 4.0, 2.0, 2.0, 4.0, 4.0]', '+I[3, 2.0, 
> 3, 2.0, 1.0, 1.0, 2.0, 2.0, 1.0, 1.0]']
> Sep 15 20:40:43 
> Sep 15 20:40:43 @classmethod
> Sep 15 20:40:43 def assert_equals(cls, actual, expected):
> Sep 15 20:40:43 if isinstance(actual, JavaObject):
> Sep 15 20:40:43 actual_py_list = cls.to_py_list(actual)
> Sep 15 20:40:43 else:
> Sep 15 20:40:43 actual_py_list = actual
> Sep 15 20:40:43 actual_py_list.sort()
> Sep 15 20:40:43 expected.sort()
> Sep 15 20:40:43 assert len(actual_py_list) == len(expected)
> Sep 15 20:40:43 >   assert all(x == y for x, y in zip(actual_py_list, 
> expected))
> Sep 15 20:40:43 E   AssertionError: assert False
> Sep 15 20:40:43 E+  where False = all( PyFlinkTestCase.assert_equals.. at 0x7f792d98b900>)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-24305) BatchPandasUDAFITTests.test_over_window_aggregate_function fails on azure

[Xintong Song (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-24305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xintong Song updated FLINK-24305:
-
Affects Version/s: 1.14.0
   1.13.2

> BatchPandasUDAFITTests.test_over_window_aggregate_function fails on azure
> -
>
> Key: FLINK-24305
> URL: https://issues.apache.org/jira/browse/FLINK-24305
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python
>Affects Versions: 1.14.0, 1.12.5, 1.13.2, 1.15.0
>Reporter: Xintong Song
>Assignee: Huang Xingbo
>Priority: Blocker
>  Labels: test-stability
> Fix For: 1.12.6, 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24170=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=c67e71ed-6451-5d26-8920-5a8cf9651901=23011
> {code}
> Sep 15 20:40:43 cls =  'pyflink.table.tests.test_pandas_udaf.BatchPandasUDAFITTests'>
> Sep 15 20:40:43 actual = JavaObject id=o8666
> Sep 15 20:40:43 expected = ['+I[1, 4.335, 13, 5.5, 3.0, 3.0, 4.335, 
> 8.0, 5.0, 5.0]', '+I[1, 4.335, 5, 4.335, 3.0, 3.0, 2.5, 4.3330, 
> 4.0, 2.0]', '+I[2, 2.0, 9, 2.0, 4.0, 4.0, 2.0, 2.0, 4.0, 4.0]', '+I[3, 2.0, 
> 3, 2.0, 1.0, 1.0, 2.0, 2.0, 1.0, 1.0]']
> Sep 15 20:40:43 
> Sep 15 20:40:43 @classmethod
> Sep 15 20:40:43 def assert_equals(cls, actual, expected):
> Sep 15 20:40:43 if isinstance(actual, JavaObject):
> Sep 15 20:40:43 actual_py_list = cls.to_py_list(actual)
> Sep 15 20:40:43 else:
> Sep 15 20:40:43 actual_py_list = actual
> Sep 15 20:40:43 actual_py_list.sort()
> Sep 15 20:40:43 expected.sort()
> Sep 15 20:40:43 assert len(actual_py_list) == len(expected)
> Sep 15 20:40:43 >   assert all(x == y for x, y in zip(actual_py_list, 
> expected))
> Sep 15 20:40:43 E   AssertionError: assert False
> Sep 15 20:40:43 E+  where False = all( PyFlinkTestCase.assert_equals.. at 0x7f792d98b900>)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-24305) BatchPandasUDAFITTests.test_over_window_aggregate_function fails on azure

[Xintong Song (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-24305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415855#comment-17415855
 ] 

Xintong Song commented on FLINK-24305:
--

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24173=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=8d78fe4f-d658-5c70-12f8-4921589024c3=22090

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24173=logs=821b528f-1eed-5598-a3b4-7f748b13f261=4fad9527-b9a5-5015-1b70-8356e5c91490=23451

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24173=logs=bdd9ea51-4de2-506a-d4d9-f3930e4d2355=98717c4f-b888-5636-bb1e-db7aca25755e=23287

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24173=logs=bf5e383b-9fd3-5f02-ca1c-8f788e2e76d3=f5211ead-5e53-5af8-f827-4dbf08df26bb=21959

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24173=logs=ff2e2ea5-07e3-5521-7b04-a4fc3ad765e9=613f538c-bcef-59e6-f9cd-9714bec9fb97=24718

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24173=logs=e92ecf6d-e207-5a42-7ff7-528ff0c5b259=d59eb898-29f7-5a99-91a7-b2dfc3e8a653=22094

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24173=logs=3e4dd1a2-fe2f-5e5d-a581-48087e718d53=45a89cfc-9ff2-5909-6443-6c732efcf06b=23426

> BatchPandasUDAFITTests.test_over_window_aggregate_function fails on azure
> -
>
> Key: FLINK-24305
> URL: https://issues.apache.org/jira/browse/FLINK-24305
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python
>Affects Versions: 1.12.5, 1.15.0
>Reporter: Xintong Song
>Assignee: Huang Xingbo
>Priority: Blocker
>  Labels: test-stability
> Fix For: 1.12.6, 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24170=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=c67e71ed-6451-5d26-8920-5a8cf9651901=23011
> {code}
> Sep 15 20:40:43 cls =  'pyflink.table.tests.test_pandas_udaf.BatchPandasUDAFITTests'>
> Sep 15 20:40:43 actual = JavaObject id=o8666
> Sep 15 20:40:43 expected = ['+I[1, 4.335, 13, 5.5, 3.0, 3.0, 4.335, 
> 8.0, 5.0, 5.0]', '+I[1, 4.335, 5, 4.335, 3.0, 3.0, 2.5, 4.3330, 
> 4.0, 2.0]', '+I[2, 2.0, 9, 2.0, 4.0, 4.0, 2.0, 2.0, 4.0, 4.0]', '+I[3, 2.0, 
> 3, 2.0, 1.0, 1.0, 2.0, 2.0, 1.0, 1.0]']
> Sep 15 20:40:43 
> Sep 15 20:40:43 @classmethod
> Sep 15 20:40:43 def assert_equals(cls, actual, expected):
> Sep 15 20:40:43 if isinstance(actual, JavaObject):
> Sep 15 20:40:43 actual_py_list = cls.to_py_list(actual)
> Sep 15 20:40:43 else:
> Sep 15 20:40:43 actual_py_list = actual
> Sep 15 20:40:43 actual_py_list.sort()
> Sep 15 20:40:43 expected.sort()
> Sep 15 20:40:43 assert len(actual_py_list) == len(expected)
> Sep 15 20:40:43 >   assert all(x == y for x, y in zip(actual_py_list, 
> expected))
> Sep 15 20:40:43 E   AssertionError: assert False
> Sep 15 20:40:43 E+  where False = all( PyFlinkTestCase.assert_equals.. at 0x7f792d98b900>)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (FLINK-24305) BatchPandasUDAFITTests.test_over_window_aggregate_function fails on azure

[Huang Xingbo (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-24305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Huang Xingbo reassigned FLINK-24305:


Assignee: Huang Xingbo

> BatchPandasUDAFITTests.test_over_window_aggregate_function fails on azure
> -
>
> Key: FLINK-24305
> URL: https://issues.apache.org/jira/browse/FLINK-24305
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python
>Affects Versions: 1.12.5, 1.15.0
>Reporter: Xintong Song
>Assignee: Huang Xingbo
>Priority: Blocker
>  Labels: test-stability
> Fix For: 1.12.6, 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24170=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=c67e71ed-6451-5d26-8920-5a8cf9651901=23011
> {code}
> Sep 15 20:40:43 cls =  'pyflink.table.tests.test_pandas_udaf.BatchPandasUDAFITTests'>
> Sep 15 20:40:43 actual = JavaObject id=o8666
> Sep 15 20:40:43 expected = ['+I[1, 4.335, 13, 5.5, 3.0, 3.0, 4.335, 
> 8.0, 5.0, 5.0]', '+I[1, 4.335, 5, 4.335, 3.0, 3.0, 2.5, 4.3330, 
> 4.0, 2.0]', '+I[2, 2.0, 9, 2.0, 4.0, 4.0, 2.0, 2.0, 4.0, 4.0]', '+I[3, 2.0, 
> 3, 2.0, 1.0, 1.0, 2.0, 2.0, 1.0, 1.0]']
> Sep 15 20:40:43 
> Sep 15 20:40:43 @classmethod
> Sep 15 20:40:43 def assert_equals(cls, actual, expected):
> Sep 15 20:40:43 if isinstance(actual, JavaObject):
> Sep 15 20:40:43 actual_py_list = cls.to_py_list(actual)
> Sep 15 20:40:43 else:
> Sep 15 20:40:43 actual_py_list = actual
> Sep 15 20:40:43 actual_py_list.sort()
> Sep 15 20:40:43 expected.sort()
> Sep 15 20:40:43 assert len(actual_py_list) == len(expected)
> Sep 15 20:40:43 >   assert all(x == y for x, y in zip(actual_py_list, 
> expected))
> Sep 15 20:40:43 E   AssertionError: assert False
> Sep 15 20:40:43 E+  where False = all( PyFlinkTestCase.assert_equals.. at 0x7f792d98b900>)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot commented on pull request #17293: Update hive_functions.md

[GitBox]

flinkbot commented on pull request #17293:
URL: https://github.com/apache/flink/pull/17293#issuecomment-920552580


   
   ## CI report:
   
   * 365dae20fd07c91a7c4a6d9bb713a7963ccff4fb UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17275: [FLINK-24271][docs] Add document for special char in JSON_VALUE and JSON_QUERY

[GitBox]

flinkbot edited a comment on pull request #17275:
URL: https://github.com/apache/flink/pull/17275#issuecomment-918884274


   
   ## CI report:
   
   * c8ae662e089e841e3c0fd870ca4e01335ef4e871 UNKNOWN
   * e02b18793c9e4c99afd3b97c72e83d62eb6030de Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24160)
 
   * 8853991853d0d10372fbae33f4a8cd6ca2969511 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24177)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17233: FLINK-23949 adapt to Flink1.12

[GitBox]

flinkbot edited a comment on pull request #17233:
URL: https://github.com/apache/flink/pull/17233#issuecomment-916715178


   
   ## CI report:
   
   * fc5f053fdd2896295254a4e18185baff3424640f Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23912)
 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24001)
 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24179)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Assigned] (FLINK-23827) Fix ModifiedMonotonicity inference for some node

[godfrey he (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-23827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

godfrey he reassigned FLINK-23827:
--

Assignee: Shuo Cheng

> Fix ModifiedMonotonicity inference for some node
> 
>
> Key: FLINK-23827
> URL: https://issues.apache.org/jira/browse/FLINK-23827
> Project: Flink
>  Issue Type: Bug
>  Components: Table SQL / Planner
>Affects Versions: 1.14.0
>Reporter: Shuo Cheng
>Assignee: Shuo Cheng
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.14.1
>
>
> ModifiedMonotonicity handler do not handle some node properly, such as 
> IntermediateTableScan, Deduplicate and LookupJoin.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot commented on pull request #17293: Update hive_functions.md

[GitBox]

flinkbot commented on pull request #17293:
URL: https://github.com/apache/flink/pull/17293#issuecomment-920548477


   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Automated Checks
   Last check on commit 365dae20fd07c91a7c4a6d9bb713a7963ccff4fb (Thu Sep 16 
03:32:32 UTC 2021)
   
   **Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up 
to date!
* **Invalid pull request title: No valid Jira ID provided**
   
   
   Mention the bot in a comment to re-run the automated checks.
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full 
explanation of the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] whb-bigdata opened a new pull request #17293: Update hive_functions.md

[GitBox]

whb-bigdata opened a new pull request #17293:
URL: https://github.com/apache/flink/pull/17293


   Should use “loadModule”
   
   
   
   ## What is the purpose of the change
   
   *(For example: This pull request makes task deployment go through the blob 
server, rather than through RPC. That way we avoid re-transferring them on each 
deployment (during recovery).)*
   
   
   ## Brief change log
   
   *(for example:)*
 - *The TaskInfo is stored in the blob store on job creation time as a 
persistent artifact*
 - *Deployments RPC transmits only the blob storage reference*
 - *TaskManagers retrieve the TaskInfo from the blob cache*
   
   
   ## Verifying this change
   
   *(Please pick either of the following options)*
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   *(or)*
   
   This change is already covered by existing tests, such as *(please describe 
tests)*.
   
   *(or)*
   
   This change added tests and can be verified as follows:
   
   *(example:)*
 - *Added integration tests for end-to-end deployment with large payloads 
(100MB)*
 - *Extended integration test for recovery after master (JobManager) 
failure*
 - *Added test that validates that TaskInfo is transferred only once across 
recoveries*
 - *Manually verified the change by running a 4 node cluser with 2 
JobManagers and 4 TaskManagers, a stateful streaming program, and killing one 
JobManager and two TaskManagers during the execution, verifying that recovery 
happens correctly.*
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): (yes / no)
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (yes / no)
 - The serializers: (yes / no / don't know)
 - The runtime per-record code paths (performance sensitive): (yes / no / 
don't know)
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
 - The S3 file system connector: (yes / no / don't know)
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (yes / no)
 - If yes, how is the feature documented? (not applicable / docs / JavaDocs 
/ not documented)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (FLINK-24159) document of entropy injection may mislead users

[Feifan Wang (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-24159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Feifan Wang updated FLINK-24159:

Description: 
FLINK-9061 incroduce entropy inject to s3 path for better scalability, but in 
document of 
[entropy-injection-for-s3-file-systems|https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/filesystems/s3/#entropy-injection-for-s3-file-systems]
 use a example with checkpoint directory 
"{color:#ff}s3://my-bucket/checkpoints/_entropy_/dashboard-job/{color}", 
with this configuration every checkpoint key will still start with constant 
checkpoints/ prefix which actually reduces scalability.

Thanks to dmtolpeko for describing this issue in his blog ( 
[flink-and-s3-entropy-injection-for-checkpoints 
|http://cloudsqale.com/2021/01/02/flink-and-s3-entropy-injection-for-checkpoints/]).
h3. Proposal

alter the checkpoint directory in document of 
[entropy-injection-for-s3-file-systems|https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/filesystems/s3/#entropy-injection-for-s3-file-systems]
 to "{color:#ff}s3://my-bucket/_entropy_/checkpoints/dashboard-job/{color}" 
(make entropy key at start of keys).

 

If this proposal is appropriate, I am glad to submit a PR to modify the 
document here. Any other ideas for this ?

  was:
FLINK-9061 incroduce entropy inject to s3 path for better scalability, but in 
document of 
[entropy-injection-for-s3-file-systems|https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/filesystems/s3/#entropy-injection-for-s3-file-systems]
 use a example with checkpoint directory 
"{color:#FF}s3://my-bucket/checkpoints/_entropy_/dashboard-job/{color}", 
with this configuration every checkpoint key will still start with constant 
checkpoints/ prefix which actually reduces scalability.

Thanks to dmtolpeko for describing this issue in his blog ( 
[flink-and-s3-entropy-injection-for-checkpoints 
|http://cloudsqale.com/2021/01/02/flink-and-s3-entropy-injection-for-checkpoints/]).
h3. Proposal

alter the checkpoint directory in document of 
[entropy-injection-for-s3-file-systems|https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/filesystems/s3/#entropy-injection-for-s3-file-systems]
 to "{color:#FF}s3://my-bucket/_entropy_/checkpoints/dashboard-job/{color}" 
(make entropy key at start of keys).

 

If this proposal is appropriate, I am glad to submit a PR to modify the 
document here. Any other ideas for this ?


> document of entropy injection may mislead users
> ---
>
> Key: FLINK-24159
> URL: https://issues.apache.org/jira/browse/FLINK-24159
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation, Runtime / Checkpointing
>Reporter: Feifan Wang
>Priority: Major
>
> FLINK-9061 incroduce entropy inject to s3 path for better scalability, but in 
> document of 
> [entropy-injection-for-s3-file-systems|https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/filesystems/s3/#entropy-injection-for-s3-file-systems]
>  use a example with checkpoint directory 
> "{color:#ff}s3://my-bucket/checkpoints/_entropy_/dashboard-job/{color}", 
> with this configuration every checkpoint key will still start with constant 
> checkpoints/ prefix which actually reduces scalability.
> Thanks to dmtolpeko for describing this issue in his blog ( 
> [flink-and-s3-entropy-injection-for-checkpoints 
> |http://cloudsqale.com/2021/01/02/flink-and-s3-entropy-injection-for-checkpoints/]).
> h3. Proposal
> alter the checkpoint directory in document of 
> [entropy-injection-for-s3-file-systems|https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/filesystems/s3/#entropy-injection-for-s3-file-systems]
>  to 
> "{color:#ff}s3://my-bucket/_entropy_/checkpoints/dashboard-job/{color}" 
> (make entropy key at start of keys).
>  
> If this proposal is appropriate, I am glad to submit a PR to modify the 
> document here. Any other ideas for this ?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot edited a comment on pull request #17275: [FLINK-24271][docs] Add document for special char in JSON_VALUE and JSON_QUERY

[GitBox]

flinkbot edited a comment on pull request #17275:
URL: https://github.com/apache/flink/pull/17275#issuecomment-918884274


   
   ## CI report:
   
   * c8ae662e089e841e3c0fd870ca4e01335ef4e871 UNKNOWN
   * e02b18793c9e4c99afd3b97c72e83d62eb6030de Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24160)
 
   * 8853991853d0d10372fbae33f4a8cd6ca2969511 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-24274) Wrong parameter order in documentation of State Processor API

[Feifan Wang (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-24274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415848#comment-17415848
 ] 

Feifan Wang commented on FLINK-24274:
-

Hi [~sjwiesman], I've submit submitted a pr to master. Is there requirement fix 
on previous versions ?

 

BTW, I‘ve mentioned anthor documentation problem in FLINK-24159 , would you 
like check out it ?

> Wrong parameter order in documentation of State Processor API 
> --
>
> Key: FLINK-24274
> URL: https://issues.apache.org/jira/browse/FLINK-24274
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Feifan Wang
>Assignee: Feifan Wang
>Priority: Major
>  Labels: pull-request-available
> Attachments: image-2021-09-14-02-09-44-334.png, 
> image-2021-09-14-02-11-12-034.png
>
>
> Wrong order of parameters path and stateBackend in example code of [State 
> Processor Api # 
> modifying-savepoints|https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/libs/state_processor_api/#modifying-savepoints]
> !image-2021-09-14-02-09-44-334.png|width=489,height=126!
> !image-2021-09-14-02-11-12-034.png|width=478,height=222!
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] Myasuka commented on a change in pull request #17203: [FLINK-22944][state] Optimize writing state changelog

[GitBox]

Myasuka commented on a change in pull request #17203:
URL: https://github.com/apache/flink/pull/17203#discussion_r709721022



##
File path: 
flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/restore/ChangelogBackendRestoreOperation.java
##
@@ -80,14 +89,19 @@
 ChangelogKeyedStateBackend backend,
 ChangelogStateBackendHandle backendHandle,
 StateChangelogHandleReader changelogHandleReader,
-ClassLoader classLoader)
+ClassLoader classLoader,
+Map> metadataByBackend)
 throws Exception {
 for (ChangelogStateHandle changelogHandle :
 backendHandle.getNonMaterializedStateHandles()) {
+Map stateIds =
+metadataByBackend.computeIfAbsent(
+backendHandle.getBackendID(), ign -> new 
HashMap<>());

Review comment:
   To answer the question that whether to add a unit test, I checked all 
tests under `flink-statatebackend-changelog` module, and found that the 
`stateIds` here is always empty. This could be explained that we would not 
restore two state handles from previous same statebackend, and the statd-ids 
are updated during `applyOperation`.
   If so, do we still need the API of `getBackendUID`?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-24086) Do not re-register SharedStateRegistry to reduce the recovery time of the job

[ming li (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-24086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415847#comment-17415847
 ] 

ming li commented on FLINK-24086:
-

Hi, [~dwysakowicz], I am sorry for not being able to update in time, I will 
update this PR as soon as possible.

> Do not re-register SharedStateRegistry to reduce the recovery time of the job
> -
>
> Key: FLINK-24086
> URL: https://issues.apache.org/jira/browse/FLINK-24086
> Project: Flink
>  Issue Type: Improvement
>  Components: Runtime / Checkpointing, Runtime / Coordination
>Reporter: ming li
>Assignee: ming li
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.15.0
>
>
> At present, we only recover the {{CompletedCheckpointStore}} when the 
> {{JobManager}} starts, so it seems that we do not need to re-register the 
> {{SharedStateRegistry}} when the task restarts.
> The reason for this issue is that in our production environment, we discard 
> part of the data and state to only restart the failed task, but found that it 
> may take several seconds to register the {{SharedStateRegistry}} (thousands 
> of tasks and dozens of TB states). When there are a large number of task 
> failures at the same time, this may take several minutes (number of tasks * 
> several seconds).
> Therefore, if the {{SharedStateRegistry}} can be reused, the time for task 
> recovery can be reduced.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] amazingJing commented on pull request #17233: FLINK-23949 adapt to Flink1.12

[GitBox]

amazingJing commented on pull request #17233:
URL: https://github.com/apache/flink/pull/17233#issuecomment-920540822


   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17205: [FLINK-24168][table-planner] Update MATCH_ROWTIME function which could receive 0 argument or 1 argument

[GitBox]

flinkbot edited a comment on pull request #17205:
URL: https://github.com/apache/flink/pull/17205#issuecomment-915449441


   
   ## CI report:
   
   * 170096fc6b1f13b86f5ceecfad89f8bf61b16181 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24123)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] Myasuka commented on a change in pull request #17203: [FLINK-22944][state] Optimize writing state changelog

[GitBox]

Myasuka commented on a change in pull request #17203:
URL: https://github.com/apache/flink/pull/17203#discussion_r709711663



##
File path: 
flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##
@@ -190,6 +194,19 @@
 
 private final ExecutorService asyncOperationsThreadPool;
 
+/**
+ * Identifies this backend. On restore, when snapshots of multiple 
backends are present
+ * (downscaling case) this allows to differentiate between state mappings 
of different backends.

Review comment:
   Actually, upscaling case could also cause this case (such as change 
parallelism 2 to 3).

##
File path: 
flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##
@@ -190,6 +194,19 @@
 
 private final ExecutorService asyncOperationsThreadPool;
 
+/**
+ * Identifies this backend. On restore, when snapshots of multiple 
backends are present
+ * (downscaling case) this allows to differentiate between state mappings 
of different backends.

Review comment:
   Actually, upscaling case could also cause this case (such as change 
parallelism from 2 to 3).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-24305) BatchPandasUDAFITTests.test_over_window_aggregate_function fails on azure

[Xintong Song (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-24305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415837#comment-17415837
 ] 

Xintong Song commented on FLINK-24305:
--

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24172=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=8d78fe4f-d658-5c70-12f8-4921589024c3=20571

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24172=logs=821b528f-1eed-5598-a3b4-7f748b13f261=4fad9527-b9a5-5015-1b70-8356e5c91490=22748

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24172=logs=bdd9ea51-4de2-506a-d4d9-f3930e4d2355=98717c4f-b888-5636-bb1e-db7aca25755e=21566

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24172=logs=bf5e383b-9fd3-5f02-ca1c-8f788e2e76d3=f5211ead-5e53-5af8-f827-4dbf08df26bb=21659

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24172=logs=ff2e2ea5-07e3-5521-7b04-a4fc3ad765e9=613f538c-bcef-59e6-f9cd-9714bec9fb97=22617

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24172=logs=e92ecf6d-e207-5a42-7ff7-528ff0c5b259=d59eb898-29f7-5a99-91a7-b2dfc3e8a653=21726

> BatchPandasUDAFITTests.test_over_window_aggregate_function fails on azure
> -
>
> Key: FLINK-24305
> URL: https://issues.apache.org/jira/browse/FLINK-24305
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python
>Affects Versions: 1.12.5, 1.15.0
>Reporter: Xintong Song
>Priority: Blocker
>  Labels: test-stability
> Fix For: 1.12.6, 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24170=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=c67e71ed-6451-5d26-8920-5a8cf9651901=23011
> {code}
> Sep 15 20:40:43 cls =  'pyflink.table.tests.test_pandas_udaf.BatchPandasUDAFITTests'>
> Sep 15 20:40:43 actual = JavaObject id=o8666
> Sep 15 20:40:43 expected = ['+I[1, 4.335, 13, 5.5, 3.0, 3.0, 4.335, 
> 8.0, 5.0, 5.0]', '+I[1, 4.335, 5, 4.335, 3.0, 3.0, 2.5, 4.3330, 
> 4.0, 2.0]', '+I[2, 2.0, 9, 2.0, 4.0, 4.0, 2.0, 2.0, 4.0, 4.0]', '+I[3, 2.0, 
> 3, 2.0, 1.0, 1.0, 2.0, 2.0, 1.0, 1.0]']
> Sep 15 20:40:43 
> Sep 15 20:40:43 @classmethod
> Sep 15 20:40:43 def assert_equals(cls, actual, expected):
> Sep 15 20:40:43 if isinstance(actual, JavaObject):
> Sep 15 20:40:43 actual_py_list = cls.to_py_list(actual)
> Sep 15 20:40:43 else:
> Sep 15 20:40:43 actual_py_list = actual
> Sep 15 20:40:43 actual_py_list.sort()
> Sep 15 20:40:43 expected.sort()
> Sep 15 20:40:43 assert len(actual_py_list) == len(expected)
> Sep 15 20:40:43 >   assert all(x == y for x, y in zip(actual_py_list, 
> expected))
> Sep 15 20:40:43 E   AssertionError: assert False
> Sep 15 20:40:43 E+  where False = all( PyFlinkTestCase.assert_equals.. at 0x7f792d98b900>)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-24305) BatchPandasUDAFITTests.test_over_window_aggregate_function fails on azure

[Xintong Song (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-24305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xintong Song updated FLINK-24305:
-
Affects Version/s: 1.12.5

> BatchPandasUDAFITTests.test_over_window_aggregate_function fails on azure
> -
>
> Key: FLINK-24305
> URL: https://issues.apache.org/jira/browse/FLINK-24305
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python
>Affects Versions: 1.12.5, 1.15.0
>Reporter: Xintong Song
>Priority: Blocker
>  Labels: test-stability
> Fix For: 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24170=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=c67e71ed-6451-5d26-8920-5a8cf9651901=23011
> {code}
> Sep 15 20:40:43 cls =  'pyflink.table.tests.test_pandas_udaf.BatchPandasUDAFITTests'>
> Sep 15 20:40:43 actual = JavaObject id=o8666
> Sep 15 20:40:43 expected = ['+I[1, 4.335, 13, 5.5, 3.0, 3.0, 4.335, 
> 8.0, 5.0, 5.0]', '+I[1, 4.335, 5, 4.335, 3.0, 3.0, 2.5, 4.3330, 
> 4.0, 2.0]', '+I[2, 2.0, 9, 2.0, 4.0, 4.0, 2.0, 2.0, 4.0, 4.0]', '+I[3, 2.0, 
> 3, 2.0, 1.0, 1.0, 2.0, 2.0, 1.0, 1.0]']
> Sep 15 20:40:43 
> Sep 15 20:40:43 @classmethod
> Sep 15 20:40:43 def assert_equals(cls, actual, expected):
> Sep 15 20:40:43 if isinstance(actual, JavaObject):
> Sep 15 20:40:43 actual_py_list = cls.to_py_list(actual)
> Sep 15 20:40:43 else:
> Sep 15 20:40:43 actual_py_list = actual
> Sep 15 20:40:43 actual_py_list.sort()
> Sep 15 20:40:43 expected.sort()
> Sep 15 20:40:43 assert len(actual_py_list) == len(expected)
> Sep 15 20:40:43 >   assert all(x == y for x, y in zip(actual_py_list, 
> expected))
> Sep 15 20:40:43 E   AssertionError: assert False
> Sep 15 20:40:43 E+  where False = all( PyFlinkTestCase.assert_equals.. at 0x7f792d98b900>)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-24305) BatchPandasUDAFITTests.test_over_window_aggregate_function fails on azure

[Xintong Song (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-24305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xintong Song updated FLINK-24305:
-
Fix Version/s: 1.12.6

> BatchPandasUDAFITTests.test_over_window_aggregate_function fails on azure
> -
>
> Key: FLINK-24305
> URL: https://issues.apache.org/jira/browse/FLINK-24305
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python
>Affects Versions: 1.12.5, 1.15.0
>Reporter: Xintong Song
>Priority: Blocker
>  Labels: test-stability
> Fix For: 1.12.6, 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24170=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=c67e71ed-6451-5d26-8920-5a8cf9651901=23011
> {code}
> Sep 15 20:40:43 cls =  'pyflink.table.tests.test_pandas_udaf.BatchPandasUDAFITTests'>
> Sep 15 20:40:43 actual = JavaObject id=o8666
> Sep 15 20:40:43 expected = ['+I[1, 4.335, 13, 5.5, 3.0, 3.0, 4.335, 
> 8.0, 5.0, 5.0]', '+I[1, 4.335, 5, 4.335, 3.0, 3.0, 2.5, 4.3330, 
> 4.0, 2.0]', '+I[2, 2.0, 9, 2.0, 4.0, 4.0, 2.0, 2.0, 4.0, 4.0]', '+I[3, 2.0, 
> 3, 2.0, 1.0, 1.0, 2.0, 2.0, 1.0, 1.0]']
> Sep 15 20:40:43 
> Sep 15 20:40:43 @classmethod
> Sep 15 20:40:43 def assert_equals(cls, actual, expected):
> Sep 15 20:40:43 if isinstance(actual, JavaObject):
> Sep 15 20:40:43 actual_py_list = cls.to_py_list(actual)
> Sep 15 20:40:43 else:
> Sep 15 20:40:43 actual_py_list = actual
> Sep 15 20:40:43 actual_py_list.sort()
> Sep 15 20:40:43 expected.sort()
> Sep 15 20:40:43 assert len(actual_py_list) == len(expected)
> Sep 15 20:40:43 >   assert all(x == y for x, y in zip(actual_py_list, 
> expected))
> Sep 15 20:40:43 E   AssertionError: assert False
> Sep 15 20:40:43 E+  where False = all( PyFlinkTestCase.assert_equals.. at 0x7f792d98b900>)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-24300) MultipleInputOperator is running much more slowly in TPCDS

[Zhilong Hong (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-24300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhilong Hong updated FLINK-24300:
-
Description: 
When we are running TPCDS with release 1.14 we find that the job with 
{{MultipleInputOperator}} is running much more slowly than before. With a 
binary search among the commits, we find that the issue may be introduced by 
FLINK-23408. 

At the commit 64570e4c56955713ca599fd1d7ae7be891a314c6, the job in TPCDS runs 
normally, as the image below illustrates:

!64570e4c56955713ca599fd1d7ae7be891a314c6.png|width=600!

At the commit e3010c16947ed8da2ecb7d89a3aa08dacecc524a, the job q2.sql gets 
stuck for a pretty long time (longer than half an hour), as the image below 
illustrates:

!e3010c16947ed8da2ecb7d89a3aa08dacecc524a.png|width=600!

The detail of the job is illustrated below:

!detail-of-the-job.png|width=600!

The job uses a {{MultipleInputOperator}} with one normal input and two chained 
FileSource. It has finished reading the normal input and start to read the 
chained source. Each chained source has one source data fetcher.

We capture the jstack of the stuck tasks and attach the file below. From the 
[^jstack.txt] we can see the main thread is blocked on waiting for the lock, 
and the lock is held by a source data fetcher. The source data fetcher is still 
running but the stack keeps on {{CompletableFuture.cleanStack}}.

This issue happens in a batch job. However, from where it get blocked, it seems 
also affects the streaming jobs.

For the reference, the code of TPCDS we are running is located at 
[https://github.com/ververica/flink-sql-benchmark/tree/dev].

  was:
When we are running TPCDS with release 1.14 we find that the job with 
MultipleInputOperator is running much more slowly than before. With a binary 
search among the commits, we find that the issue may be introduced by 
FLINK-23408. 

At the commit 64570e4c56955713ca599fd1d7ae7be891a314c6, the job runs normally 
in TPCDS, as the image below illustrates:

!64570e4c56955713ca599fd1d7ae7be891a314c6.png|width=600!

At the commit e3010c16947ed8da2ecb7d89a3aa08dacecc524a, the job q2.sql gets 
stuck for a pretty long time (longer than half an hour), as the image below 
illustrates:

!e3010c16947ed8da2ecb7d89a3aa08dacecc524a.png|width=600!

The detail of the job is illustrated below:

!detail-of-the-job.png|width=600!

The job uses a {{MultipleInputOperator}} with one normal input and two chained 
FileSource. It has finished reading the normal input and start to read the 
chained source. Each chained source has one source data fetcher.

We capture the jstack of the stuck tasks and attach the file below. From the 
[^jstack.txt] we can see the main thread is blocked on waiting for the lock, 
and the lock is held by a source data fetcher. The source data fetcher is still 
running but the stack keeps on {{CompletableFuture.cleanStack}}.

This issue happens in a batch job. However, from where it get blocked, it seems 
also affects the streaming jobs.

For the reference, the code of TPCDS we are running is located at 
[https://github.com/ververica/flink-sql-benchmark/tree/dev].


> MultipleInputOperator is running much more slowly in TPCDS
> --
>
> Key: FLINK-24300
> URL: https://issues.apache.org/jira/browse/FLINK-24300
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / Network
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Zhilong Hong
>Priority: Major
> Attachments: 64570e4c56955713ca599fd1d7ae7be891a314c6.png, 
> detail-of-the-job.png, e3010c16947ed8da2ecb7d89a3aa08dacecc524a.png, 
> jstack-2.txt, jstack.txt
>
>
> When we are running TPCDS with release 1.14 we find that the job with 
> {{MultipleInputOperator}} is running much more slowly than before. With a 
> binary search among the commits, we find that the issue may be introduced by 
> FLINK-23408. 
> At the commit 64570e4c56955713ca599fd1d7ae7be891a314c6, the job in TPCDS runs 
> normally, as the image below illustrates:
> !64570e4c56955713ca599fd1d7ae7be891a314c6.png|width=600!
> At the commit e3010c16947ed8da2ecb7d89a3aa08dacecc524a, the job q2.sql gets 
> stuck for a pretty long time (longer than half an hour), as the image below 
> illustrates:
> !e3010c16947ed8da2ecb7d89a3aa08dacecc524a.png|width=600!
> The detail of the job is illustrated below:
> !detail-of-the-job.png|width=600!
> The job uses a {{MultipleInputOperator}} with one normal input and two 
> chained FileSource. It has finished reading the normal input and start to 
> read the chained source. Each chained source has one source data fetcher.
> We capture the jstack of the stuck tasks and attach the file below. From the 
> [^jstack.txt] we can see the main thread is blocked on waiting for the lock, 
> and the lock is held by a source data fetcher. The 

[jira] [Updated] (FLINK-24293) Tasks from the same job on a machine share user jar

[huntercc (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-24293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

huntercc updated FLINK-24293:
-
Description: 
In the current blob storage design, tasks executed by the same TaskExecutor 
will share BLOBs storage dir and tasks executed by different TaskExecutor use 
different dir. As a result, a TaskExecutor has to download user jar even if 
there has been the same user jar downloaded by other TaskExecutors on the 
machine. We believe that there is no need to download many copies of the same 
user jar to the local, two main problems will by exposed:
 # The NIC bandwidth of the distribution terminal may become a bottleneck  
!image-2021-09-15-20-43-17-304.png|width=695,height=193! 
As shown in the figure above, 24640 Mbps of the total 25000 Mbps NIC bandwidth 
is used when we launched a flink job with 4000 TaskManagers, which will cause a 
long deployment time and akka timeout exception.
 # Take up more disk space

We expect to optimize the sharing mechanism of user jar by allowing tasks from 
the same job on a machine to share blob storage dir, more specifically, share 
the user jar in the dir. Only one task deployed to the machine will download 
the user jar from BLOB server or distributed file storage, and the subsequent 
tasks just use the localized user jar. In this way, the user jar of one job 
only needs to be downloaded once on a machine. Here is a comparison of job 
startup time before and after optimization.
||num of TM||before optimization||after optimization||
|1000|62s|37s|
|2000|104s|40s|
|3000|170s|43s|
|4000|211s|45s|

 

  was:
In the current blob storage design, tasks executed by the same TaskExecutor 
will share BLOBs storage dir and tasks executed by different TaskExecutor use 
different dir. As a result, a TaskExecutor has to download user jar even if 
there has been the same user jar downloaded by other TaskExecutors on the 
machine. We believe that there is no need to download many copies of the same 
user jar to the local, two main problems will by exposed:
 # The NIC bandwidth of the distribution terminal may become a bottlenec 
!image-2021-09-15-20-43-17-304.png|width=695,height=193! As shown in the figure 
above, 24640 Mbps of the total 25000 Mbps NIC bandwidth is used when we 
launched a flink job with 4000 TaskManagers, which will cause a long deployment 
time and akka timeout exception.
 # Take up more disk space

We expect to optimize the sharing mechanism of user jar by allowing tasks from 
the same job on a machine to share blob storage dir, more specifically, share 
the user jar in the dir. Only one task deployed to the machine will download 
the user jar from BLOB server or distributed file storage, and the subsequent 
tasks just use the localized user jar. In this way, the user jar of one job 
only needs to be downloaded once on a machine. Here is a comparison of job 
startup time before and after optimization.
||num of TM||before optimization||after optimization||
|1000|62s|37s|
|2000|104s|40s|
|3000|170s|43s|
|4000|211s|45s|

 


> Tasks from the same job on a machine share user jar 
> 
>
> Key: FLINK-24293
> URL: https://issues.apache.org/jira/browse/FLINK-24293
> Project: Flink
>  Issue Type: Improvement
>  Components: Runtime / Coordination
>Reporter: huntercc
>Priority: Major
>  Labels: pull-request-available
> Attachments: image-2021-09-15-20-43-11-758.png, 
> image-2021-09-15-20-43-17-304.png
>
>
> In the current blob storage design, tasks executed by the same TaskExecutor 
> will share BLOBs storage dir and tasks executed by different TaskExecutor use 
> different dir. As a result, a TaskExecutor has to download user jar even if 
> there has been the same user jar downloaded by other TaskExecutors on the 
> machine. We believe that there is no need to download many copies of the same 
> user jar to the local, two main problems will by exposed:
>  # The NIC bandwidth of the distribution terminal may become a bottleneck  
> !image-2021-09-15-20-43-17-304.png|width=695,height=193! 
> As shown in the figure above, 24640 Mbps of the total 25000 Mbps NIC 
> bandwidth is used when we launched a flink job with 4000 TaskManagers, which 
> will cause a long deployment time and akka timeout exception.
>  # Take up more disk space
> We expect to optimize the sharing mechanism of user jar by allowing tasks 
> from the same job on a machine to share blob storage dir, more specifically, 
> share the user jar in the dir. Only one task deployed to the machine will 
> download the user jar from BLOB server or distributed file storage, and the 
> subsequent tasks just use the localized user jar. In this way, the user jar 
> of one job only needs to be downloaded once on a machine. Here is a 
> comparison of job startup time before 

[jira] [Commented] (FLINK-24305) BatchPandasUDAFITTests.test_over_window_aggregate_function fails on azure

[Xintong Song (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-24305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415836#comment-17415836
 ] 

Xintong Song commented on FLINK-24305:
--

cc [~dian.fu] [~hxbks2ks]

> BatchPandasUDAFITTests.test_over_window_aggregate_function fails on azure
> -
>
> Key: FLINK-24305
> URL: https://issues.apache.org/jira/browse/FLINK-24305
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python
>Affects Versions: 1.15.0
>Reporter: Xintong Song
>Priority: Blocker
>  Labels: test-stability
> Fix For: 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24170=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=c67e71ed-6451-5d26-8920-5a8cf9651901=23011
> {code}
> Sep 15 20:40:43 cls =  'pyflink.table.tests.test_pandas_udaf.BatchPandasUDAFITTests'>
> Sep 15 20:40:43 actual = JavaObject id=o8666
> Sep 15 20:40:43 expected = ['+I[1, 4.335, 13, 5.5, 3.0, 3.0, 4.335, 
> 8.0, 5.0, 5.0]', '+I[1, 4.335, 5, 4.335, 3.0, 3.0, 2.5, 4.3330, 
> 4.0, 2.0]', '+I[2, 2.0, 9, 2.0, 4.0, 4.0, 2.0, 2.0, 4.0, 4.0]', '+I[3, 2.0, 
> 3, 2.0, 1.0, 1.0, 2.0, 2.0, 1.0, 1.0]']
> Sep 15 20:40:43 
> Sep 15 20:40:43 @classmethod
> Sep 15 20:40:43 def assert_equals(cls, actual, expected):
> Sep 15 20:40:43 if isinstance(actual, JavaObject):
> Sep 15 20:40:43 actual_py_list = cls.to_py_list(actual)
> Sep 15 20:40:43 else:
> Sep 15 20:40:43 actual_py_list = actual
> Sep 15 20:40:43 actual_py_list.sort()
> Sep 15 20:40:43 expected.sort()
> Sep 15 20:40:43 assert len(actual_py_list) == len(expected)
> Sep 15 20:40:43 >   assert all(x == y for x, y in zip(actual_py_list, 
> expected))
> Sep 15 20:40:43 E   AssertionError: assert False
> Sep 15 20:40:43 E+  where False = all( PyFlinkTestCase.assert_equals.. at 0x7f792d98b900>)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-24293) Tasks from the same job on a machine share user jar

[huntercc (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-24293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

huntercc updated FLINK-24293:
-
Description: 
In the current blob storage design, tasks executed by the same TaskExecutor 
will share BLOBs storage dir and tasks executed by different TaskExecutor use 
different dir. As a result, a TaskExecutor has to download user jar even if 
there has been the same user jar downloaded by other TaskExecutors on the 
machine. We believe that there is no need to download many copies of the same 
user jar to the local, two main problems will by exposed:
 # The NIC bandwidth of the distribution terminal may become a bottlenec 
!image-2021-09-15-20-43-17-304.png|width=695,height=193! As shown in the figure 
above, 24640 Mbps of the total 25000 Mbps NIC bandwidth is used when we 
launched a flink job with 4000 TaskManagers, which will cause a long deployment 
time and akka timeout exception.
 # Take up more disk space

We expect to optimize the sharing mechanism of user jar by allowing tasks from 
the same job on a machine to share blob storage dir, more specifically, share 
the user jar in the dir. Only one task deployed to the machine will download 
the user jar from BLOB server or distributed file storage, and the subsequent 
tasks just use the localized user jar. In this way, the user jar of one job 
only needs to be downloaded once on a machine. Here is a comparison of job 
startup time before and after optimization.
||num of TM||before optimization||after optimization||
|1000|62s|37s|
|2000|104s|40s|
|3000|170s|43s|
|4000|211s|45s|

 

  was:
In the current blob storage design, tasks executed by the same TaskExecutor 
will share BLOBs storage dir and tasks executed by different TaskExecutor use 
different dir. As a result, a TaskExecutor has to download user jar even if 
there has been the same user jar downloaded by other TaskExecutors on the 
machine. We believe that there is no need to download many copies of the same 
user jar to the local, two main problems will by exposed:
 # The NIC bandwidth of the distribution terminal may become a bottleneck    
!image-2021-09-15-20-43-17-304.png|width=695,height=193! As shown in the figure 
above, 24640 Mbps of the total 25000 Mbps NIC bandwidth is used when we 
launched a flink job with 4000 TaskManagers, which will cause a long deployment 
time and akka timeout exception.
 # Take up more disk space

We expect to optimize the sharing mechanism of user jar by allowing tasks from 
the same job on a machine to share blob storage dir, more specifically, share 
the user jar in the dir. Only one task deployed to the machine will download 
the user jar from BLOB server or distributed file storage, and the subsequent 
tasks just use the localized user jar. In this way, the user jar of one job 
only needs to be downloaded once on a machine. Here is a comparison of job 
startup time before and after optimization.
||num of TM||before optimization||after optimization||
|1000|62s|37s|
|2000|104s|40s|
|3000|170s|43s|
|4000|211s|45s|

 


> Tasks from the same job on a machine share user jar 
> 
>
> Key: FLINK-24293
> URL: https://issues.apache.org/jira/browse/FLINK-24293
> Project: Flink
>  Issue Type: Improvement
>  Components: Runtime / Coordination
>Reporter: huntercc
>Priority: Major
>  Labels: pull-request-available
> Attachments: image-2021-09-15-20-43-11-758.png, 
> image-2021-09-15-20-43-17-304.png
>
>
> In the current blob storage design, tasks executed by the same TaskExecutor 
> will share BLOBs storage dir and tasks executed by different TaskExecutor use 
> different dir. As a result, a TaskExecutor has to download user jar even if 
> there has been the same user jar downloaded by other TaskExecutors on the 
> machine. We believe that there is no need to download many copies of the same 
> user jar to the local, two main problems will by exposed:
>  # The NIC bandwidth of the distribution terminal may become a bottlenec 
> !image-2021-09-15-20-43-17-304.png|width=695,height=193! As shown in the 
> figure above, 24640 Mbps of the total 25000 Mbps NIC bandwidth is used when 
> we launched a flink job with 4000 TaskManagers, which will cause a long 
> deployment time and akka timeout exception.
>  # Take up more disk space
> We expect to optimize the sharing mechanism of user jar by allowing tasks 
> from the same job on a machine to share blob storage dir, more specifically, 
> share the user jar in the dir. Only one task deployed to the machine will 
> download the user jar from BLOB server or distributed file storage, and the 
> subsequent tasks just use the localized user jar. In this way, the user jar 
> of one job only needs to be downloaded once on a machine. Here is a 
> comparison of job startup time before 

[jira] [Commented] (FLINK-24305) BatchPandasUDAFITTests.test_over_window_aggregate_function fails on azure

[Xintong Song (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-24305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415835#comment-17415835
 ] 

Xintong Song commented on FLINK-24305:
--

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24171=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=c67e71ed-6451-5d26-8920-5a8cf9651901=26255

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24171=logs=821b528f-1eed-5598-a3b4-7f748b13f261=6bb545dd-772d-5d8c-f258-f5085fba3295=25944

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24171=logs=bdd9ea51-4de2-506a-d4d9-f3930e4d2355=dd50312f-73b5-56b5-c172-4d81d03e2ef1=26108

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24171=logs=bf5e383b-9fd3-5f02-ca1c-8f788e2e76d3=85189c57-d8a0-5c9c-b61d-fc05cfac62cf=26510

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24171=logs=ff2e2ea5-07e3-5521-7b04-a4fc3ad765e9=1ec6382b-bafe-5817-63ae-eda7d4be718e=25486

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24171=logs=e92ecf6d-e207-5a42-7ff7-528ff0c5b259=40fc352e-9b4c-5fd8-363f-628f24b01ec2=24576

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24171=logs=3e4dd1a2-fe2f-5e5d-a581-48087e718d53=b4612f28-e3b5-5853-8a8b-610ae894217a=22622

> BatchPandasUDAFITTests.test_over_window_aggregate_function fails on azure
> -
>
> Key: FLINK-24305
> URL: https://issues.apache.org/jira/browse/FLINK-24305
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python
>Affects Versions: 1.15.0
>Reporter: Xintong Song
>Priority: Blocker
>  Labels: test-stability
> Fix For: 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24170=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=c67e71ed-6451-5d26-8920-5a8cf9651901=23011
> {code}
> Sep 15 20:40:43 cls =  'pyflink.table.tests.test_pandas_udaf.BatchPandasUDAFITTests'>
> Sep 15 20:40:43 actual = JavaObject id=o8666
> Sep 15 20:40:43 expected = ['+I[1, 4.335, 13, 5.5, 3.0, 3.0, 4.335, 
> 8.0, 5.0, 5.0]', '+I[1, 4.335, 5, 4.335, 3.0, 3.0, 2.5, 4.3330, 
> 4.0, 2.0]', '+I[2, 2.0, 9, 2.0, 4.0, 4.0, 2.0, 2.0, 4.0, 4.0]', '+I[3, 2.0, 
> 3, 2.0, 1.0, 1.0, 2.0, 2.0, 1.0, 1.0]']
> Sep 15 20:40:43 
> Sep 15 20:40:43 @classmethod
> Sep 15 20:40:43 def assert_equals(cls, actual, expected):
> Sep 15 20:40:43 if isinstance(actual, JavaObject):
> Sep 15 20:40:43 actual_py_list = cls.to_py_list(actual)
> Sep 15 20:40:43 else:
> Sep 15 20:40:43 actual_py_list = actual
> Sep 15 20:40:43 actual_py_list.sort()
> Sep 15 20:40:43 expected.sort()
> Sep 15 20:40:43 assert len(actual_py_list) == len(expected)
> Sep 15 20:40:43 >   assert all(x == y for x, y in zip(actual_py_list, 
> expected))
> Sep 15 20:40:43 E   AssertionError: assert False
> Sep 15 20:40:43 E+  where False = all( PyFlinkTestCase.assert_equals.. at 0x7f792d98b900>)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot edited a comment on pull request #17205: [FLINK-24168][table-planner] Update MATCH_ROWTIME function which could receive 0 argument or 1 argument

[GitBox]

flinkbot edited a comment on pull request #17205:
URL: https://github.com/apache/flink/pull/17205#issuecomment-915449441


   
   ## CI report:
   
   * 170096fc6b1f13b86f5ceecfad89f8bf61b16181 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24123)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (FLINK-24305) BatchPandasUDAFITTests.test_over_window_aggregate_function fails on azure

[Xintong Song (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-24305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xintong Song updated FLINK-24305:
-
Priority: Blocker  (was: Major)

> BatchPandasUDAFITTests.test_over_window_aggregate_function fails on azure
> -
>
> Key: FLINK-24305
> URL: https://issues.apache.org/jira/browse/FLINK-24305
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python
>Affects Versions: 1.15.0
>Reporter: Xintong Song
>Priority: Blocker
>  Labels: test-stability
> Fix For: 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24170=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=c67e71ed-6451-5d26-8920-5a8cf9651901=23011
> {code}
> Sep 15 20:40:43 cls =  'pyflink.table.tests.test_pandas_udaf.BatchPandasUDAFITTests'>
> Sep 15 20:40:43 actual = JavaObject id=o8666
> Sep 15 20:40:43 expected = ['+I[1, 4.335, 13, 5.5, 3.0, 3.0, 4.335, 
> 8.0, 5.0, 5.0]', '+I[1, 4.335, 5, 4.335, 3.0, 3.0, 2.5, 4.3330, 
> 4.0, 2.0]', '+I[2, 2.0, 9, 2.0, 4.0, 4.0, 2.0, 2.0, 4.0, 4.0]', '+I[3, 2.0, 
> 3, 2.0, 1.0, 1.0, 2.0, 2.0, 1.0, 1.0]']
> Sep 15 20:40:43 
> Sep 15 20:40:43 @classmethod
> Sep 15 20:40:43 def assert_equals(cls, actual, expected):
> Sep 15 20:40:43 if isinstance(actual, JavaObject):
> Sep 15 20:40:43 actual_py_list = cls.to_py_list(actual)
> Sep 15 20:40:43 else:
> Sep 15 20:40:43 actual_py_list = actual
> Sep 15 20:40:43 actual_py_list.sort()
> Sep 15 20:40:43 expected.sort()
> Sep 15 20:40:43 assert len(actual_py_list) == len(expected)
> Sep 15 20:40:43 >   assert all(x == y for x, y in zip(actual_py_list, 
> expected))
> Sep 15 20:40:43 E   AssertionError: assert False
> Sep 15 20:40:43 E+  where False = all( PyFlinkTestCase.assert_equals.. at 0x7f792d98b900>)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-23054) Correct upsert optimization by upsert keys

[Jingsong Lee (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-23054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jingsong Lee updated FLINK-23054:
-
Fix Version/s: 1.13.3

> Correct upsert optimization by upsert keys
> --
>
> Key: FLINK-23054
> URL: https://issues.apache.org/jira/browse/FLINK-23054
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table SQL / Runtime
>Reporter: Jingsong Lee
>Assignee: Jingsong Lee
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.14.0, 1.13.3
>
>
> After FLINK-22901.
> We can use upsert keys to fix upsert join, upsert rank, and upsert sink.
>  * For join and rank: if input has no upsert keys, do not use upsert 
> optimization.
>  * For upsert sink: if input has unique keys but no upsert keys, we need add 
> a materialize operator to produce upsert records.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Closed] (FLINK-23054) Correct upsert optimization by upsert keys

[Jingsong Lee (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-23054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jingsong Lee closed FLINK-23054.

Resolution: Fixed

release-1.13:
2c546e85e922e28f21b2972392622ea2ca30b5cd
68451e9268d32c6110bf5b844a4eee5dc4248066
bf4d675ef2a7a31413f00a58d710c4b74718457e
fe19395f62a98d2368e49753b109f0f02363066b

> Correct upsert optimization by upsert keys
> --
>
> Key: FLINK-23054
> URL: https://issues.apache.org/jira/browse/FLINK-23054
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table SQL / Runtime
>Reporter: Jingsong Lee
>Assignee: Jingsong Lee
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.14.0
>
>
> After FLINK-22901.
> We can use upsert keys to fix upsert join, upsert rank, and upsert sink.
>  * For join and rank: if input has no upsert keys, do not use upsert 
> optimization.
>  * For upsert sink: if input has unique keys but no upsert keys, we need add 
> a materialize operator to produce upsert records.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-22899) ValuesUpsertSinkFunction needs to use global upsert

[Jingsong Lee (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-22899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415833#comment-17415833
 ] 

Jingsong Lee commented on FLINK-22899:
--

release-1.13: e17331e9636870d12599e66416b207d16e8e57cf

> ValuesUpsertSinkFunction needs to use global upsert
> ---
>
> Key: FLINK-22899
> URL: https://issues.apache.org/jira/browse/FLINK-22899
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table SQL / Runtime
>Reporter: Jingsong Lee
>Assignee: Jingsong Lee
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.14.0
>
>
> At present, each task does its own upsert. We need to simulate the external 
> connector and use the global map to do the upsert.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] JingsongLi merged pull request #17230: [FLINK-23054][table-blink] Correct upsert optimization by upsert keys

[GitBox]

JingsongLi merged pull request #17230:
URL: https://github.com/apache/flink/pull/17230


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] cshuo commented on pull request #17230: [FLINK-23054][table-blink] Correct upsert optimization by upsert keys

[GitBox]

cshuo commented on pull request #17230:
URL: https://github.com/apache/flink/pull/17230#issuecomment-920525293


   LGTM, +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (FLINK-24305) BatchPandasUDAFITTests.test_over_window_aggregate_function fails on azure

[Xintong Song (Jira)]

Xintong Song created FLINK-24305:


 Summary: 
BatchPandasUDAFITTests.test_over_window_aggregate_function fails on azure
 Key: FLINK-24305
 URL: https://issues.apache.org/jira/browse/FLINK-24305
 Project: Flink
  Issue Type: Bug
  Components: API / Python
Affects Versions: 1.15.0
Reporter: Xintong Song
 Fix For: 1.15.0


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24170=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=c67e71ed-6451-5d26-8920-5a8cf9651901=23011

{code}
Sep 15 20:40:43 cls = 
Sep 15 20:40:43 actual = JavaObject id=o8666
Sep 15 20:40:43 expected = ['+I[1, 4.335, 13, 5.5, 3.0, 3.0, 4.335, 
8.0, 5.0, 5.0]', '+I[1, 4.335, 5, 4.335, 3.0, 3.0, 2.5, 4.3330, 
4.0, 2.0]', '+I[2, 2.0, 9, 2.0, 4.0, 4.0, 2.0, 2.0, 4.0, 4.0]', '+I[3, 2.0, 3, 
2.0, 1.0, 1.0, 2.0, 2.0, 1.0, 1.0]']
Sep 15 20:40:43 
Sep 15 20:40:43 @classmethod
Sep 15 20:40:43 def assert_equals(cls, actual, expected):
Sep 15 20:40:43 if isinstance(actual, JavaObject):
Sep 15 20:40:43 actual_py_list = cls.to_py_list(actual)
Sep 15 20:40:43 else:
Sep 15 20:40:43 actual_py_list = actual
Sep 15 20:40:43 actual_py_list.sort()
Sep 15 20:40:43 expected.sort()
Sep 15 20:40:43 assert len(actual_py_list) == len(expected)
Sep 15 20:40:43 >   assert all(x == y for x, y in zip(actual_py_list, 
expected))
Sep 15 20:40:43 E   AssertionError: assert False
Sep 15 20:40:43 E+  where False = all(. at 0x7f792d98b900>)
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] JingsongLi commented on pull request #17186: [FLINK-16203][table] Support JSON_OBJECT()

[GitBox]

JingsongLi commented on pull request #17186:
URL: https://github.com/apache/flink/pull/17186#issuecomment-920524614


   @Airblader Do you want to squash or keep hotfixs with last `Support 
JSON_OBJECT` commit?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (FLINK-24304) ProcessFailureCancelingITCase.testCancelingOnProcessFailure fails due to leader retrieval timeout

[Xintong Song (Jira)]

Xintong Song created FLINK-24304:


 Summary: 
ProcessFailureCancelingITCase.testCancelingOnProcessFailure fails due to leader 
retrieval timeout
 Key: FLINK-24304
 URL: https://issues.apache.org/jira/browse/FLINK-24304
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.12.5
Reporter: Xintong Song
 Fix For: 1.12.6


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24145=logs=5c8e7682-d68f-54d1-16a2-a09310218a49=f508e270-48d6-5f1e-3138-42a17e0714f0=4153

{code}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 26.597 
s <<< FAILURE! - in org.apache.flink.test.recovery.ProcessFailureCancelingITCase
[ERROR] 
testCancelingOnProcessFailure(org.apache.flink.test.recovery.ProcessFailureCancelingITCase)
  Time elapsed: 25.988 s  <<< ERROR!
org.apache.flink.runtime.leaderretrieval.LeaderRetrievalException: Could not 
retrieve the leader address and leader session ID.
at 
org.apache.flink.runtime.util.LeaderRetrievalUtils.retrieveLeaderConnectionInfo(LeaderRetrievalUtils.java:63)
at 
org.apache.flink.test.recovery.ProcessFailureCancelingITCase.retrieveDispatcherGateway(ProcessFailureCancelingITCase.java:283)
at 
org.apache.flink.test.recovery.ProcessFailureCancelingITCase.testCancelingOnProcessFailure(ProcessFailureCancelingITCase.java:216)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at 
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: java.util.concurrent.TimeoutException
at 
java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1784)
at 
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
at 
org.apache.flink.runtime.util.LeaderRetrievalUtils.retrieveLeaderConnectionInfo(LeaderRetrievalUtils.java:61)
... 32 more
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] RocMarshal commented on pull request #16962: [FLINK-15352][connector-jdbc] Develop MySQLCatalog to connect Flink with MySQL tables and ecosystem.

[GitBox]

RocMarshal commented on pull request #16962:
URL: https://github.com/apache/flink/pull/16962#issuecomment-920520153


   Hi, @wuchong . I made some changes in the ITCase part of the PR.  Could you 
help me to review this PR if you have free time? Thank you.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] beyond1920 commented on pull request #17205: [FLINK-24168][table-planner] Update MATCH_ROWTIME function which could receive 0 argument or 1 argument

[GitBox]

beyond1920 commented on pull request #17205:
URL: https://github.com/apache/flink/pull/17205#issuecomment-920515148


   @flinkbot run travis


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #16798: [FLINK-23651][python] Support RabbitMQ in PyFlink

[GitBox]

flinkbot edited a comment on pull request #16798:
URL: https://github.com/apache/flink/pull/16798#issuecomment-897826601


   
   ## CI report:
   
   * 79547c7995e297da35a88c28517c2463c9c930c3 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23959)
 
   * a2de26625d24fb8c554a1f2d286652af75cb1234 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24176)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (FLINK-21765) Remove implementation-specific MetricGroup parents

[Flink Jira Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-21765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Flink Jira Bot updated FLINK-21765:
---
Labels: auto-deprioritized-major auto-unassigned stale-assigned starter  
(was: auto-deprioritized-major auto-unassigned starter)

I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help 
the community manage its development. I see this issue is assigned but has not 
received an update in 30 days, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a 
comment updating the community on your progress.  If this issue is waiting on 
feedback, please consider this a reminder to the committer/reviewer. Flink is a 
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone 
else may work on it.


> Remove implementation-specific MetricGroup parents
> --
>
> Key: FLINK-21765
> URL: https://issues.apache.org/jira/browse/FLINK-21765
> Project: Flink
>  Issue Type: Technical Debt
>  Components: Runtime / Metrics
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Minor
>  Labels: auto-deprioritized-major, auto-unassigned, 
> stale-assigned, starter
> Fix For: 1.14.0
>
>
> MetricGroups currently form a bi-directly graph, usually with explicit 
> requirements that type the parent must have. For example, an OperatorMG has a 
> hard requirement that the parent is a TaskMG.
> As a result they are quite inflexible, which particular shows in tests, as 
> you can't just create one metric group, but have to build an entire tree.
> The end goal of this ticket is to remove AbstractMetricGroup#parent, and 
> along the way we'll decouple the various MG implementations from each other.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot edited a comment on pull request #16798: [FLINK-23651][python] Support RabbitMQ in PyFlink

[GitBox]

flinkbot edited a comment on pull request #16798:
URL: https://github.com/apache/flink/pull/16798#issuecomment-897826601


   
   ## CI report:
   
   * 79547c7995e297da35a88c28517c2463c9c930c3 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23959)
 
   * a2de26625d24fb8c554a1f2d286652af75cb1234 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (FLINK-24303) SourceCoordinator exception may fail Session Cluster

[Seth Wiesman (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-24303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seth Wiesman updated FLINK-24303:
-
Component/s: Connectors / Common

> SourceCoordinator exception may fail Session Cluster
> 
>
> Key: FLINK-24303
> URL: https://issues.apache.org/jira/browse/FLINK-24303
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / Common
>Reporter: Seth Wiesman
>Priority: Major
>
> The SourceCoordinator currently forwards all exceptions from 
> `Source#createEnumerator` up the stack triggering a JobMaster failover. 
> However, JobMaster failover only works if HA is enabled[1]. If HA is not 
> enabled the fatal error handler will simply exit the JM process killing the 
> entire cluster. This is problematic in the case of a session cluster where 
> there may be multiple jobs running. It also does not play well with external 
> tooling that does not expect job failure to cause a full cluster failure. 
>  
> It would be preferable if failure to create an enumerator did not take down 
> the entire cluster, but instead failed that particular job. 
>  
> [1] 
> [https://github.com/apache/flink/blob/7f69331294ab2ab73f77b40a4320cdda53246afe/flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java#L898-L903]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] EelisK commented on a change in pull request #16798: [FLINK-23651][python] Support RabbitMQ in PyFlink

[GitBox]

EelisK commented on a change in pull request #16798:
URL: https://github.com/apache/flink/pull/16798#discussion_r709608571



##
File path: flink-connectors/flink-connector-rabbitmq/pom.xml
##
@@ -92,4 +92,31 @@ under the License.
 

 
+   
+   
+   
+   org.apache.maven.plugins
+   maven-shade-plugin
+   
+   
+   shade-flink
+   
+   
+   
+   
com.rabbitmq:amqp-client
+   
+   
+   
+   
+   
com.rabbitmq
+   
org.apache.flink.rabbitmq.shaded.com.rabbitmq
+   
+   
+   
+   
+   
+   
+   
+   
+

Review comment:
   @dianfu Ah, I see :+1: I addressed your comments in the latest change




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (FLINK-24303) SourceCoordinator exception may fail Session Cluster

[Seth Wiesman (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-24303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seth Wiesman updated FLINK-24303:
-
Description: 
The SourceCoordinator currently forwards all exceptions from 
`Source#createEnumerator` up the stack triggering a JobMaster failover. 
However, JobMaster failover only works if HA is enabled[1]. If HA is not 
enabled the fatal error handler will simply exit the JM process killing the 
entire cluster. This is problematic in the case of a session cluster where 
there may be multiple jobs running. It also does not play well with external 
tooling that does not expect job failure to cause a full cluster failure. 

 

It would be preferable if failure to create an enumerator did not take down the 
entire cluster, but instead failed that particular job. 

 

[1] 
[https://github.com/apache/flink/blob/7f69331294ab2ab73f77b40a4320cdda53246afe/flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java#L898-L903]

  was:
The SourceCoordinator currently forwards all exceptions from 
`Source#createEnumerator` up the stack triggering a JobMaster failover. 
However, JobMaster failover only works if HA is enabled[1]. If HA is not 
enabled the fatal error handler will simply exit the JM process killing the 
entire cluster. 

 

It would be preferable if failure to create an enumerator did not take down the 
entire cluster, but instead failed that particular job. 

 

[1] 
[https://github.com/apache/flink/blob/7f69331294ab2ab73f77b40a4320cdda53246afe/flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java#L898-L903]


> SourceCoordinator exception may fail Session Cluster
> 
>
> Key: FLINK-24303
> URL: https://issues.apache.org/jira/browse/FLINK-24303
> Project: Flink
>  Issue Type: Bug
>Reporter: Seth Wiesman
>Priority: Major
>
> The SourceCoordinator currently forwards all exceptions from 
> `Source#createEnumerator` up the stack triggering a JobMaster failover. 
> However, JobMaster failover only works if HA is enabled[1]. If HA is not 
> enabled the fatal error handler will simply exit the JM process killing the 
> entire cluster. This is problematic in the case of a session cluster where 
> there may be multiple jobs running. It also does not play well with external 
> tooling that does not expect job failure to cause a full cluster failure. 
>  
> It would be preferable if failure to create an enumerator did not take down 
> the entire cluster, but instead failed that particular job. 
>  
> [1] 
> [https://github.com/apache/flink/blob/7f69331294ab2ab73f77b40a4320cdda53246afe/flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java#L898-L903]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot edited a comment on pull request #17203: [FLINK-22944][state] Optimize writing state changelog

[GitBox]

flinkbot edited a comment on pull request #17203:
URL: https://github.com/apache/flink/pull/17203#issuecomment-915404842


   
   ## CI report:
   
   * a4683e8c521180f39b7714b9884cd71520dac1c6 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23795)
 
   * 4bd2434f31574c63e93fea17893e4a38dce60f0b Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24175)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17238: [FLINK-24233][runtime] Create view reader always on the first PartitionRequest even if request subpartition view fails in order to si

[GitBox]

flinkbot edited a comment on pull request #17238:
URL: https://github.com/apache/flink/pull/17238#issuecomment-916817365


   
   ## CI report:
   
   * 49ee064fd07710fbba08485c8037135f91ad110d Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24167)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17203: [FLINK-22944][state] Optimize writing state changelog

[GitBox]

flinkbot edited a comment on pull request #17203:
URL: https://github.com/apache/flink/pull/17203#issuecomment-915404842


   
   ## CI report:
   
   * a4683e8c521180f39b7714b9884cd71520dac1c6 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23795)
 
   * 4bd2434f31574c63e93fea17893e4a38dce60f0b UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17288: [FLINK-24292][connectors/kafka] Use KafkaSink in examples instead of …

[GitBox]

flinkbot edited a comment on pull request #17288:
URL: https://github.com/apache/flink/pull/17288#issuecomment-919945771


   
   ## CI report:
   
   * 7b4aa92287cf00aa2a2b0e82bf5069c868c52c8a Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24164)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17256: [FLINK-23385][table] Implement COALESCE function

[GitBox]

flinkbot edited a comment on pull request #17256:
URL: https://github.com/apache/flink/pull/17256#issuecomment-918040209


   
   ## CI report:
   
   * 25fb8010703524981f7934402ebe46b266dc551e UNKNOWN
   * 521bf3c8656aaa4d7c9fc4b213c3e47e0ba95f03 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24162)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17204: [FLINK-22942] [sql/planner] Disable UPSERT INTO statement

[GitBox]

flinkbot edited a comment on pull request #17204:
URL: https://github.com/apache/flink/pull/17204#issuecomment-915424597


   
   ## CI report:
   
   * 08737f676be9cb8dd784258b17eb6a92e4b1b715 UNKNOWN
   * 56a3348b9dd0289aec214280cb304284706b543d UNKNOWN
   * b76a8ae926dd135e2ff81a894b48f83ab31b642d Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24166)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] rkhachatryan commented on pull request #17203: [FLINK-22944][state] Optimize writing state changelog

[GitBox]

rkhachatryan commented on pull request #17203:
URL: https://github.com/apache/flink/pull/17203#issuecomment-920364763


   Regarding the tests, there are already in `ChangelogDelegateHashMapTest` and 
similar, e.g. `testKeyGroupSnapshotRestoreScaleDown`, 
`testKeyGroupSnapshotRestoreScaleUp`, 
`testKeyGroupsSnapshotRestoreScaleUpUnEvenDistribute`, etc.
   which seems enough, WDYT?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (FLINK-24303) SourceCoordinator exception may fail Session Cluster

[Seth Wiesman (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-24303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seth Wiesman updated FLINK-24303:
-
Description: 
The SourceCoordinator currently forwards all exceptions from 
`Source#createEnumerator` up the stack triggering a JobMaster failover. 
However, JobMaster failover only works if HA is enabled[1]. If HA is not 
enabled the fatal error handler will simply exit the JM process killing the 
entire cluster. 

 

It would be preferable if failure to create an enumerator did not take down the 
entire cluster, but instead failed that particular job. 

 

[1] 
[https://github.com/apache/flink/blob/7f69331294ab2ab73f77b40a4320cdda53246afe/flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java#L898-L903]

  was:
The SourceCoordinator currently forwards all exceptions from 
`Source#createEnumerator` up the stack triggering a JobMaster failover. 
However, JobMaster failover only works if HA is enabled[1]. If not, the fatal 
error handler will simply exit the JM process killing the entire cluster. 

 

It would be preferable if failure to create an enumerator did not take down the 
entire cluster, but instead failed that particular job. 

 

[1] 
https://github.com/apache/flink/blob/7f69331294ab2ab73f77b40a4320cdda53246afe/flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java#L898-L903


> SourceCoordinator exception may fail Session Cluster
> 
>
> Key: FLINK-24303
> URL: https://issues.apache.org/jira/browse/FLINK-24303
> Project: Flink
>  Issue Type: Bug
>Reporter: Seth Wiesman
>Priority: Major
>
> The SourceCoordinator currently forwards all exceptions from 
> `Source#createEnumerator` up the stack triggering a JobMaster failover. 
> However, JobMaster failover only works if HA is enabled[1]. If HA is not 
> enabled the fatal error handler will simply exit the JM process killing the 
> entire cluster. 
>  
> It would be preferable if failure to create an enumerator did not take down 
> the entire cluster, but instead failed that particular job. 
>  
> [1] 
> [https://github.com/apache/flink/blob/7f69331294ab2ab73f77b40a4320cdda53246afe/flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java#L898-L903]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-24303) SourceCoordinator exception may fail Session Cluster

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-24303:


 Summary: SourceCoordinator exception may fail Session Cluster
 Key: FLINK-24303
 URL: https://issues.apache.org/jira/browse/FLINK-24303
 Project: Flink
  Issue Type: Bug
Reporter: Seth Wiesman


The SourceCoordinator currently forwards all exceptions from 
`Source#createEnumerator` up the stack triggering a JobMaster failover. 
However, JobMaster failover only works if HA is enabled[1]. If not, the fatal 
error handler will simply exit the JM process killing the entire cluster. 

 

It would be preferable if failure to create an enumerator did not take down the 
entire cluster, but instead failed that particular job. 

 

[1] 
https://github.com/apache/flink/blob/7f69331294ab2ab73f77b40a4320cdda53246afe/flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java#L898-L903



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot edited a comment on pull request #17268: [FLINK-24133][core] Network failure test replaced by comment in the code due to high expense of stabilizing such low important test

[GitBox]

flinkbot edited a comment on pull request #17268:
URL: https://github.com/apache/flink/pull/17268#issuecomment-918320586


   
   ## CI report:
   
   * 830515768c30f19fe454fc244ae1d4ba79c50cf2 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24168)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Comment Edited] (FLINK-24286) Flink TableEnvironment executesql on IntelliJ leads to Could not find a file system implementation for scheme 's3a'

[James Kim (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-24286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415710#comment-17415710
 ] 

James Kim edited comment on FLINK-24286 at 9/15/21, 8:08 PM:
-

[~jark] I've created a thread on the mailing list on this issue 
(https://lists.apache.org/thread.html/r254822aa27f8da80b65c0c6757e776f2835d4429dba013833e32fc7b%40%3Cuser.flink.apache.org%3E).

 

Could you take a look at it when you have a chance?

 

I apologize for creating this as a critical issue. I'll ask on the mailing 
questions from now on.


was (Author: kgh475926):
[~jark] I've created a thread on the mailing list on this issue. Could you take 
a look at it when you have a chance?

 

I apologize for creating this as a critical issue. I'll ask on the mailing 
questions from now on.

> Flink TableEnvironment executesql on IntelliJ leads to Could not find a file 
> system implementation for scheme 's3a'
> ---
>
> Key: FLINK-24286
> URL: https://issues.apache.org/jira/browse/FLINK-24286
> Project: Flink
>  Issue Type: Bug
>  Components: Table SQL / API, Table SQL / Client
>Affects Versions: 1.13.2
> Environment: Ubuntu 18.04
>Reporter: James Kim
>Priority: Critical
>
> I'm trying to use the Table API in a simple Java class to create tables, run 
> queries, retrieve the results and use that data for computation. The data is 
> a CSV file from s3a (S3 compatible storage).
> When I open a terminal tab, start the cluster (standalone) in the flink 
> directory, and on another tab for Flink SQL client embedded and run queries 
> it works fine. I have the proper confs in conf/flink-conf.yaml. 
>  
> However, now i'm tyring to do this programmatically from code so I created a 
> separate project directory on IntelliJ but when I run the program, I get the 
> following error:
> "Caused by: java.lang.RuntimeException: 
> org.apache.flink.runtime.JobException: Creating the input splits caused an 
> error: Could not find a file system implementation for scheme 's3a'. The 
> scheme is directly supported by Flink through the following plugin: 
> flink-s3-fs-hadoop. Please ensure that each plugin resides within its own 
> subfolder within the plugins directory. See 
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/plugins.html for 
> more information. If you want to use a Hadoop file system for that scheme, 
> please add the scheme to the configuration fs.allowed-fallback-filesystems."
>  
> I've seen and fixed this error when running on the terminal but I run the 
> Main class directly from IntelliJ, I get the above error.
> Is there a way to configure the Main class to read from the flink-conf.yaml 
> file which is in a different path?
>  
> Main.java:
> {code:java}
> import org.apache.flink.table.api.EnvironmentSettings;
> import org.apache.flink.table.api.Table;
> import org.apache.flink.table.api.TableEnvironment;
> import org.apache.flink.table.api.TableResult;
> import org.apache.flink.types.Row;
> import org.apache.flink.util.CloseableIterator;
> public class Main {
> public static void main(String[] args) {
> // create a TableEnvironment for batch or streaming execution
> EnvironmentSettings settings = EnvironmentSettings
> .newInstance()
> .inBatchMode()
> .build();
> TableEnvironment tableEnv = TableEnvironment.create(settings);
> // create an input Table
> TableResult tempResult = tableEnv.executeSql(
> //"create temporary table ATHLETES (\n" +
> "create table ATHLETES (\n" +
> "name varchar,\n" +
> "country varchar,\n" +
> "sport varchar\n" +
> ") with (\n" +
> "'connector' = 'filesystem',\n" +
> 
> "'path'='s3a://testbucket/james_experiment/2020_Tokyo_Olympics/Athletes.csv',\n"
>  +
> "'format'='csv'\n" +
> ")\n");
> TableResult table2 = tableEnv.executeSql("select * from ATHLETES");
> }{code}
>  
> pom.xml:
>  
> {code:java}
> 
> http://maven.apache.org/POM/4.0.0;
>  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
>  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> groupId
> flink-ecs-sample
> 1.0-SNAPSHOT
> 
> 8
> 8
> 
> 
> 
> org.apache.flink
> flink-table-api-java-bridge_2.11
> 1.13.2
> compile
> 
> 
> 
> org.apache.flink
> flink-table-planner-blink_2.11
> 1.13.2
> compile
> 
> 
>  

[jira] [Commented] (FLINK-24286) Flink TableEnvironment executesql on IntelliJ leads to Could not find a file system implementation for scheme 's3a'

[James Kim (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-24286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415710#comment-17415710
 ] 

James Kim commented on FLINK-24286:
---

[~jark] I've created a thread on the mailing list on this issue. Could you take 
a look at it when you have a chance?

 

I apologize for creating this as a critical issue. I'll ask on the mailing 
questions from now on.

> Flink TableEnvironment executesql on IntelliJ leads to Could not find a file 
> system implementation for scheme 's3a'
> ---
>
> Key: FLINK-24286
> URL: https://issues.apache.org/jira/browse/FLINK-24286
> Project: Flink
>  Issue Type: Bug
>  Components: Table SQL / API, Table SQL / Client
>Affects Versions: 1.13.2
> Environment: Ubuntu 18.04
>Reporter: James Kim
>Priority: Critical
>
> I'm trying to use the Table API in a simple Java class to create tables, run 
> queries, retrieve the results and use that data for computation. The data is 
> a CSV file from s3a (S3 compatible storage).
> When I open a terminal tab, start the cluster (standalone) in the flink 
> directory, and on another tab for Flink SQL client embedded and run queries 
> it works fine. I have the proper confs in conf/flink-conf.yaml. 
>  
> However, now i'm tyring to do this programmatically from code so I created a 
> separate project directory on IntelliJ but when I run the program, I get the 
> following error:
> "Caused by: java.lang.RuntimeException: 
> org.apache.flink.runtime.JobException: Creating the input splits caused an 
> error: Could not find a file system implementation for scheme 's3a'. The 
> scheme is directly supported by Flink through the following plugin: 
> flink-s3-fs-hadoop. Please ensure that each plugin resides within its own 
> subfolder within the plugins directory. See 
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/plugins.html for 
> more information. If you want to use a Hadoop file system for that scheme, 
> please add the scheme to the configuration fs.allowed-fallback-filesystems."
>  
> I've seen and fixed this error when running on the terminal but I run the 
> Main class directly from IntelliJ, I get the above error.
> Is there a way to configure the Main class to read from the flink-conf.yaml 
> file which is in a different path?
>  
> Main.java:
> {code:java}
> import org.apache.flink.table.api.EnvironmentSettings;
> import org.apache.flink.table.api.Table;
> import org.apache.flink.table.api.TableEnvironment;
> import org.apache.flink.table.api.TableResult;
> import org.apache.flink.types.Row;
> import org.apache.flink.util.CloseableIterator;
> public class Main {
> public static void main(String[] args) {
> // create a TableEnvironment for batch or streaming execution
> EnvironmentSettings settings = EnvironmentSettings
> .newInstance()
> .inBatchMode()
> .build();
> TableEnvironment tableEnv = TableEnvironment.create(settings);
> // create an input Table
> TableResult tempResult = tableEnv.executeSql(
> //"create temporary table ATHLETES (\n" +
> "create table ATHLETES (\n" +
> "name varchar,\n" +
> "country varchar,\n" +
> "sport varchar\n" +
> ") with (\n" +
> "'connector' = 'filesystem',\n" +
> 
> "'path'='s3a://testbucket/james_experiment/2020_Tokyo_Olympics/Athletes.csv',\n"
>  +
> "'format'='csv'\n" +
> ")\n");
> TableResult table2 = tableEnv.executeSql("select * from ATHLETES");
> }{code}
>  
> pom.xml:
>  
> {code:java}
> 
> http://maven.apache.org/POM/4.0.0;
>  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
>  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> groupId
> flink-ecs-sample
> 1.0-SNAPSHOT
> 
> 8
> 8
> 
> 
> 
> org.apache.flink
> flink-table-api-java-bridge_2.11
> 1.13.2
> compile
> 
> 
> 
> org.apache.flink
> flink-table-planner-blink_2.11
> 1.13.2
> compile
> 
> 
> 
> org.apache.flink
> flink-streaming-scala_2.11
> 1.13.2
> compile
> 
> 
> 
> org.apache.flink
> flink-table-common
> 1.13.2
> compile
> 
> 
> 
> org.apache.flink
> flink-csv
> 1.13.2
> 
> 
> org.apache.flink
> 

[GitHub] [flink] flinkbot edited a comment on pull request #17292: [FLINK-24274][docs] Wrong parameter order in documentation of State Processor API

[GitBox]

flinkbot edited a comment on pull request #17292:
URL: https://github.com/apache/flink/pull/17292#issuecomment-920217923


   
   ## CI report:
   
   * 5cb9c76088321a3c1e367961d64980a01e031afd Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24169)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17291: [BP-1.14][FLINK-24292][connectors/kafka] Use KafkaSink in examples instead of FlinkKafkaProducer

[GitBox]

flinkbot edited a comment on pull request #17291:
URL: https://github.com/apache/flink/pull/17291#issuecomment-920128782


   
   ## CI report:
   
   * 812cd8bde4eab0c427020da31b2c0060300a0656 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24165)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-24286) Flink TableEnvironment executesql on IntelliJ leads to Could not find a file system implementation for scheme 's3a'

[James Kim (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-24286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415706#comment-17415706
 ] 

James Kim commented on FLINK-24286:
---

Yep, that did the trick. However, now I'm getting a different issue regarding 
s3a endpoint.


When I build and run directly from the IDE, I get an error saying a couple 
things
- INFO: Error when creating PropertyDescriptor for public final void 
org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)!
 Ignoring this property.
- com.amazonaws.SdkClientException: Failed to connect to service endpoint: 
- Caused by: java.net.NoRouteToHostException: No route to host (Host 
unreachable)
- Caused by: java.net.SocketTimeoutException: connect timed out

 

I think the issue is the s3 configurations that I've added in the pom.xml are 
not taken into account. Is pom.xml the correct place to add the s3a access key, 
s3a secret key, etc? I wasn't able to find information on the docs.

 

The pom.xml that I have now looks like the following:
{code:java}
http://maven.apache.org/POM/4.0.0;
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
4.0.0

groupId
flink-ecs-sample
1.0-SNAPSHOT


8
8

fs.s3a.access.key
myAccessKey

fs.s3a.secret.key
mySecretKey

fs.s3a.endpoint
myEndPoint







org.apache.flink
flink-table-api-java-bridge_2.11
1.13.2
compile



org.apache.flink
flink-table-planner-blink_2.11
1.13.2
compile


org.apache.flink
flink-streaming-scala_2.11
1.13.2
compile


org.apache.flink
flink-table-common
1.13.2
compile



org.apache.flink
flink-csv
1.13.2



org.apache.flink
flink-clients_2.11
1.13.2




org.apache.flink
flink-s3-fs-hadoop
1.13.2





{code}

> Flink TableEnvironment executesql on IntelliJ leads to Could not find a file 
> system implementation for scheme 's3a'
> ---
>
> Key: FLINK-24286
> URL: https://issues.apache.org/jira/browse/FLINK-24286
> Project: Flink
>  Issue Type: Bug
>  Components: Table SQL / API, Table SQL / Client
>Affects Versions: 1.13.2
> Environment: Ubuntu 18.04
>Reporter: James Kim
>Priority: Critical
>
> I'm trying to use the Table API in a simple Java class to create tables, run 
> queries, retrieve the results and use that data for computation. The data is 
> a CSV file from s3a (S3 compatible storage).
> When I open a terminal tab, start the cluster (standalone) in the flink 
> directory, and on another tab for Flink SQL client embedded and run queries 
> it works fine. I have the proper confs in conf/flink-conf.yaml. 
>  
> However, now i'm tyring to do this programmatically from code so I created a 
> separate project directory on IntelliJ but when I run the program, I get the 
> following error:
> "Caused by: java.lang.RuntimeException: 
> org.apache.flink.runtime.JobException: Creating the input splits caused an 
> error: Could not find a file system implementation for scheme 's3a'. The 
> scheme is directly supported by Flink through the following plugin: 
> flink-s3-fs-hadoop. Please ensure that each plugin resides within its own 
> subfolder within the plugins directory. See 
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/plugins.html for 
> more information. If you want to use a Hadoop file system for that scheme, 
> please add the scheme to the configuration fs.allowed-fallback-filesystems."
>  
> I've seen and fixed this error when running on the terminal but I run the 
> Main class directly from IntelliJ, I get the above error.
> Is there a way to configure the Main class to read from the flink-conf.yaml 
> file which is in a different path?
>  
> Main.java:
> {code:java}
> import org.apache.flink.table.api.EnvironmentSettings;
> import org.apache.flink.table.api.Table;
> import org.apache.flink.table.api.TableEnvironment;
> import org.apache.flink.table.api.TableResult;
> import org.apache.flink.types.Row;
> import org.apache.flink.util.CloseableIterator;
> public class Main {
> public static void main(String[] args) {
> // create a TableEnvironment for batch or streaming execution
> 

[jira] [Commented] (FLINK-24234) [FLIP-171] Byte Based & Time Based Flushing for AsyncSinkBase

[Danny Cranmer (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-24234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415700#comment-17415700
 ] 

Danny Cranmer commented on FLINK-24234:
---

Thanks [~CrynetLogistics], I have merged this. Resolving ticket

> [FLIP-171] Byte Based & Time Based Flushing for AsyncSinkBase
> -
>
> Key: FLINK-24234
> URL: https://issues.apache.org/jira/browse/FLINK-24234
> Project: Flink
>  Issue Type: New Feature
>  Components: Connectors / Common
>Reporter: Zichen Liu
>Assignee: Zichen Liu
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.15.0
>
>
> *User stories*
>  * As a Sink user, I’d like to configure the batch size for items to send to 
> the destination at once (e.g. “flush if there are x number of items in the 
> batch”)
>  * As a Sink user, I’d like to configure the batching logic so that I can 
> flush the batch of requests based on time period (e.g. “flush every 2 
> seconds”)
>  * As a Sink user I’d like to specify the number of bytes for the batch of 
> requests to be flushed (e.g. ”submit the batch after the total number of 
> bytes in it is above 1KB”)
>  * As a Sink developer, I’d like to use the configuration mechanism provided 
> to allow Sink users to configure my Sink implementation
>  * 
> *Scope*
>  * Allow Sink developers and users to pass batch size config to the 
> AsyncSinkWriter
>  * Add support for time-based flushing (e.g. “flush after x miliseconds”) 
> using the ProcessingTimeService which is part of the Sink interface
>  * Add support for byte-based flushing
>  * Consider the combination of time-based flushing and byte-based flushing, 
> if there are more bytes than configured in the time-based batch, then the 
> last few (however many necessary) items should go in the next batch to 
> satisfy the requirement for the number of bytes.
> *References*
> More details to be found 
> [https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (FLINK-24234) [FLIP-171] Byte Based & Time Based Flushing for AsyncSinkBase

[Danny Cranmer (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-24234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Danny Cranmer resolved FLINK-24234.
---
Resolution: Done

> [FLIP-171] Byte Based & Time Based Flushing for AsyncSinkBase
> -
>
> Key: FLINK-24234
> URL: https://issues.apache.org/jira/browse/FLINK-24234
> Project: Flink
>  Issue Type: New Feature
>  Components: Connectors / Common
>Reporter: Zichen Liu
>Assignee: Zichen Liu
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.15.0
>
>
> *User stories*
>  * As a Sink user, I’d like to configure the batch size for items to send to 
> the destination at once (e.g. “flush if there are x number of items in the 
> batch”)
>  * As a Sink user, I’d like to configure the batching logic so that I can 
> flush the batch of requests based on time period (e.g. “flush every 2 
> seconds”)
>  * As a Sink user I’d like to specify the number of bytes for the batch of 
> requests to be flushed (e.g. ”submit the batch after the total number of 
> bytes in it is above 1KB”)
>  * As a Sink developer, I’d like to use the configuration mechanism provided 
> to allow Sink users to configure my Sink implementation
>  * 
> *Scope*
>  * Allow Sink developers and users to pass batch size config to the 
> AsyncSinkWriter
>  * Add support for time-based flushing (e.g. “flush after x miliseconds”) 
> using the ProcessingTimeService which is part of the Sink interface
>  * Add support for byte-based flushing
>  * Consider the combination of time-based flushing and byte-based flushing, 
> if there are more bytes than configured in the time-based batch, then the 
> last few (however many necessary) items should go in the next batch to 
> satisfy the requirement for the number of bytes.
> *References*
> More details to be found 
> [https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] dannycranmer merged pull request #17244: [FLINK-24234][connectors] FLIP-171: Adding Time & Byte Size Based Flushing to Async Sink Writer

[GitBox]

dannycranmer merged pull request #17244:
URL: https://github.com/apache/flink/pull/17244


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17286: [FLINK-24281][connectors/kafka] Migrate all format tests from FlinkKafkaProducer to KafkaSink

[GitBox]

flinkbot edited a comment on pull request #17286:
URL: https://github.com/apache/flink/pull/17286#issuecomment-919857208


   
   ## CI report:
   
   * cdc812d630cb4be65fdd2e6d4a53d21abecfc632 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24163)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17229: [FLINK-23381][state] Back-pressure on reaching state change to upload limit

[GitBox]

flinkbot edited a comment on pull request #17229:
URL: https://github.com/apache/flink/pull/17229#issuecomment-916519510


   
   ## CI report:
   
   * 7d28a7c11daddc37548f4d25b404161bc045a0ea Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24159)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17292: [FLINK-24274][docs] Wrong parameter order in documentation of State Processor API

[GitBox]

flinkbot edited a comment on pull request #17292:
URL: https://github.com/apache/flink/pull/17292#issuecomment-920217923


   
   ## CI report:
   
   * 5cb9c76088321a3c1e367961d64980a01e031afd Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24169)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17275: [FLINK-24271][docs] Add document for special char in JSON_VALUE and JSON_QUERY

[GitBox]

flinkbot edited a comment on pull request #17275:
URL: https://github.com/apache/flink/pull/17275#issuecomment-918884274


   
   ## CI report:
   
   * c8ae662e089e841e3c0fd870ca4e01335ef4e871 UNKNOWN
   * e02b18793c9e4c99afd3b97c72e83d62eb6030de Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24160)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17268: [FLINK-24133][core] Network failure test replaced by comment in the code due to high expense of stabilizing such low important test

[GitBox]

flinkbot edited a comment on pull request #17268:
URL: https://github.com/apache/flink/pull/17268#issuecomment-918320586


   
   ## CI report:
   
   * 530f9c472c1c4861ff0c6c3b751424969a33ed84 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24101)
 
   * 830515768c30f19fe454fc244ae1d4ba79c50cf2 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24168)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17238: [FLINK-24233][runtime] Create view reader always on the first PartitionRequest even if request subpartition view fails in order to si

[GitBox]

flinkbot edited a comment on pull request #17238:
URL: https://github.com/apache/flink/pull/17238#issuecomment-916817365


   
   ## CI report:
   
   * 0b1e925a05a2eb992ab0396381234ba661ad66c6 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23924)
 
   * 49ee064fd07710fbba08485c8037135f91ad110d Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24167)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #16962: [FLINK-15352][connector-jdbc] Develop MySQLCatalog to connect Flink with MySQL tables and ecosystem.

[GitBox]

flinkbot edited a comment on pull request #16962:
URL: https://github.com/apache/flink/pull/16962#issuecomment-904541061


   
   ## CI report:
   
   * 6d397f18b8f43900da233d017c88ec2c357470bf Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24158)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] syhily commented on a change in pull request #17271: [FLINK-23969][connector/pulsar] Create e2e tests for pulsar connector.

[GitBox]

syhily commented on a change in pull request #17271:
URL: https://github.com/apache/flink/pull/17271#discussion_r709410737



##
File path: 
flink-end-to-end-tests/flink-end-to-end-tests-pulsar/src/test/java/org/apache/flink/tests/util/pulsar/common/FlinkContainerWithPulsarEnvironment.java
##
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.tests.util.pulsar.common;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.tests.util.TestUtils;
+import org.apache.flink.tests.util.flink.FlinkContainerTestEnvironment;
+
+import static 
org.apache.flink.configuration.TaskManagerOptions.TASK_OFF_HEAP_MEMORY;
+
+/** A Flink Container which would bundles pulsar connector in its classpath. */
+public class FlinkContainerWithPulsarEnvironment extends 
FlinkContainerTestEnvironment {
+
+public FlinkContainerWithPulsarEnvironment(int numTaskManagers, int 
numSlotsPerTaskManager) {
+super(
+numTaskManagers,
+numSlotsPerTaskManager,
+resourcePath("pulsar-connector.jar"),
+resourcePath("pulsar-client-all.jar"),
+resourcePath("pulsar-client-api.jar"),
+resourcePath("pulsar-admin-api.jar"),
+resourcePath("jul-to-slf4j.jar"));
+}
+
+private static String resourcePath(String jarName) {
+return TestUtils.getResource(jarName).toAbsolutePath().toString();
+}
+
+@Override
+protected Configuration flinkConfiguration() {
+Configuration configuration = super.flinkConfiguration();
+// Increase the off heap memory for avoiding direct buffer memory 
error on Pulsar e2e tests.
+configuration.set(TASK_OFF_HEAP_MEMORY, MemorySize.ofMebiBytes(100));

Review comment:
   @fapaul I have created a JIRA for tracking this issue. 
https://issues.apache.org/jira/projects/FLINK/issues/FLINK-24302




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] syhily edited a comment on pull request #17271: [FLINK-23969][connector/pulsar] Create e2e tests for pulsar connector.

[GitBox]

syhily edited a comment on pull request #17271:
URL: https://github.com/apache/flink/pull/17271#issuecomment-920228041


   @fapaul @dawidwys It seems that all the E2E tests were passed on Java 8 and 
11. All the review issues should be resolved. Can we merge this into Flink 
1.14.0 release? The direct memory leak on Java 11 has an extra JIRA. We can 
just wait for the new Pulsar release.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] syhily commented on pull request #17271: [FLINK-23969][connector/pulsar] Create e2e tests for pulsar connector.

[GitBox]

syhily commented on pull request #17271:
URL: https://github.com/apache/flink/pull/17271#issuecomment-920228041


   It seems that all the E2E tests were passed on Java 8 and 11. I think that 
the code condition matches the review requirements. Can we merge this into 
Flink 1.14.0 release? The direct memory leak on Java 11 has an extra JIRA. We 
can just wait for the new Pulsar release.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] syhily commented on a change in pull request #17271: [FLINK-23969][connector/pulsar] Create e2e tests for pulsar connector.

[GitBox]

syhily commented on a change in pull request #17271:
URL: https://github.com/apache/flink/pull/17271#discussion_r709410737



##
File path: 
flink-end-to-end-tests/flink-end-to-end-tests-pulsar/src/test/java/org/apache/flink/tests/util/pulsar/common/FlinkContainerWithPulsarEnvironment.java
##
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.tests.util.pulsar.common;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.tests.util.TestUtils;
+import org.apache.flink.tests.util.flink.FlinkContainerTestEnvironment;
+
+import static 
org.apache.flink.configuration.TaskManagerOptions.TASK_OFF_HEAP_MEMORY;
+
+/** A Flink Container which would bundles pulsar connector in its classpath. */
+public class FlinkContainerWithPulsarEnvironment extends 
FlinkContainerTestEnvironment {
+
+public FlinkContainerWithPulsarEnvironment(int numTaskManagers, int 
numSlotsPerTaskManager) {
+super(
+numTaskManagers,
+numSlotsPerTaskManager,
+resourcePath("pulsar-connector.jar"),
+resourcePath("pulsar-client-all.jar"),
+resourcePath("pulsar-client-api.jar"),
+resourcePath("pulsar-admin-api.jar"),
+resourcePath("jul-to-slf4j.jar"));
+}
+
+private static String resourcePath(String jarName) {
+return TestUtils.getResource(jarName).toAbsolutePath().toString();
+}
+
+@Override
+protected Configuration flinkConfiguration() {
+Configuration configuration = super.flinkConfiguration();
+// Increase the off heap memory for avoiding direct buffer memory 
error on Pulsar e2e tests.
+configuration.set(TASK_OFF_HEAP_MEMORY, MemorySize.ofMebiBytes(100));

Review comment:
   @fapaul I have created a JIRA for tracking this issue. 
https://issues.apache.org/jira/projects/FLINK/issues/FLINK-24233




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (FLINK-24302) Direct buffer memory leak on Pulsar connector with Java 11

[Yufan Sheng (Jira)]

Yufan Sheng created FLINK-24302:
---

 Summary: Direct buffer memory leak on Pulsar connector with Java 11
 Key: FLINK-24302
 URL: https://issues.apache.org/jira/browse/FLINK-24302
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Pulsar
Affects Versions: 1.14.0
Reporter: Yufan Sheng


Running the Pulsar connector with multiple split readers on Java 11 could throw 
{{a java.lang.OutOfMemoryError exception}}.

 
{code:java}
Caused by: java.util.concurrent.CompletionException: 
org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: 
Could not complete the operation. Number of retries has been exhausted. Failed 
reason: java.lang.OutOfMemoryError: Direct buffer memory
at 
java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
at 
java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
at 
java.base/java.util.concurrent.CompletableFuture$OrApply.tryFire(CompletableFuture.java:1503)
... 42 more
Caused by: 
org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: 
Could not complete the operation. Number of retries has been exhausted. Failed 
reason: java.lang.OutOfMemoryError: Direct buffer memory
at 
org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:249)
... 39 more
Caused by: org.apache.pulsar.shade.io.netty.handler.codec.EncoderException: 
java.lang.OutOfMemoryError: Direct buffer memory
at 
org.apache.pulsar.shade.io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:104)
at 
org.apache.pulsar.shade.io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:346)
at 
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)
at 
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)
at 
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)
at 
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)
at 
org.apache.pulsar.shade.io.netty.handler.stream.ChunkedWriteHandler.doFlush(ChunkedWriteHandler.java:303)
at 
org.apache.pulsar.shade.io.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:132)
at 
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750)
at 
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:765)
at 
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:790)
at 
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:758)
at 
org.apache.pulsar.shade.io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1020)
at 
org.apache.pulsar.shade.io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:311)
at 
org.apache.pulsar.shade.org.asynchttpclient.netty.request.NettyRequestSender.writeRequest(NettyRequestSender.java:420)
... 23 more
{code}

The reason is that under Java 11, the Netty will allocate memory from the pool 
of Java Direct Memory and is affected by the MaxDirectMemory limit. Under Java 
8, it allocates native memory and is not affected by that setting.

We have to reduce the direct memory usage by using a newer Pulsar client which 
has a memory-limits configuration.

This issue is addressed on Pulsar, and 
[PIP-74|https://github.com/apache/pulsar/wiki/PIP-74%3A-Pulsar-client-memory-limits]
 has been created for resolving this issue.

We should keep this issue open with no resolved versions until Pulsar provides 
a new client with memory limits.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot commented on pull request #17292: [FLINK-24274][docs] Wrong parameter order in documentation of State Processor API

[GitBox]

flinkbot commented on pull request #17292:
URL: https://github.com/apache/flink/pull/17292#issuecomment-920217923


   
   ## CI report:
   
   * 5cb9c76088321a3c1e367961d64980a01e031afd UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17268: [FLINK-24133][core] Network failure test replaced by comment in the code due to high expense of stabilizing such low important test

[GitBox]

flinkbot edited a comment on pull request #17268:
URL: https://github.com/apache/flink/pull/17268#issuecomment-918320586


   
   ## CI report:
   
   * 530f9c472c1c4861ff0c6c3b751424969a33ed84 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24101)
 
   * 830515768c30f19fe454fc244ae1d4ba79c50cf2 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17256: [FLINK-23385][table] Implement COALESCE function

[GitBox]

flinkbot edited a comment on pull request #17256:
URL: https://github.com/apache/flink/pull/17256#issuecomment-918040209


   
   ## CI report:
   
   * 25fb8010703524981f7934402ebe46b266dc551e UNKNOWN
   * 07e698c121920399ea16be34674219d3d8b0e5d9 Azure: 
[CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24157)
 
   * 521bf3c8656aaa4d7c9fc4b213c3e47e0ba95f03 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24162)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17238: [FLINK-24233][runtime] Create view reader always on the first PartitionRequest even if request subpartition view fails in order to si

[GitBox]

flinkbot edited a comment on pull request #17238:
URL: https://github.com/apache/flink/pull/17238#issuecomment-916817365


   
   ## CI report:
   
   * 0b1e925a05a2eb992ab0396381234ba661ad66c6 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23924)
 
   * 49ee064fd07710fbba08485c8037135f91ad110d UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org