[jira] [Updated] (FLINK-14818) StreamNetworkBenchmarkEnvironment incorrectly setups a receiving InputGate

[ https://issues.apache.org/jira/browse/FLINK-14818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-14818: Fix Version/s: 1.11.0 > StreamNetworkBenchmarkEnvironment incorrectly setups a receiving InputGate

[jira] [Updated] (FLINK-16298) GroupWindowTableAggregateITCase.testEventTimeTumblingWindow fails on Travis

[ https://issues.apache.org/jira/browse/FLINK-16298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-16298: Issue Type: Bug (was: Test) > GroupWindowTableAggregateITCase.testEventTimeTumblingWindow fails o

[jira] [Created] (FLINK-16298) GroupWindowTableAggregateITCase.testEventTimeTumblingWindow fails on Travis

Yingjie Cao created FLINK-16298: --- Summary: GroupWindowTableAggregateITCase.testEventTimeTumblingWindow fails on Travis Key: FLINK-16298 URL: https://issues.apache.org/jira/browse/FLINK-16298 Project: Fl

[jira] [Commented] (FLINK-16069) Creation of TaskDeploymentDescriptor can block main thread for long time

[ https://issues.apache.org/jira/browse/FLINK-16069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17045107#comment-17045107 ] Yingjie Cao commented on FLINK-16069: - Maybe we can split the tasks to be deployed i

[jira] [Commented] (FLINK-14818) StreamNetworkBenchmarkEnvironment incorrectly setups a receiving InputGate

[ https://issues.apache.org/jira/browse/FLINK-14818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041608#comment-17041608 ] Yingjie Cao commented on FLINK-14818: - I have opened a PR for this issue and Zhijian

[jira] [Comment Edited] (FLINK-16142) Memory Leak causes Metaspace OOM error on repeated job submission

[ https://issues.apache.org/jira/browse/FLINK-16142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17040625#comment-17040625 ] Yingjie Cao edited comment on FLINK-16142 at 2/20/20 5:11 AM:

[jira] [Commented] (FLINK-16142) Memory Leak causes Metaspace OOM error on repeated job submission

[ https://issues.apache.org/jira/browse/FLINK-16142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17040625#comment-17040625 ] Yingjie Cao commented on FLINK-16142: - Could you also print the contextClassLoader o

[jira] [Commented] (FLINK-16142) Memory Leak causes Metaspace OOM error on repeated job submission

[ https://issues.apache.org/jira/browse/FLINK-16142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17039633#comment-17039633 ] Yingjie Cao commented on FLINK-16142: - I guess there is thread leak which prevents t

[jira] [Commented] (FLINK-16030) Add heartbeat between netty server and client to detect long connection alive

[ https://issues.apache.org/jira/browse/FLINK-16030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17036811#comment-17036811 ] Yingjie Cao commented on FLINK-16030: - I also ever encountered this problem (though

[jira] [Commented] (FLINK-16012) Reduce the default number of exclusive buffers from 2 to 1 on receiver side

[ https://issues.apache.org/jira/browse/FLINK-16012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17036715#comment-17036715 ] Yingjie Cao commented on FLINK-16012: - [~wind_ljy] I have updated the results and th

[jira] [Commented] (FLINK-16012) Reduce the default number of exclusive buffers from 2 to 1 on receiver side

[ https://issues.apache.org/jira/browse/FLINK-16012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17036714#comment-17036714 ] Yingjie Cao commented on FLINK-16012: - Theoretically, reducing the number of buffers

[jira] [Updated] (FLINK-16012) Reduce the default number of exclusive buffers from 2 to 1 on receiver side

[ https://issues.apache.org/jira/browse/FLINK-16012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-16012: Attachment: (was: image-2020-02-14-09-13-07-967.png) > Reduce the default number of exclusive

[jira] [Updated] (FLINK-16012) Reduce the default number of exclusive buffers from 2 to 1 on receiver side

[ https://issues.apache.org/jira/browse/FLINK-16012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-16012: Attachment: (was: image-2020-02-14-07-23-16-171.png) > Reduce the default number of exclusive

[jira] [Updated] (FLINK-16012) Reduce the default number of exclusive buffers from 2 to 1 on receiver side

[ https://issues.apache.org/jira/browse/FLINK-16012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-16012: Attachment: (was: image-2020-02-13-21-54-05-026.png) > Reduce the default number of exclusive

[jira] [Updated] (FLINK-16012) Reduce the default number of exclusive buffers from 2 to 1 on receiver side

[ https://issues.apache.org/jira/browse/FLINK-16012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-16012: Attachment: (was: image-2020-02-13-23-30-17-951.png) > Reduce the default number of exclusive

[jira] [Updated] (FLINK-16012) Reduce the default number of exclusive buffers from 2 to 1 on receiver side

[ https://issues.apache.org/jira/browse/FLINK-16012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-16012: Attachment: image-2020-02-14-09-13-07-967.png > Reduce the default number of exclusive buffers fro

[jira] [Updated] (FLINK-16012) Reduce the default number of exclusive buffers from 2 to 1 on receiver side

[ https://issues.apache.org/jira/browse/FLINK-16012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-16012: Attachment: image-2020-02-14-07-23-16-171.png > Reduce the default number of exclusive buffers fro

[jira] [Updated] (FLINK-16012) Reduce the default number of exclusive buffers from 2 to 1 on receiver side

[ https://issues.apache.org/jira/browse/FLINK-16012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-16012: Attachment: image-2020-02-13-23-30-17-951.png > Reduce the default number of exclusive buffers fro

[jira] [Updated] (FLINK-16012) Reduce the default number of exclusive buffers from 2 to 1 on receiver side

[ https://issues.apache.org/jira/browse/FLINK-16012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-16012: Attachment: image-2020-02-13-21-54-05-026.png > Reduce the default number of exclusive buffers fro

[jira] [Commented] (FLINK-15665) External shuffle service support in Kubernetes mode

[ https://issues.apache.org/jira/browse/FLINK-15665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029738#comment-17029738 ] Yingjie Cao commented on FLINK-15665: - [~felixzheng] Is it a duplicate of FLINK-1324

[jira] [Commented] (FLINK-15305) MemoryMappedBoundedDataTest fails with IOException on ppc64le

[ https://issues.apache.org/jira/browse/FLINK-15305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029667#comment-17029667 ] Yingjie Cao commented on FLINK-15305: - OK, I will open a PR to fix it soon. > Memor

[jira] [Commented] (FLINK-15305) MemoryMappedBoundedDataTest fails with IOException on ppc64le

[ https://issues.apache.org/jira/browse/FLINK-15305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029597#comment-17029597 ] Yingjie Cao commented on FLINK-15305: - Sorry for the late response, we were on Chine

[jira] [Commented] (FLINK-10462) Remove ConnectionIndex for further sharing tcp connection in credit-based mode

[ https://issues.apache.org/jira/browse/FLINK-10462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006711#comment-17006711 ] Yingjie Cao commented on FLINK-10462: - Recently, we encountered the same problem und

[jira] [Created] (FLINK-15455) Enable TCP connection reuse across multiple jobs.

Yingjie Cao created FLINK-15455: --- Summary: Enable TCP connection reuse across multiple jobs. Key: FLINK-15455 URL: https://issues.apache.org/jira/browse/FLINK-15455 Project: Flink Issue Type: I

[jira] [Closed] (FLINK-15308) Job failed when enable pipelined-shuffle.compression and numberOfTaskSlots > 1

[ https://issues.apache.org/jira/browse/FLINK-15308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao closed FLINK-15308. --- > Job failed when enable pipelined-shuffle.compression and numberOfTaskSlots > 1 > -

[jira] [Resolved] (FLINK-15308) Job failed when enable pipelined-shuffle.compression and numberOfTaskSlots > 1

[ https://issues.apache.org/jira/browse/FLINK-15308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao resolved FLINK-15308. - Release Note: taskmanager.network.pipelined-shuffle.compression.enabled was removed. Data compre

[jira] [Commented] (FLINK-15308) Job failed when enable pipelined-shuffle.compression and numberOfTaskSlots > 1

[ https://issues.apache.org/jira/browse/FLINK-15308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17000982#comment-17000982 ] Yingjie Cao commented on FLINK-15308: - Fix via 8525c378b91c16245d2e0456d423ed39f5c9b

[jira] [Closed] (FLINK-15311) Lz4BlockCompressionFactory should use native compressor instead of java unsafe

[ https://issues.apache.org/jira/browse/FLINK-15311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao closed FLINK-15311. --- > Lz4BlockCompressionFactory should use native compressor instead of java unsafe > -

[jira] [Commented] (FLINK-15311) Lz4BlockCompressionFactory should use native compressor instead of java unsafe

[ https://issues.apache.org/jira/browse/FLINK-15311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17000971#comment-17000971 ] Yingjie Cao commented on FLINK-15311: - Fix via 5a30c79b0db7c99784fbb40346b7682d0ac64

[jira] [Resolved] (FLINK-15311) Lz4BlockCompressionFactory should use native compressor instead of java unsafe

[ https://issues.apache.org/jira/browse/FLINK-15311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao resolved FLINK-15311. - Resolution: Fixed > Lz4BlockCompressionFactory should use native compressor instead of java unsa

[jira] [Commented] (FLINK-15308) Job failed when enable pipelined-shuffle.compression and numberOfTaskSlots > 1

[ https://issues.apache.org/jira/browse/FLINK-15308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16999868#comment-16999868 ] Yingjie Cao commented on FLINK-15308: - The problem is cause by race of multi netty t

[jira] [Commented] (FLINK-15308) Job failed when enable pipelined-shuffle.compression and numberOfTaskSlots > 1

[ https://issues.apache.org/jira/browse/FLINK-15308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16999708#comment-16999708 ] Yingjie Cao commented on FLINK-15308: - [~fengjiajie] I also reproduced it. > Job fa

[jira] [Issue Comment Deleted] (FLINK-15308) Job failed when enable pipelined-shuffle.compression and numberOfTaskSlots > 1

[ https://issues.apache.org/jira/browse/FLINK-15308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-15308: Comment: was deleted (was: [~fengjiajie] I can not reproduce the problem in my test environment. I

[jira] [Updated] (FLINK-15308) Job failed when enable pipelined-shuffle.compression and numberOfTaskSlots > 1

[ https://issues.apache.org/jira/browse/FLINK-15308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-15308: Attachment: image-2019-12-19-10-55-30-644.png > Job failed when enable pipelined-shuffle.compressi

[jira] [Commented] (FLINK-15308) Job failed when enable pipelined-shuffle.compression and numberOfTaskSlots > 1

[ https://issues.apache.org/jira/browse/FLINK-15308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16999694#comment-16999694 ] Yingjie Cao commented on FLINK-15308: - [~fengjiajie] I can not reproduce the problem

[jira] [Commented] (FLINK-15311) Lz4BlockCompressionFactory should use native compressor instead of java unsafe

[ https://issues.apache.org/jira/browse/FLINK-15311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998946#comment-16998946 ] Yingjie Cao commented on FLINK-15311: - [~lzljs3620320] Thanks for reporting this iss

[jira] [Commented] (FLINK-15308) Job failed when enable pipelined-shuffle.compression and numberOfTaskSlots > 1

[ https://issues.apache.org/jira/browse/FLINK-15308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998934#comment-16998934 ] Yingjie Cao commented on FLINK-15308: - [~fengjiajie] Thanks for reporting the issue

[jira] [Commented] (FLINK-15308) Job failed when enable pipelined-shuffle.compression and numberOfTaskSlots > 1

[ https://issues.apache.org/jira/browse/FLINK-15308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998892#comment-16998892 ] Yingjie Cao commented on FLINK-15308: - [~fengjiajie] Could you share your code? I wo

[jira] [Commented] (FLINK-15305) MemoryMappedBoundedDataTest fails with IOException on ppc64le

[ https://issues.apache.org/jira/browse/FLINK-15305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998886#comment-16998886 ] Yingjie Cao commented on FLINK-15305: - I guess this is because that the page size on

[jira] [Commented] (FLINK-15239) TM Metaspace memory leak

[ https://issues.apache.org/jira/browse/FLINK-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16997100#comment-16997100 ] Yingjie Cao commented on FLINK-15239: - Not only TM, JM also suffers from the problem

[jira] [Resolved] (FLINK-15140) Shuffle data compression does not work with BroadcastRecordWriter.

[ https://issues.apache.org/jira/browse/FLINK-15140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao resolved FLINK-15140. - Resolution: Fixed > Shuffle data compression does not work with BroadcastRecordWriter. > ---

[jira] [Commented] (FLINK-15140) Shuffle data compression does not work with BroadcastRecordWriter.

[ https://issues.apache.org/jira/browse/FLINK-15140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16995511#comment-16995511 ] Yingjie Cao commented on FLINK-15140: - Fix via 5139282e7f48ece00528c6ade9480f10fa3dd

[jira] [Closed] (FLINK-15140) Shuffle data compression does not work with BroadcastRecordWriter.

[ https://issues.apache.org/jira/browse/FLINK-15140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao closed FLINK-15140. --- > Shuffle data compression does not work with BroadcastRecordWriter. > -

[jira] [Closed] (FLINK-15166) Shuffle data compression wrongly decrease the buffer reference count.

[ https://issues.apache.org/jira/browse/FLINK-15166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao closed FLINK-15166. --- > Shuffle data compression wrongly decrease the buffer reference count. > --

[jira] [Created] (FLINK-15166) Shuffle data compression wrongly decrease the buffer reference count.

Yingjie Cao created FLINK-15166: --- Summary: Shuffle data compression wrongly decrease the buffer reference count. Key: FLINK-15166 URL: https://issues.apache.org/jira/browse/FLINK-15166 Project: Flink

[jira] [Updated] (FLINK-15140) Shuffle data compression does not work with BroadcastRecordWriter.

[ https://issues.apache.org/jira/browse/FLINK-15140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-15140: Description: I tested the newest code of master branch last weekend with more test cases. Unfortu

[jira] [Created] (FLINK-15140) Shuffle data compression does not work with BroadcastRecordWriter.

Yingjie Cao created FLINK-15140: --- Summary: Shuffle data compression does not work with BroadcastRecordWriter. Key: FLINK-15140 URL: https://issues.apache.org/jira/browse/FLINK-15140 Project: Flink

[jira] [Resolved] (FLINK-14845) Introduce data compression to blocking shuffle.

[ https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao resolved FLINK-14845. - Release Note: Now user can set taskmanager.network.blocking-shuffle.compression.enabled to true

[jira] [Commented] (FLINK-14845) Introduce data compression to blocking shuffle.

[ https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16990697#comment-16990697 ] Yingjie Cao commented on FLINK-14845: - Fix via 66d4d7da2d8b717f420509d9785fad0880562

[jira] [Commented] (FLINK-14872) Potential deadlock for task reading from blocking ResultPartition.

[ https://issues.apache.org/jira/browse/FLINK-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16990532#comment-16990532 ] Yingjie Cao commented on FLINK-14872: - [~pnowojski] Thanks for the review. I have al

[jira] [Commented] (FLINK-14872) Potential deadlock for task reading from blocking ResultPartition.

[ https://issues.apache.org/jira/browse/FLINK-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989957#comment-16989957 ] Yingjie Cao commented on FLINK-14872: - As discussed, we would implement a temporary

[jira] [Commented] (FLINK-14872) Potential deadlock for task reading from blocking ResultPartition.

[ https://issues.apache.org/jira/browse/FLINK-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989800#comment-16989800 ] Yingjie Cao commented on FLINK-14872: - [~pnowojski] I mean creating a zero size buff

[jira] [Commented] (FLINK-14872) Potential deadlock for task reading from blocking ResultPartition.

[ https://issues.apache.org/jira/browse/FLINK-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989755#comment-16989755 ] Yingjie Cao commented on FLINK-14872: - By the way, I will test the change by running

[jira] [Commented] (FLINK-14872) Potential deadlock for task reading from blocking ResultPartition.

[ https://issues.apache.org/jira/browse/FLINK-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989723#comment-16989723 ] Yingjie Cao commented on FLINK-14872: - Sadly, I find the number of exclusive buffer

[jira] [Commented] (FLINK-14952) Yarn containers can exceed physical memory limits when using BoundedBlockingSubpartition.

[ https://issues.apache.org/jira/browse/FLINK-14952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989448#comment-16989448 ] Yingjie Cao commented on FLINK-14952: - [~gjy] By "FILE-FILE" mode, I mean using Boun

[jira] [Commented] (FLINK-14872) Potential deadlock for task reading from blocking ResultPartition.

[ https://issues.apache.org/jira/browse/FLINK-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988808#comment-16988808 ] Yingjie Cao commented on FLINK-14872: - [~pnowojski] You are right. We may add a rela

[jira] [Commented] (FLINK-14872) Potential deadlock for task reading from blocking ResultPartition.

[ https://issues.apache.org/jira/browse/FLINK-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16987470#comment-16987470 ] Yingjie Cao commented on FLINK-14872: - Thanks for the comment. [~pnowojski] Making

[jira] [Updated] (FLINK-14845) Introduce data compression to blocking shuffle.

[ https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-14845: Fix Version/s: 1.10.0 > Introduce data compression to blocking shuffle. >

[jira] [Created] (FLINK-15030) Potential deadlock for bounded blocking ResultPartition.

Yingjie Cao created FLINK-15030: --- Summary: Potential deadlock for bounded blocking ResultPartition. Key: FLINK-15030 URL: https://issues.apache.org/jira/browse/FLINK-15030 Project: Flink Issue

[jira] [Updated] (FLINK-15024) System classloader memory leak after loading too many codegen classes.

[ https://issues.apache.org/jira/browse/FLINK-15024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-15024: Description: We are using Flink session cluster as a service for ad-hoc queries. After running so

[jira] [Created] (FLINK-15024) System classloader memory leak after loading too many codegen classes.

Yingjie Cao created FLINK-15024: --- Summary: System classloader memory leak after loading too many codegen classes. Key: FLINK-15024 URL: https://issues.apache.org/jira/browse/FLINK-15024 Project: Flink

[jira] [Commented] (FLINK-14872) Potential deadlock for task reading from blocking ResultPartition.

[ https://issues.apache.org/jira/browse/FLINK-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985948#comment-16985948 ] Yingjie Cao commented on FLINK-14872: - > If we had to implement something, I would b

[jira] [Commented] (FLINK-14845) Introduce data compression to blocking shuffle.

[ https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985873#comment-16985873 ] Yingjie Cao commented on FLINK-14845: - Based on the above discussions, I have given

[jira] [Updated] (FLINK-14845) Introduce data compression to blocking shuffle.

[ https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-14845: Labels: pull-request-available (was: ) > Introduce data compression to blocking shuffle. > --

[jira] [Commented] (FLINK-14952) Yarn containers can exceed physical memory limits when using BoundedBlockingSubpartition.

[ https://issues.apache.org/jira/browse/FLINK-14952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16984103#comment-16984103 ] Yingjie Cao commented on FLINK-14952: - > Doesn't this defeat the purpose of using mm

[jira] [Commented] (FLINK-14845) Introduce data compression to blocking shuffle.

[ https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16984097#comment-16984097 ] Yingjie Cao commented on FLINK-14845: - > So I think it's fine to not make it pluggab

[jira] [Commented] (FLINK-14952) Yarn containers can exceed physical memory limits when using BoundedBlockingSubpartition.

[ https://issues.apache.org/jira/browse/FLINK-14952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16983397#comment-16983397 ] Yingjie Cao commented on FLINK-14952: - Can we manage the memory mapped region and re

[jira] [Commented] (FLINK-14845) Introduce data compression to blocking shuffle.

[ https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16983338#comment-16983338 ] Yingjie Cao commented on FLINK-14845: - > Would it be more complicate to request a bu

[jira] [Commented] (FLINK-14845) Introduce data compression to blocking shuffle.

[ https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16982271#comment-16982271 ] Yingjie Cao commented on FLINK-14845: - > Where are you going to compress data into?

[jira] [Comment Edited] (FLINK-14872) Potential deadlock for task reading from blocking ResultPartition.

[ https://issues.apache.org/jira/browse/FLINK-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979866#comment-16979866 ] Yingjie Cao edited comment on FLINK-14872 at 11/22/19 5:52 AM: ---

[jira] [Commented] (FLINK-14845) Introduce data compression to blocking shuffle.

[ https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979871#comment-16979871 ] Yingjie Cao commented on FLINK-14845: - [~pnowojski] I totally agree with you. Then I

[jira] [Commented] (FLINK-14872) Potential deadlock for task reading from blocking ResultPartition.

[ https://issues.apache.org/jira/browse/FLINK-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979866#comment-16979866 ] Yingjie Cao commented on FLINK-14872: - There are some choices we may consider: # Li

[jira] [Commented] (FLINK-14818) StreamNetworkBenchmarkEnvironment incorrectly setups a receiving InputGate

[ https://issues.apache.org/jira/browse/FLINK-14818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979219#comment-16979219 ] Yingjie Cao commented on FLINK-14818: - I found there seems no very straightforward w

[jira] [Commented] (FLINK-14872) Potential deadlock for task reading from blocking ResultPartition.

[ https://issues.apache.org/jira/browse/FLINK-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979191#comment-16979191 ] Yingjie Cao commented on FLINK-14872: - [~pnowojski] They are relevant and the causes

[jira] [Commented] (FLINK-14872) Potential deadlock for task reading from blocking ResultPartition.

[ https://issues.apache.org/jira/browse/FLINK-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979069#comment-16979069 ] Yingjie Cao commented on FLINK-14872: - Theoretically, the problem is also exist with

[jira] [Commented] (FLINK-14872) Potential deadlock for task reading from blocking ResultPartition.

[ https://issues.apache.org/jira/browse/FLINK-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979037#comment-16979037 ] Yingjie Cao commented on FLINK-14872: - Only limit the number of Buffers can be alloc

[jira] [Updated] (FLINK-14872) Potential deadlock for task reading from blocking ResultPartition.

[ https://issues.apache.org/jira/browse/FLINK-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-14872: Fix Version/s: 1.10.0 > Potential deadlock for task reading from blocking ResultPartition. > -

[jira] [Commented] (FLINK-14845) Introduce data compression to blocking shuffle.

[ https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979002#comment-16979002 ] Yingjie Cao commented on FLINK-14845: - [~pnowojski] , thanks for proposing these opt

[jira] [Commented] (FLINK-14845) Introduce data compression to blocking shuffle.

[ https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16978288#comment-16978288 ] Yingjie Cao commented on FLINK-14845: - [~chesnay] A choice maybe make the compressio

[jira] [Commented] (FLINK-14845) Introduce data compression to blocking shuffle.

[ https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16978284#comment-16978284 ] Yingjie Cao commented on FLINK-14845: - [~pnowojski]  You are right, taking both Bloc

[jira] [Commented] (FLINK-14845) Introduce data compression to blocking shuffle.

[ https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16978241#comment-16978241 ] Yingjie Cao commented on FLINK-14845: - Doing compression and decompression with task

[jira] [Commented] (FLINK-14845) Introduce data compression to blocking shuffle.

[ https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16978184#comment-16978184 ] Yingjie Cao commented on FLINK-14845: - [~lzljs3620320] For small data, for example,

[jira] [Updated] (FLINK-14872) Potential deadlock for task reading from blocking ResultPartition.

[ https://issues.apache.org/jira/browse/FLINK-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-14872: Description: Currently, the buffer pool size of InputGate reading from blocking ResultPartition i

[jira] [Created] (FLINK-14872) Potential deadlock for task reading from blocking ResultPartition.

Yingjie Cao created FLINK-14872: --- Summary: Potential deadlock for task reading from blocking ResultPartition. Key: FLINK-14872 URL: https://issues.apache.org/jira/browse/FLINK-14872 Project: Flink

[jira] [Commented] (FLINK-14845) Introduce data compression to blocking shuffle.

[ https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16978153#comment-16978153 ] Yingjie Cao commented on FLINK-14845: - To verify how many improvements we could gain

[jira] [Updated] (FLINK-14845) Introduce data compression to blocking shuffle.

[ https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-14845: Description: Currently, blocking shuffle writer writes raw output data to disk without compressio

[jira] [Created] (FLINK-14845) Introduce data compression to blocking shuffle.

Yingjie Cao created FLINK-14845: --- Summary: Introduce data compression to blocking shuffle. Key: FLINK-14845 URL: https://issues.apache.org/jira/browse/FLINK-14845 Project: Flink Issue Type: Imp

[jira] [Closed] (FLINK-14603) NetworkBufferPoolTest.testBlockingRequestFromMultiLocalBufferPool timeout in travis

[ https://issues.apache.org/jira/browse/FLINK-14603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao closed FLINK-14603. --- > NetworkBufferPoolTest.testBlockingRequestFromMultiLocalBufferPool timeout in > travis > -

[jira] [Commented] (FLINK-14603) NetworkBufferPoolTest.testBlockingRequestFromMultiLocalBufferPool timeout in travis

[ https://issues.apache.org/jira/browse/FLINK-14603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16967377#comment-16967377 ] Yingjie Cao commented on FLINK-14603: - Thanks, [~trohrmann] and [~ykt836]. I will tr

[jira] [Closed] (FLINK-14498) Introduce NetworkBufferPool#isAvailable() for non-blocking output

[ https://issues.apache.org/jira/browse/FLINK-14498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao closed FLINK-14498. --- > Introduce NetworkBufferPool#isAvailable() for non-blocking output > --

[jira] [Commented] (FLINK-14498) Introduce NetworkBufferPool#isAvailable() for interacting with LocalBufferPool

[ https://issues.apache.org/jira/browse/FLINK-14498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16960764#comment-16960764 ] Yingjie Cao commented on FLINK-14498: - [~wind_ljy] The PR is based on another PR  [h

[jira] [Closed] (FLINK-14195) Nightly connectors-jdk11 fails because of missing jaxb classes

[ https://issues.apache.org/jira/browse/FLINK-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao closed FLINK-14195. --- > Nightly connectors-jdk11 fails because of missing jaxb classes > -

[jira] [Commented] (FLINK-14195) Nightly connectors-jdk11 fails because of missing jaxb classes

[ https://issues.apache.org/jira/browse/FLINK-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938472#comment-16938472 ] Yingjie Cao commented on FLINK-14195: - Thanks [~trohrmann]. My branch is  [https://

[jira] [Commented] (FLINK-12576) inputQueueLength metric does not work for LocalInputChannels

[ https://issues.apache.org/jira/browse/FLINK-12576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938406#comment-16938406 ] Yingjie Cao commented on FLINK-12576: - [~alpinegizmo] Because only one of the two up

[jira] [Commented] (FLINK-14195) Nightly connectors-jdk11 fails because of missing jaxb classes

[ https://issues.apache.org/jira/browse/FLINK-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938383#comment-16938383 ] Yingjie Cao commented on FLINK-14195: - Hi [~trohrmann], I am trying to fix the probl

[jira] [Commented] (FLINK-12576) inputQueueLength metric does not work for LocalInputChannels

[ https://issues.apache.org/jira/browse/FLINK-12576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938232#comment-16938232 ] Yingjie Cao commented on FLINK-12576: - [~alpinegizmo] I tried the instruction gave a

[jira] [Updated] (FLINK-12576) inputQueueLength metric does not work for LocalInputChannels

[ https://issues.apache.org/jira/browse/FLINK-12576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-12576: Attachment: image-2019-09-26-11-36-06-027.png > inputQueueLength metric does not work for LocalInp

[jira] [Updated] (FLINK-12576) inputQueueLength metric does not work for LocalInputChannels

[ https://issues.apache.org/jira/browse/FLINK-12576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-12576: Attachment: image-2019-09-26-11-34-24-878.png > inputQueueLength metric does not work for LocalInp

[jira] [Created] (FLINK-14185) AbstractServerTest failed on Travis

Yingjie Cao created FLINK-14185: --- Summary: AbstractServerTest failed on Travis Key: FLINK-14185 URL: https://issues.apache.org/jira/browse/FLINK-14185 Project: Flink Issue Type: Bug C

[jira] [Created] (FLINK-14180) Enable config of maximum capacity of FileArchivedExecutionGraphStore.

Yingjie Cao created FLINK-14180: --- Summary: Enable config of maximum capacity of FileArchivedExecutionGraphStore. Key: FLINK-14180 URL: https://issues.apache.org/jira/browse/FLINK-14180 Project: Flink

<    2   3   4   5   6   7   8   >