Hi devs,
Currently, Flink adopts a hash-style blocking shuffle implementation which
writes data sent to different reducer tasks into separate files
concurrently. Compared to sort-merge based approach writes those data
together into a single file and merges those small files into bigger ones,
Yingjie Cao created FLINK-19614:
---
Summary: Further optimization of sort-merge based blocking shuffle
Key: FLINK-19614
URL: https://issues.apache.org/jira/browse/FLINK-19614
Project: Flink
Yingjie Cao created FLINK-19603:
---
Summary: Introduce shuffle data compression to sort-merge based
blocking shuffle
Key: FLINK-19603
URL: https://issues.apache.org/jira/browse/FLINK-19603
Project: Flink
Yingjie Cao created FLINK-19602:
---
Summary: Introduce new config options to enable sort-merge based
blocking shuffle
Key: FLINK-19602
URL: https://issues.apache.org/jira/browse/FLINK-19602
Project
Yingjie Cao created FLINK-19601:
---
Summary: Introduce sort-merge based blocking result partition
SortMergeResultPartition and the corresponding subpartition reader
Key: FLINK-19601
URL: https://issues.apache.org
Yingjie Cao created FLINK-19600:
---
Summary: Introduce PartitionedFile and the corresponding
writer/reader for sort-merge based blocking shuffle
Key: FLINK-19600
URL: https://issues.apache.org/jira/browse/FLINK-19600
Yingjie Cao created FLINK-19597:
---
Summary: Introduce SortBuffer and its implementation
PartitionSortedBuffer for sort-merge based blocking shuffle
Key: FLINK-19597
URL: https://issues.apache.org/jira/browse/FLINK
Yingjie Cao created FLINK-19582:
---
Summary: Introduce sort-merge based blocking shuffle to Flink
Key: FLINK-19582
URL: https://issues.apache.org/jira/browse/FLINK-19582
Project: Flink
Issue
Yingjie Cao created FLINK-19344:
---
Summary:
DispatcherResourceCleanupTest#testJobSubmissionUnderSameJobId is unstable on
Azure Pipeline
Key: FLINK-19344
URL: https://issues.apache.org/jira/browse/FLINK-19344
Yingjie Cao created FLINK-19323:
---
Summary: Small optimization of network layer record serialization
Key: FLINK-19323
URL: https://issues.apache.org/jira/browse/FLINK-19323
Project: Flink
Issue
Yingjie Cao created FLINK-19320:
---
Summary: Remove clearBuffers from the public interfaces of
RecordWriter
Key: FLINK-19320
URL: https://issues.apache.org/jira/browse/FLINK-19320
Project: Flink
Yingjie Cao created FLINK-19312:
---
Summary: Introduce BufferWritingResultPartition which wraps the
logic of writing buffers to ResultSubpartition
Key: FLINK-19312
URL: https://issues.apache.org/jira/browse/FLINK
Yingjie Cao created FLINK-19302:
---
Summary: Flushing of BoundedBlockingResultPartition should finish
current BufferBuilder
Key: FLINK-19302
URL: https://issues.apache.org/jira/browse/FLINK-19302
Project
Yingjie Cao created FLINK-19299:
---
Summary: NettyShuffleEnvironmentBuilder#setBufferSize does not
take effect
Key: FLINK-19299
URL: https://issues.apache.org/jira/browse/FLINK-19299
Project: Flink
Yingjie Cao created FLINK-19297:
---
Summary: Make ResultPartitionWriter record-oriented
Key: FLINK-19297
URL: https://issues.apache.org/jira/browse/FLINK-19297
Project: Flink
Issue Type
Yingjie Cao created FLINK-18762:
---
Summary: Make network buffers per incoming/outgoing channel can be
configured separately
Key: FLINK-18762
URL: https://issues.apache.org/jira/browse/FLINK-18762
Yingjie Cao created FLINK-18728:
---
Summary: Make initialCredit of RemoteInputChannel final
Key: FLINK-18728
URL: https://issues.apache.org/jira/browse/FLINK-18728
Project: Flink
Issue Type
Yingjie Cao created FLINK-18727:
---
Summary: Remove the previous finished empty Buffer in
PipelinedSubpartition when adding a new Buffer
Key: FLINK-18727
URL: https://issues.apache.org/jira/browse/FLINK-18727
Hi Thomas,
Thanks a lot for offering these information.
We have decided to try to reproduce the regression on AWS. It will be
really appreciated if you can share some demo code with us, and if it is
not convenient, could you give us some more information about the record
type and size, the
Hi Thomas,
I tried to reproduce the regression by constructing a Job with the same
topology, parallelism and checkpoint interval (Kinesis source and sink are
replaced for we do not have the test environment). But unfortunately, no
regression is observed both for back pressure and no back pressure
Yingjie Cao created FLINK-18453:
---
Summary: Stack overflow of
AggregateITCase#testAggregationCodeSplit on Azure
Key: FLINK-18453
URL: https://issues.apache.org/jira/browse/FLINK-18453
Project: Flink
Yingjie Cao created FLINK-18408:
---
Summary: BinaryRowData should use de-/serialization method of
MemorySegment with explicit endianness
Key: FLINK-18408
URL: https://issues.apache.org/jira/browse/FLINK-18408
Yingjie Cao created FLINK-17665:
---
Summary: Serialize buffer data type of Buffer into BufferResponse
Key: FLINK-17665
URL: https://issues.apache.org/jira/browse/FLINK-17665
Project: Flink
Issue
Yingjie Cao created FLINK-17572:
---
Summary: Remove checkpoint alignment buffered metric from webui
Key: FLINK-17572
URL: https://issues.apache.org/jira/browse/FLINK-17572
Project: Flink
Issue
Yingjie Cao created FLINK-17568:
---
Summary: Task may consume data after checkpoint barrier before
performing checkpoint for unaligned checkpoint
Key: FLINK-17568
URL: https://issues.apache.org/jira/browse/FLINK
Yingjie Cao created FLINK-17564:
---
Summary: Inflight data of incoming channel may be disordered for
unaligned checkpoint
Key: FLINK-17564
URL: https://issues.apache.org/jira/browse/FLINK-17564
Project
Yingjie Cao created FLINK-17299:
---
Summary: Add remote channel throughput benchmark
Key: FLINK-17299
URL: https://issues.apache.org/jira/browse/FLINK-17299
Project: Flink
Issue Type: Sub-task
Yingjie Cao created FLINK-17208:
---
Summary: Reduce redundant data available notification of
PipelinedSubpartition
Key: FLINK-17208
URL: https://issues.apache.org/jira/browse/FLINK-17208
Project: Flink
Yingjie Cao created FLINK-17107:
---
Summary: CheckpointCoordinatorConfiguration#isExactlyOnce() is
inconsistent with StreamConfig#getCheckpointMode()
Key: FLINK-17107
URL: https://issues.apache.org/jira/browse/FLINK
Congratulations!
Best,
Yingjie
Xintong Song 于2020年4月1日周三 下午6:27写道：
> Congratulations Konstantin, Dawid and Zhijiang~!
>
> Thank you~
>
> Xintong Song
>
>
>
> On Wed, Apr 1, 2020 at 6:19 PM Yu Li wrote:
>
> > Congratulations Konstantin, Dawid and Zhijiang! Well deserved.
> >
> > Best Regards,
+1 (non-binding)
Checkpoint timeout in cases of backpressure is hard to tune. I and our
users ever spent lots of time on that. It is great to have this feature.
Arvid Heise 于2020年3月10日周二 下午9:33写道：
> Hi all,
>
> I would like to start the vote for FLIP-76 [1], which is discussed and
> reached a
Yingjie Cao created FLINK-16298:
---
Summary:
GroupWindowTableAggregateITCase.testEventTimeTumblingWindow fails on Travis
Key: FLINK-16298
URL: https://issues.apache.org/jira/browse/FLINK-16298
Project
Yingjie Cao created FLINK-15455:
---
Summary: Enable TCP connection reuse across multiple jobs.
Key: FLINK-15455
URL: https://issues.apache.org/jira/browse/FLINK-15455
Project: Flink
Issue Type
Dec 18, 2019 at 8:21 AM Yingjie Cao
> wrote:
>
> > Hi Till & Biao,
> >
> > Thanks for the reply.
> >
> > I agree that supplying some stress or stability tests can really help,
> > except for the jvm resource leak mentioned above, there may be other t
e these kind of problems. Maybe
> one
> > could automate it so that a dev only needs to provide a user jar and then
> > this jar is being executed several times and the cluster is checked for
> > anomalies.
> >
> > Cheers,
> > Till
> >
> > On Tue, Dec 17,
Hi community,
After running tpc-ds test suite for several days on a session cluster, we
found a resource leak problem of OrcInputFormat which was reported in
FLINK-15239. The problem comes from the dependent third party library which
creates new internal thread (pool) and never release it. As a
Yingjie Cao created FLINK-15166:
---
Summary: Shuffle data compression wrongly decrease the buffer
reference count.
Key: FLINK-15166
URL: https://issues.apache.org/jira/browse/FLINK-15166
Project: Flink
Yingjie Cao created FLINK-15140:
---
Summary: Shuffle data compression does not work with
BroadcastRecordWriter.
Key: FLINK-15140
URL: https://issues.apache.org/jira/browse/FLINK-15140
Project: Flink
Yingjie Cao created FLINK-15030:
---
Summary: Potential deadlock for bounded blocking ResultPartition.
Key: FLINK-15030
URL: https://issues.apache.org/jira/browse/FLINK-15030
Project: Flink
Issue
Yingjie Cao created FLINK-14872:
---
Summary: Potential deadlock for task reading from blocking
ResultPartition.
Key: FLINK-14872
URL: https://issues.apache.org/jira/browse/FLINK-14872
Project: Flink
Yingjie Cao created FLINK-14845:
---
Summary: Introduce data compression to blocking shuffle.
Key: FLINK-14845
URL: https://issues.apache.org/jira/browse/FLINK-14845
Project: Flink
Issue Type
Yingjie Cao created FLINK-14185:
---
Summary: AbstractServerTest failed on Travis
Key: FLINK-14185
URL: https://issues.apache.org/jira/browse/FLINK-14185
Project: Flink
Issue Type: Bug
Yingjie Cao created FLINK-14180:
---
Summary: Enable config of maximum capacity of
FileArchivedExecutionGraphStore.
Key: FLINK-14180
URL: https://issues.apache.org/jira/browse/FLINK-14180
Project: Flink
Yingjie Cao created FLINK-14139:
---
Summary: Fix potential memory leak of rest server when using
session/standalone cluster
Key: FLINK-14139
URL: https://issues.apache.org/jira/browse/FLINK-14139
Project
Yingjie Cao created FLINK-14118:
---
Summary: Reduce the unnecessary flushing when there is no data
available for flush
Key: FLINK-14118
URL: https://issues.apache.org/jira/browse/FLINK-14118
Project
Congratulations Zhijiang!
yangtao.yt 于2019年7月23日周二 上午10:17写道：
> Congrats, Zhejiang!
>
> Best,
> Tao Yang
>
> > 在 2019年7月23日，上午9:46，boshu Zheng 写道：
> >
> > Congratulations Zhijiang
> >
> > 发自我的 iPhone
> >
> >> 在 2019年7月23日，上午12:55，Xuefu Z 写道：
> >>
> >> Congratulations, Zhijiang!
> >>
> >>> On
Yingjie Cao created FLINK-12329:
---
Summary: Netty thread deadlock bug of the SpilledSubpartitionView
Key: FLINK-12329
URL: https://issues.apache.org/jira/browse/FLINK-12329
Project: Flink
Issue
Yingjie Cao created FLINK-11859:
---
Summary: Improve SpanningRecordSerializer performance by
serializing record length to serialization buffer directly
Key: FLINK-11859
URL: https://issues.apache.org/jira/browse
Thanks, Yun.
If the external resource manager cannot fully obey the blacklist, then we
have two
choices. The first one is do nothing and using the returned resources
directly. The
other one is checking the returned resource, then releasing the unsatisfied
resource
and reallocating until success
You are right. I think, at least, we need a new interface to be implemented
to collect the failure information.
Best,
Yingjie
Guowei Ma 于2018年11月28日周三 下午2:23写道：
> thanks yingjie to share this doc and I think this is very important feature
> for production.
>
> As you mentioned in your
Thanks, Weihua.
Your suggestions make a lot of sense to me. Currently, all blacklisted
resources
will be released from blacklist if there is no available resource. Maybe
only releasing
a portion of the blacklisted resources based on the number of slots needed
and some
LRU like algorithm is a
Thanks Fabian.
Fabian Hueske 于2018年11月26日周一 下午6:03写道：
> Hi Yingjie,
>
> I've granted you contributor permissions.
>
> Best, Fabian
>
> Am Sa., 24. Nov. 2018 um 07:31 Uhr schrieb Yingjie Cao <
> kevin.ying...@gmail.com>:
>
> > Hi guys,
> >
&g
Hi guys,
Could anyone kindly give me the contributor permission? My JIRA ID is
kevin.cyj.
Thanks,
Yingjie
Yingjie Cao created FLINK-11000:
---
Summary: Introduce Resource Blacklist Mechanism
Key: FLINK-11000
URL: https://issues.apache.org/jira/browse/FLINK-11000
Project: Flink
Issue Type: Improvement
101 - 154 of 154 matches
Mail list logo
