[
https://issues.apache.org/jira/browse/FLINK-8020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weihua Jiang updated FLINK-8020:
--------------------------------
Attachment: jstack53009(2).out
Attach a new jstack file.
After removing our collector.collect() call out of our lock region, the
deadlock was gone. However, the system is still NOT able to run as we find that
flink collector hangs.
See the attached jstack 53009(2).out file, the thread "cache-process0 ->
async-operator0 -> Sink: hbase-sink0 (1/10)" holds the lock 0x00000007b692ad98
and is still waiting for THIS exact lock again. This makes it hang.
Our guess is that this is some issue related to async io. After changing our
async redis lookup to sync redis lookup (by not using async io), the issue is
gone.
> Deadlock found in Flink Streaming job
> -------------------------------------
>
> Key: FLINK-8020
> URL: https://issues.apache.org/jira/browse/FLINK-8020
> Project: Flink
> Issue Type: Bug
> Components: Kafka Connector, Streaming, Streaming Connectors
> Affects Versions: 1.3.2
> Environment: Kafka 0.8.2 and Flink 1.3.2 on YARN mode
> Reporter: Weihua Jiang
> Priority: Blocker
> Attachments: jstack53009(2).out, jstack67976-2.log
>
>
> Our streaming job run into trouble in these days after a long time smooth
> running. One issue we found is
> [https://issues.apache.org/jira/browse/FLINK-8019] and another one is this
> one.
> After analyzing the jstack, we believe we found a DEAD LOCK in flink:
> 1. The thread "cache-process0 -> async-operator0 -> Sink: hbase-sink0 (8/8)"
> hold lock 0x00000007b6aa1788 and is waiting for lock 0x00000007b6aa1940.
> 2. The thread "Time Trigger for cache-process0 -> async-operator0 -> Sink:
> hbase-sink0 (8/8)" hold lock 0x00000007b6aa1940 and is waiting for lock
> 0x00000007b6aa1788.
> This DEADLOCK made the job fail to proceed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
