GoGoWen opened a new issue, #139: URL: https://github.com/apache/doris-flink-connector/issues/139
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version version: 1.14_2.11-1.1.1 ### What's Wrong? flink interrupted with multi-thread issue. the root cause is dorisStreamLoad.handlePreCommitResponse handling the same future [details refer to steps in repro]. please refer to the log: 2023-04-24 05:09:45.779 INFO [Sink doris-sink-minu: (13/20)#0] org.apache.doris.flink.sink.writer.DorisStreamLoad - load Result { "TxnId": 19257242, "Label": "minutely_zt_video_effects_1682243486014_12_9760", "TwoPhaseCommit": "true", "Status": "Success", "Message": "OK", "NumberTotalRows": 0, "NumberLoadedRows": 0, "NumberFilteredRows": 0, "NumberUnselectedRows": 0, "LoadBytes": 0, "LoadTimeMs": 30053, "BeginTxnTimeMs": 0, "StreamLoadPutTimeMs": 21, "ReadDataTimeMs": 0, "WriteDataTimeMs": 29957, "CommitAndPublishTimeMs": 0 } 2023-04-24 05:09:45.779 ERROR [stream-load-check-th:k-thread-1] org.apache.doris.flink.sink.writer.DorisWriter - stream load finished unexpectedly, interrupt worker thread! Stream closed 2023-04-24 05:09:45.779 INFO [Sink doris-sink-minu: (13/20)#0] org.apache.doris.flink.sink.writer.RecordBuffer - start buffer data, read queue size 0, write queue size 400 2023-04-24 05:09:45.779 INFO [Sink doris-sink-minu: (13/20)#0] org.apache.doris.flink.sink.writer.DorisStreamLoad - stream load started for minutely_zt_video_effects_1682243486014_12_9761 2023-04-24 05:09:45.779 INFO [stream-load-upload-t:d-thread-1] org.apache.doris.flink.sink.writer.DorisStreamLoad - start execute load 2023-04-24 05:09:45.786 INFO [stream-load-upload-t:d-thread-1] org.apache.http.impl.execchain.RetryExec - I/O exception (java.net.SocketException) caught when processing request to {}->http://11.148.133.129:8240: Socket is closed 2023-04-24 05:09:45.786 INFO [stream-load-upload-t:d-thread-1] org.apache.http.impl.execchain.RetryExec - Retrying request to {}->http://11.148.133.129:8240 2023-04-24 05:09:45.789 INFO [Sink doris-sink-dail: (13/20)#0] org.apache.doris.flink.sink.writer.DorisStreamLoad - load Result { the log also may be: stream closed 2023-04-24 05:09:45.779 INFO [Sink doris-sink-minu: (13/20)#0] org.apache.doris.flink.sink.writer.DorisStreamLoad - load Result { "TxnId": 19257242, "Label": "minutely_zt_video_effects_1682243486014_12_9760", "TwoPhaseCommit": "true", "Status": "Success", "Message": "OK", "NumberTotalRows": 0, "NumberLoadedRows": 0, "NumberFilteredRows": 0, "NumberUnselectedRows": 0, "LoadBytes": 0, "LoadTimeMs": 30053, "BeginTxnTimeMs": 0, "StreamLoadPutTimeMs": 21, "ReadDataTimeMs": 0, "WriteDataTimeMs": 29957, "CommitAndPublishTimeMs": 0 } 2023-04-24 05:09:45.779 ERROR [stream-load-check-th:k-thread-1] org.apache.doris.flink.sink.writer.DorisWriter - stream load finished unexpectedly, interrupt worker thread! Stream closed 2023-04-24 05:09:45.779 INFO [Sink doris-sink-minu: (13/20)#0] org.apache.doris.flink.sink.writer.RecordBuffer - start buffer data, read queue size 0, write queue size 400 2023-04-24 05:09:45.779 INFO [Sink doris-sink-minu: (13/20)#0] org.apache.doris.flink.sink.writer.DorisStreamLoad - stream load started for minutely_zt_video_effects_1682243486014_12_9761 2023-04-24 05:09:45.779 INFO [stream-load-upload-t:d-thread-1] org.apache.doris.flink.sink.writer.DorisStreamLoad - start execute load 2023-04-24 05:09:45.786 INFO [stream-load-upload-t:d-thread-1] org.apache.http.impl.execchain.RetryExec - I/O exception (java.net.SocketException) caught when processing request to {}->http://11.148.133.129:8240: Socket is closed 2023-04-24 05:09:45.786 INFO [stream-load-upload-t:d-thread-1] org.apache.http.impl.execchain.RetryExec - Retrying request to {}->http://11.148.133.129:8240 2023-04-24 05:09:45.789 INFO [Sink doris-sink-dail: (13/20)#0] org.apache.doris.flink.sink.writer.DorisStreamLoad - load Result { ### What You Expected? works well. ### How to Reproduce? steps to reproduce：  1， main thread call prepareCommit , set loading =false to stop the load-1，http request return success，this time dorisStreamLoad.getPendingLoadFuture().isDone() is true. now the checker thread start to run. 2， checker thread run to "if (dorisStreamLoad.getPendingLoadFuture() != null&& dorisStreamLoad.getPendingLoadFuture().isDone())" ，now this check pass like step 1； then main thread continue to start a new load-2, set the loading =true and new Future. turn to checker thread, the thread continue to "if(!loading)", currently this check also will pass, then checker thread continue to "RespContent content = dorisStreamLoad.handlePreCommitResponse(dorisStreamLoad.getPendingLoadFuture().get());", as loading is running, it blocked, will be wait loading to finish. 3， after a while, main thread call prepareCommit to finish load-2, the future return, wake up main thread and checker thread to process response concurrently, the checker will interrupt itself. ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
