Re: [PR] [fix](Export) Fix the problem of exporting stuck [doris]
morningman merged PR #44944: URL: https://github.com/apache/doris/pull/44944 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Export) Fix the problem of exporting stuck [doris]
github-actions[bot] commented on PR #44944: URL: https://github.com/apache/doris/pull/44944#issuecomment-2522122916 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Export) Fix the problem of exporting stuck [doris]
github-actions[bot] commented on PR #44944: URL: https://github.com/apache/doris/pull/44944#issuecomment-2522122964 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Export) Fix the problem of exporting stuck [doris]
BePPPower commented on PR #44944: URL: https://github.com/apache/doris/pull/44944#issuecomment-2519428906 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Export) Fix the problem of exporting stuck [doris]
morningman commented on code in PR #44944:
URL: https://github.com/apache/doris/pull/44944#discussion_r1869857965
##
fe/fe-core/src/main/java/org/apache/doris/load/ExportMgr.java:
##
@@ -108,26 +108,24 @@ public void addExportJobAndRegisterTask(ExportJob job)
throws Exception {
}
}
unprotectAddJob(job);
-// delete existing files
-if (Config.enable_delete_existing_files &&
Boolean.parseBoolean(job.getDeleteExistingFiles())) {
-if (job.getBrokerDesc() == null) {
-throw new AnalysisException("Local file system does not
support delete existing files");
-}
-String fullPath = job.getExportPath();
-BrokerUtil.deleteDirectoryWithFileSystem(fullPath.substring(0,
fullPath.lastIndexOf('/') + 1),
-job.getBrokerDesc());
-}
-Env.getCurrentEnv().getEditLog().logExportCreate(job);
-// ATTN: Must add task after edit log, otherwise the job may
finish before adding job.
-job.getCopiedTaskExecutors().forEach(executor -> {
-
Env.getCurrentEnv().getTransientTaskManager().addMemoryTask(executor);
-});
-LOG.info("add export job. {}", job);
-
} finally {
writeUnlock();
}
-
+// delete existing files
+if (Config.enable_delete_existing_files &&
Boolean.parseBoolean(job.getDeleteExistingFiles())) {
+if (job.getBrokerDesc() == null) {
+throw new AnalysisException("Local file system does not
support delete existing files");
+}
+String fullPath = job.getExportPath();
+BrokerUtil.deleteDirectoryWithFileSystem(fullPath.substring(0,
fullPath.lastIndexOf('/') + 1),
+job.getBrokerDesc());
+}
+Env.getCurrentEnv().getEditLog().logExportCreate(job);
Review Comment:
I think `Env.getCurrentEnv().getEditLog().logExportCreate(job);` should be
done within the lock.
1. unprotectAddJob(job);
2. Env.getCurrentEnv().getEditLog().logExportCreate(job);
3. delete existing files
4. addMemoryTask
##
fe/fe-core/src/main/java/org/apache/doris/scheduler/disruptor/TaskDisruptor.java:
##
@@ -119,15 +120,15 @@ public void tryPublish(Long jobId, Long taskId, TaskType
taskType) {
*
* @param taskId task id
*/
-public void tryPublishTask(Long taskId) {
+public void tryPublishTask(Long taskId) throws JobException {
if (isClosed) {
log.info("tryPublish failed, disruptor is closed, taskId: {}",
taskId);
return;
}
-try {
+if (disruptor.getRingBuffer().hasAvailableCapacity(2)) {
Review Comment:
Add comment to explain what does `2` mean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Export) Fix the problem of exporting stuck [doris]
BePPPower commented on PR #44944: URL: https://github.com/apache/doris/pull/44944#issuecomment-2517127923 run cloud_p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](Export) Fix the problem of exporting stuck [doris]
BePPPower opened a new pull request, #44944: URL: https://github.com/apache/doris/pull/44944 ### What problem does this PR solve? Problem Summary: The `disruptor` will be witing when the ringbuffer does not have enough capacity. At the same time, `addExportJobAndRegisterTask` will not release the lock of `ExportMgr`. This prevents other methods from obtaining the lock. ### Release note ### Check List (For Author) - Test - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [x] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [x] Other reason - Behavior changed: - [x] No. - [ ] Yes. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Export) Fix the problem of exporting stuck [doris]
doris-robot commented on PR #44944: URL: https://github.com/apache/doris/pull/44944#issuecomment-2514410255 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Export) Fix the problem of exporting stuck [doris]
BePPPower commented on PR #44944: URL: https://github.com/apache/doris/pull/44944#issuecomment-2514410337 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
