Re: [PR] [HUDI-7606] Unpersist RDDs after table services, mainly compaction [hudi]
hudi-bot commented on PR #11000: URL: https://github.com/apache/hudi/pull/11000#issuecomment-2051442199 ## CI report: * 6c81f312f55df6d28363cb836202aa8ec7173a3e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23215) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7606] Unpersist RDDs after table services, mainly compaction [hudi]
nsivabalan commented on code in PR #11000: URL: https://github.com/apache/hudi/pull/11000#discussion_r1562262882 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java: ## @@ -268,6 +268,7 @@ public boolean commitStats(String instantTime, HoodieData writeStat commitCallback.call(new HoodieWriteCommitCallbackMessage( instantTime, config.getTableName(), config.getBasePath(), stats, Option.of(commitActionType), extraMetadata)); } +releaseResources(instantTime); Review Comment: this covers all writers correct? spark ds writer, and deltastreamer? ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java: ## @@ -268,6 +268,7 @@ public boolean commitStats(String instantTime, HoodieData writeStat commitCallback.call(new HoodieWriteCommitCallbackMessage( instantTime, config.getTableName(), config.getBasePath(), stats, Option.of(commitActionType), extraMetadata)); } +releaseResources(instantTime); Review Comment: this code path covers all writers correct? spark ds writer, and deltastreamer? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7606] Unpersist RDDs after table services, mainly compaction [hudi]
hudi-bot commented on PR #11000: URL: https://github.com/apache/hudi/pull/11000#issuecomment-2051329377 ## CI report: * 12cf06d732847bf9ca925bf2bb4e2e0eb39b8855 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23205) * 6c81f312f55df6d28363cb836202aa8ec7173a3e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23215) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7606] Unpersist RDDs after table services, mainly compaction [hudi]
hudi-bot commented on PR #11000: URL: https://github.com/apache/hudi/pull/11000#issuecomment-2051316642 ## CI report: * 12cf06d732847bf9ca925bf2bb4e2e0eb39b8855 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23205) * 6c81f312f55df6d28363cb836202aa8ec7173a3e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7606] Unpersist RDDs after table services, mainly compaction [hudi]
rmahindra123 commented on code in PR #11000:
URL: https://github.com/apache/hudi/pull/11000#discussion_r1562199875
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##
@@ -236,8 +236,8 @@ public boolean commitStats(String instantTime,
HoodieData writeStat
commit(table, commitActionType, instantTime, metadata, stats,
writeStatuses);
postCommit(table, metadata, instantTime, extraMetadata);
LOG.info("Committed " + instantTime);
- releaseResources(instantTime);
} catch (IOException e) {
+ releaseResources(instantTime);
Review Comment:
makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7606] Unpersist RDDs after table services, mainly compaction [hudi]
nsivabalan commented on code in PR #11000:
URL: https://github.com/apache/hudi/pull/11000#discussion_r1562194553
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##
@@ -236,8 +236,8 @@ public boolean commitStats(String instantTime,
HoodieData writeStat
commit(table, commitActionType, instantTime, metadata, stats,
writeStatuses);
postCommit(table, metadata, instantTime, extraMetadata);
LOG.info("Committed " + instantTime);
- releaseResources(instantTime);
} catch (IOException e) {
+ releaseResources(instantTime);
Review Comment:
gotcha. then shouldn't we try to do the same at L255
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7606] Unpersist RDDs after table services, mainly compaction [hudi]
rmahindra123 commented on code in PR #11000:
URL: https://github.com/apache/hudi/pull/11000#discussion_r1562176259
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##
@@ -236,8 +236,8 @@ public boolean commitStats(String instantTime,
HoodieData writeStat
commit(table, commitActionType, instantTime, metadata, stats,
writeStatuses);
postCommit(table, metadata, instantTime, extraMetadata);
LOG.info("Committed " + instantTime);
- releaseResources(instantTime);
} catch (IOException e) {
+ releaseResources(instantTime);
Review Comment:
we want to release resources after table services. I added it to the catch
block in case of early exit. Makes sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7606] Unpersist RDDs after table services, mainly compaction [hudi]
rmahindra123 commented on code in PR #11000:
URL: https://github.com/apache/hudi/pull/11000#discussion_r1562176259
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##
@@ -236,8 +236,8 @@ public boolean commitStats(String instantTime,
HoodieData writeStat
commit(table, commitActionType, instantTime, metadata, stats,
writeStatuses);
postCommit(table, metadata, instantTime, extraMetadata);
LOG.info("Committed " + instantTime);
- releaseResources(instantTime);
} catch (IOException e) {
+ releaseResources(instantTime);
Review Comment:
we want to release resources after table services
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7606] Unpersist RDDs after table services, mainly compaction [hudi]
nsivabalan commented on code in PR #11000:
URL: https://github.com/apache/hudi/pull/11000#discussion_r1562161402
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##
@@ -236,8 +236,8 @@ public boolean commitStats(String instantTime,
HoodieData writeStat
commit(table, commitActionType, instantTime, metadata, stats,
writeStatuses);
postCommit(table, metadata, instantTime, extraMetadata);
LOG.info("Committed " + instantTime);
- releaseResources(instantTime);
} catch (IOException e) {
+ releaseResources(instantTime);
Review Comment:
why not move releaseResources to finally block ? and so we can avoid L 260
right ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7606] Unpersist RDDs after table services, mainly compaction [hudi]
hudi-bot commented on PR #11000: URL: https://github.com/apache/hudi/pull/11000#issuecomment-2050873544 ## CI report: * 12cf06d732847bf9ca925bf2bb4e2e0eb39b8855 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23205) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7606] Unpersist RDDs after table services, mainly compaction [hudi]
hudi-bot commented on PR #11000: URL: https://github.com/apache/hudi/pull/11000#issuecomment-2050777340 ## CI report: * 12cf06d732847bf9ca925bf2bb4e2e0eb39b8855 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23205) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7606] Unpersist RDDs after table services, mainly compaction [hudi]
hudi-bot commented on PR #11000: URL: https://github.com/apache/hudi/pull/11000#issuecomment-2050771888 ## CI report: * 12cf06d732847bf9ca925bf2bb4e2e0eb39b8855 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
