Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1913949435 ## CI report: * 05bed31829f2362de479344215d29ccca99bd449 UNKNOWN * c809e66dd23358183eacb62c16d7d8f1488543cf Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22191) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1913870779 ## CI report: * 05bed31829f2362de479344215d29ccca99bd449 UNKNOWN * 855ede2626b0c95d0649a939aad51de277562fa5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21782) * c809e66dd23358183eacb62c16d7d8f1488543cf Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22191) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1913865773 ## CI report: * 05bed31829f2362de479344215d29ccca99bd449 UNKNOWN * 855ede2626b0c95d0649a939aad51de277562fa5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21782) * c809e66dd23358183eacb62c16d7d8f1488543cf UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1875114385 @bvaradar could you give some review and whether need more changes? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1873877127 ## CI report: * 05bed31829f2362de479344215d29ccca99bd449 UNKNOWN * 855ede2626b0c95d0649a939aad51de277562fa5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21782) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1873678295 ## CI report: * 05bed31829f2362de479344215d29ccca99bd449 UNKNOWN * 0bf07811af39bc66fa42ea2fe0fee65e3525f26c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21767) * 855ede2626b0c95d0649a939aad51de277562fa5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21782) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1873674110 ## CI report: * 05bed31829f2362de479344215d29ccca99bd449 UNKNOWN * 0bf07811af39bc66fa42ea2fe0fee65e3525f26c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21767) * 855ede2626b0c95d0649a939aad51de277562fa5 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1439187089
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##
@@ -765,6 +765,14 @@ public class HoodieWriteConfig extends HoodieConfig {
.withDocumentation("Avro schema of the partial updates. This is
automatically set by the "
+ "Hudi write client and user is not expected to manually change the
value.");
+ public static final ConfigProperty WRITE_IGNORE_FAILED =
ConfigProperty
Review Comment:
@bvaradar @danny0405 Had fix it in spark layer，and leave the config for
it，PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1439135524
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##
@@ -765,6 +765,14 @@ public class HoodieWriteConfig extends HoodieConfig {
.withDocumentation("Avro schema of the partial updates. This is
automatically set by the "
+ "Hudi write client and user is not expected to manually change the
value.");
+ public static final ConfigProperty WRITE_IGNORE_FAILED =
ConfigProperty
Review Comment:
@bvaradar OK，if so I would handle this in hudi-spark layer，that seems handle
this in a better way. but may alse need a switch config for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1439135524
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##
@@ -765,6 +765,14 @@ public class HoodieWriteConfig extends HoodieConfig {
.withDocumentation("Avro schema of the partial updates. This is
automatically set by the "
+ "Hudi write client and user is not expected to manually change the
value.");
+ public static final ConfigProperty WRITE_IGNORE_FAILED =
ConfigProperty
Review Comment:
@bvaradar OK，if so I would handle this in hudi-spark layer，that seems handle
this in a better way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1438954814
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##
@@ -765,6 +765,14 @@ public class HoodieWriteConfig extends HoodieConfig {
.withDocumentation("Avro schema of the partial updates. This is
automatically set by the "
+ "Hudi write client and user is not expected to manually change the
value.");
+ public static final ConfigProperty WRITE_IGNORE_FAILED =
ConfigProperty
Review Comment:
@danny0405 also need to be changed or keep the pr current logic to it？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1438954814
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##
@@ -765,6 +765,14 @@ public class HoodieWriteConfig extends HoodieConfig {
.withDocumentation("Avro schema of the partial updates. This is
automatically set by the "
+ "Hudi write client and user is not expected to manually change the
value.");
+ public static final ConfigProperty WRITE_IGNORE_FAILED =
ConfigProperty
Review Comment:
@danny0405 also need to be changed or keep the pr current logic to it？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1438475893
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##
@@ -765,6 +765,14 @@ public class HoodieWriteConfig extends HoodieConfig {
.withDocumentation("Avro schema of the partial updates. This is
automatically set by the "
+ "Hudi write client and user is not expected to manually change the
value.");
+ public static final ConfigProperty WRITE_IGNORE_FAILED =
ConfigProperty
Review Comment:
Ok，I would try to do the change@bvaradar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
danny0405 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1438949910
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##
@@ -765,6 +765,14 @@ public class HoodieWriteConfig extends HoodieConfig {
.withDocumentation("Avro schema of the partial updates. This is
automatically set by the "
+ "Hudi write client and user is not expected to manually change the
value.");
+ public static final ConfigProperty WRITE_IGNORE_FAILED =
ConfigProperty
Review Comment:
Makes sense, Flink also has the duplicate logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1438475893
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##
@@ -765,6 +765,14 @@ public class HoodieWriteConfig extends HoodieConfig {
.withDocumentation("Avro schema of the partial updates. This is
automatically set by the "
+ "Hudi write client and user is not expected to manually change the
value.");
+ public static final ConfigProperty WRITE_IGNORE_FAILED =
ConfigProperty
Review Comment:
Ok，I would try to do the change@bvaradar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
bvaradar commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1438474702
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##
@@ -765,6 +765,14 @@ public class HoodieWriteConfig extends HoodieConfig {
.withDocumentation("Avro schema of the partial updates. This is
automatically set by the "
+ "Hudi write client and user is not expected to manually change the
value.");
+ public static final ConfigProperty WRITE_IGNORE_FAILED =
ConfigProperty
Review Comment:
@xuzifu666 : This is handled by the Hudi layer above core part. For example
: In HoodieStreamer, we have this flag --commit-on-errors. Also Spark writers
handles this way :
https://github.com/apache/hudi/blob/f39cb1d2e3e5eece1542252316f4135cec83299b/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala#L956
. Adding this config would make the other configs and related code redundant.
Can you look into exploring those sections of code more and handling this at
the hudi-spark layer ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1438464674
##
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##
@@ -292,6 +294,9 @@ protected void
setCommitMetadata(HoodieWriteMetadata> re
@Override
protected void commit(HoodieWriteMetadata> result) {
context.setJobStatus(this.getClass().getSimpleName(), "Commit write status
collect: " + config.getTableName());
+if (!config.getBoolean(IGNORE_ERROR_WRITE) &&
result.getWriteStatuses().filter(status -> status.getErrors().size() >
0).count() > 0) {
+ throw new HoodieException("Error writing record in the job");
+}
commit(result.getWriteStatuses(), result,
result.getWriteStats().isPresent()
Review Comment:
@danny0405 PATL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1872139372 ## CI report: * 05bed31829f2362de479344215d29ccca99bd449 UNKNOWN * 0bf07811af39bc66fa42ea2fe0fee65e3525f26c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21767) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1871978751 ## CI report: * 05bed31829f2362de479344215d29ccca99bd449 UNKNOWN * e16c1436b39b856f47564d7c02fb09c3ffdf8e0b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21765) * 0bf07811af39bc66fa42ea2fe0fee65e3525f26c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21767) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1871948621 ## CI report: * 05bed31829f2362de479344215d29ccca99bd449 UNKNOWN * c03b1658a59d5064acc05cf9a92b6130999f7ee6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21762) * e16c1436b39b856f47564d7c02fb09c3ffdf8e0b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21765) * 0bf07811af39bc66fa42ea2fe0fee65e3525f26c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21767) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1871943188 ## CI report: * 05bed31829f2362de479344215d29ccca99bd449 UNKNOWN * c03b1658a59d5064acc05cf9a92b6130999f7ee6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21762) * e16c1436b39b856f47564d7c02fb09c3ffdf8e0b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21765) * 0bf07811af39bc66fa42ea2fe0fee65e3525f26c UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1871937709 ## CI report: * 05bed31829f2362de479344215d29ccca99bd449 UNKNOWN * 814e79d8e5f756f0dac433164a0862a426874ac0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21745) * c03b1658a59d5064acc05cf9a92b6130999f7ee6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21762) * e16c1436b39b856f47564d7c02fb09c3ffdf8e0b UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1871852318 ## CI report: * 05bed31829f2362de479344215d29ccca99bd449 UNKNOWN * 814e79d8e5f756f0dac433164a0862a426874ac0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21745) * c03b1658a59d5064acc05cf9a92b6130999f7ee6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21762) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1871847811 ## CI report: * 05bed31829f2362de479344215d29ccca99bd449 UNKNOWN * 814e79d8e5f756f0dac433164a0862a426874ac0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21745) * c03b1658a59d5064acc05cf9a92b6130999f7ee6 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1438091481
##
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##
@@ -292,6 +294,9 @@ protected void
setCommitMetadata(HoodieWriteMetadata> re
@Override
protected void commit(HoodieWriteMetadata> result) {
context.setJobStatus(this.getClass().getSimpleName(), "Commit write status
collect: " + config.getTableName());
+if (!config.getBoolean(IGNORE_ERROR_WRITE) &&
result.getWriteStatuses().filter(status -> status.getErrors().size() >
0).count() > 0) {
+ throw new HoodieException("Error writing record in the job");
+}
commit(result.getWriteStatuses(), result,
result.getWriteStats().isPresent()
Review Comment:
OK，fix in BaseCommitActionExecutor::commit Method @danny0405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1438091481
##
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##
@@ -292,6 +294,9 @@ protected void
setCommitMetadata(HoodieWriteMetadata> re
@Override
protected void commit(HoodieWriteMetadata> result) {
context.setJobStatus(this.getClass().getSimpleName(), "Commit write status
collect: " + config.getTableName());
+if (!config.getBoolean(IGNORE_ERROR_WRITE) &&
result.getWriteStatuses().filter(status -> status.getErrors().size() >
0).count() > 0) {
+ throw new HoodieException("Error writing record in the job");
+}
commit(result.getWriteStatuses(), result,
result.getWriteStats().isPresent()
Review Comment:
ok fix in BaseCommitActionExecutor::commit Method @danny0405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1437448466
##
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##
@@ -292,6 +294,9 @@ protected void
setCommitMetadata(HoodieWriteMetadata> re
@Override
protected void commit(HoodieWriteMetadata> result) {
context.setJobStatus(this.getClass().getSimpleName(), "Commit write status
collect: " + config.getTableName());
+if (!config.getBoolean(IGNORE_ERROR_WRITE) &&
result.getWriteStatuses().filter(status -> status.getErrors().size() >
0).count() > 0) {
+ throw new HoodieException("Error writing record in the job");
+}
commit(result.getWriteStatuses(), result,
result.getWriteStats().isPresent()
Review Comment:
fix it with: ignore error writestaus by filter them in default，then continue
commit logic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1438079654
##
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##
@@ -292,8 +294,16 @@ protected void
setCommitMetadata(HoodieWriteMetadata> re
@Override
protected void commit(HoodieWriteMetadata> result) {
context.setJobStatus(this.getClass().getSimpleName(), "Commit write status
collect: " + config.getTableName());
-commit(result.getWriteStatuses(), result,
result.getWriteStats().isPresent()
-? result.getWriteStats().get() :
result.getWriteStatuses().map(WriteStatus::getStat).collectAsList());
+if (!config.getBoolean(WRITE_IGNORE_FAILED) &&
result.getWriteStatuses().filter(status -> status.getErrors().size() >
0).count() > 0) {
Review Comment:
OK，move it latter~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
beyond1920 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1438079373
##
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##
@@ -292,8 +294,16 @@ protected void
setCommitMetadata(HoodieWriteMetadata> re
@Override
protected void commit(HoodieWriteMetadata> result) {
context.setJobStatus(this.getClass().getSimpleName(), "Commit write status
collect: " + config.getTableName());
-commit(result.getWriteStatuses(), result,
result.getWriteStats().isPresent()
-? result.getWriteStats().get() :
result.getWriteStatuses().map(WriteStatus::getStat).collectAsList());
+if (!config.getBoolean(WRITE_IGNORE_FAILED) &&
result.getWriteStatuses().filter(status -> status.getErrors().size() >
0).count() > 0) {
Review Comment:
How about move those fix logical into
`BaseCommitActionExecutor#commit(HoodieData writeStatuses,
HoodieWriteMetadata result, List writeStats)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1438077824
##
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##
@@ -292,8 +294,16 @@ protected void
setCommitMetadata(HoodieWriteMetadata> re
@Override
protected void commit(HoodieWriteMetadata> result) {
context.setJobStatus(this.getClass().getSimpleName(), "Commit write status
collect: " + config.getTableName());
-commit(result.getWriteStatuses(), result,
result.getWriteStats().isPresent()
-? result.getWriteStats().get() :
result.getWriteStatuses().map(WriteStatus::getStat).collectAsList());
+if (!config.getBoolean(WRITE_IGNORE_FAILED) &&
result.getWriteStatuses().filter(status -> status.getErrors().size() >
0).count() > 0) {
+ WriteStatus writeStatus = result.getWriteStatuses().filter(status ->
status.getErrors().size() > 0).collectAsList().get(0);
+ HashMap errors = writeStatus.getErrors();
+ // get first one of error info to show error details
+ HoodieKey key = errors.keySet().iterator().next();
+ throw new HoodieException("Error writing record in the job",
errors.get(key));
+}
+HoodieData filterResult =
result.getWriteStatuses().filter(status -> status.getErrors().size() == 0);
+commit(filterResult, result, result.getWriteStats().isPresent()
Review Comment:
Yeah，+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
beyond1920 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1438077207
##
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##
@@ -292,8 +294,16 @@ protected void
setCommitMetadata(HoodieWriteMetadata> re
@Override
protected void commit(HoodieWriteMetadata> result) {
context.setJobStatus(this.getClass().getSimpleName(), "Commit write status
collect: " + config.getTableName());
-commit(result.getWriteStatuses(), result,
result.getWriteStats().isPresent()
-? result.getWriteStats().get() :
result.getWriteStatuses().map(WriteStatus::getStat).collectAsList());
+if (!config.getBoolean(WRITE_IGNORE_FAILED) &&
result.getWriteStatuses().filter(status -> status.getErrors().size() >
0).count() > 0) {
+ WriteStatus writeStatus = result.getWriteStatuses().filter(status ->
status.getErrors().size() > 0).collectAsList().get(0);
+ HashMap errors = writeStatus.getErrors();
+ // get first one of error info to show error details
+ HoodieKey key = errors.keySet().iterator().next();
+ throw new HoodieException("Error writing record in the job",
errors.get(key));
+}
+HoodieData filterResult =
result.getWriteStatuses().filter(status -> status.getErrors().size() == 0);
+commit(filterResult, result, result.getWriteStats().isPresent()
Review Comment:
Yes.
Even though a file including some write errors, it might also including
other valid records. Filters out those files is not reasonable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1438075209
##
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##
@@ -292,8 +294,16 @@ protected void
setCommitMetadata(HoodieWriteMetadata> re
@Override
protected void commit(HoodieWriteMetadata> result) {
context.setJobStatus(this.getClass().getSimpleName(), "Commit write status
collect: " + config.getTableName());
-commit(result.getWriteStatuses(), result,
result.getWriteStats().isPresent()
-? result.getWriteStats().get() :
result.getWriteStatuses().map(WriteStatus::getStat).collectAsList());
+if (!config.getBoolean(WRITE_IGNORE_FAILED) &&
result.getWriteStatuses().filter(status -> status.getErrors().size() >
0).count() > 0) {
+ WriteStatus writeStatus = result.getWriteStatuses().filter(status ->
status.getErrors().size() > 0).collectAsList().get(0);
+ HashMap errors = writeStatus.getErrors();
+ // get first one of error info to show error details
+ HoodieKey key = errors.keySet().iterator().next();
+ throw new HoodieException("Error writing record in the job",
errors.get(key));
+}
+HoodieData filterResult =
result.getWriteStatuses().filter(status -> status.getErrors().size() == 0);
+commit(filterResult, result, result.getWriteStats().isPresent()
Review Comment:
seems not need do this operation，I would revert it @beyond1920
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1438075042
##
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##
@@ -292,8 +294,16 @@ protected void
setCommitMetadata(HoodieWriteMetadata> re
@Override
protected void commit(HoodieWriteMetadata> result) {
context.setJobStatus(this.getClass().getSimpleName(), "Commit write status
collect: " + config.getTableName());
-commit(result.getWriteStatuses(), result,
result.getWriteStats().isPresent()
-? result.getWriteStats().get() :
result.getWriteStatuses().map(WriteStatus::getStat).collectAsList());
+if (!config.getBoolean(WRITE_IGNORE_FAILED) &&
result.getWriteStatuses().filter(status -> status.getErrors().size() >
0).count() > 0) {
+ WriteStatus writeStatus = result.getWriteStatuses().filter(status ->
status.getErrors().size() > 0).collectAsList().get(0);
+ HashMap errors = writeStatus.getErrors();
+ // get first one of error info to show error details
+ HoodieKey key = errors.keySet().iterator().next();
Review Comment:
OK，I would add more infomation for it，Thanks~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
beyond1920 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1438072890
##
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##
@@ -292,8 +294,16 @@ protected void
setCommitMetadata(HoodieWriteMetadata> re
@Override
protected void commit(HoodieWriteMetadata> result) {
context.setJobStatus(this.getClass().getSimpleName(), "Commit write status
collect: " + config.getTableName());
-commit(result.getWriteStatuses(), result,
result.getWriteStats().isPresent()
-? result.getWriteStats().get() :
result.getWriteStatuses().map(WriteStatus::getStat).collectAsList());
+if (!config.getBoolean(WRITE_IGNORE_FAILED) &&
result.getWriteStatuses().filter(status -> status.getErrors().size() >
0).count() > 0) {
+ WriteStatus writeStatus = result.getWriteStatuses().filter(status ->
status.getErrors().size() > 0).collectAsList().get(0);
+ HashMap errors = writeStatus.getErrors();
+ // get first one of error info to show error details
+ HoodieKey key = errors.keySet().iterator().next();
Review Comment:
The exception information is not enough.
If get the exception, we should provide more logs to find out what happened.
For example, which error happened when writing to which files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
beyond1920 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1438074169
##
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##
@@ -292,8 +294,16 @@ protected void
setCommitMetadata(HoodieWriteMetadata> re
@Override
protected void commit(HoodieWriteMetadata> result) {
context.setJobStatus(this.getClass().getSimpleName(), "Commit write status
collect: " + config.getTableName());
-commit(result.getWriteStatuses(), result,
result.getWriteStats().isPresent()
-? result.getWriteStats().get() :
result.getWriteStatuses().map(WriteStatus::getStat).collectAsList());
+if (!config.getBoolean(WRITE_IGNORE_FAILED) &&
result.getWriteStatuses().filter(status -> status.getErrors().size() >
0).count() > 0) {
+ WriteStatus writeStatus = result.getWriteStatuses().filter(status ->
status.getErrors().size() > 0).collectAsList().get(0);
+ HashMap errors = writeStatus.getErrors();
+ // get first one of error info to show error details
+ HoodieKey key = errors.keySet().iterator().next();
+ throw new HoodieException("Error writing record in the job",
errors.get(key));
+}
+HoodieData filterResult =
result.getWriteStatuses().filter(status -> status.getErrors().size() == 0);
+commit(filterResult, result, result.getWriteStats().isPresent()
Review Comment:
Why filter out write status with errors?
##
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##
@@ -292,8 +294,16 @@ protected void
setCommitMetadata(HoodieWriteMetadata> re
@Override
protected void commit(HoodieWriteMetadata> result) {
context.setJobStatus(this.getClass().getSimpleName(), "Commit write status
collect: " + config.getTableName());
-commit(result.getWriteStatuses(), result,
result.getWriteStats().isPresent()
-? result.getWriteStats().get() :
result.getWriteStatuses().map(WriteStatus::getStat).collectAsList());
+if (!config.getBoolean(WRITE_IGNORE_FAILED) &&
result.getWriteStatuses().filter(status -> status.getErrors().size() >
0).count() > 0) {
+ WriteStatus writeStatus = result.getWriteStatuses().filter(status ->
status.getErrors().size() > 0).collectAsList().get(0);
+ HashMap errors = writeStatus.getErrors();
+ // get first one of error info to show error details
+ HoodieKey key = errors.keySet().iterator().next();
Review Comment:
The exception is too simple.
If get the exception, we should provide more logs to find out what happened.
For example, which error happened when writing to which files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1871802867 CI error seems not relete to the changed code，and Azure CI is success @danny0405 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1871281365 ## CI report: * 05bed31829f2362de479344215d29ccca99bd449 UNKNOWN * 814e79d8e5f756f0dac433164a0862a426874ac0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21745) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1871233640 ## CI report: * 05bed31829f2362de479344215d29ccca99bd449 UNKNOWN * f420338d9f04b034a9fac66fbf57e262840557a0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21743) * 814e79d8e5f756f0dac433164a0862a426874ac0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21745) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1871136207 ## CI report: * 05bed31829f2362de479344215d29ccca99bd449 UNKNOWN * f4571180b13ac61c97491a6ec4a005582e5daf58 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21739) * f420338d9f04b034a9fac66fbf57e262840557a0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21743) * 814e79d8e5f756f0dac433164a0862a426874ac0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21745) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1871130456 ## CI report: * 05bed31829f2362de479344215d29ccca99bd449 UNKNOWN * f4571180b13ac61c97491a6ec4a005582e5daf58 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21739) * f420338d9f04b034a9fac66fbf57e262840557a0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21743) * 814e79d8e5f756f0dac433164a0862a426874ac0 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1871093469 ## CI report: * 05bed31829f2362de479344215d29ccca99bd449 UNKNOWN * f4571180b13ac61c97491a6ec4a005582e5daf58 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21739) * f420338d9f04b034a9fac66fbf57e262840557a0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21743) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1871087778 ## CI report: * 05bed31829f2362de479344215d29ccca99bd449 UNKNOWN * f4571180b13ac61c97491a6ec4a005582e5daf58 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21739) * f420338d9f04b034a9fac66fbf57e262840557a0 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1437565057
##
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##
@@ -292,8 +294,12 @@ protected void
setCommitMetadata(HoodieWriteMetadata> re
@Override
protected void commit(HoodieWriteMetadata> result) {
context.setJobStatus(this.getClass().getSimpleName(), "Commit write status
collect: " + config.getTableName());
-commit(result.getWriteStatuses(), result,
result.getWriteStats().isPresent()
-? result.getWriteStats().get() :
result.getWriteStatuses().map(WriteStatus::getStat).collectAsList());
+if (!config.getBoolean(WRITE_IGNORE_FAILED) &&
result.getWriteStatuses().filter(status -> status.getErrors().size() >
0).count() > 0) {
+ throw new HoodieException("Error writing record in the job");
Review Comment:
OK，had add first one of error info to exception details instead of add
logs，because this would error out with details @beyond1920
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
beyond1920 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1437554450
##
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##
@@ -292,8 +294,12 @@ protected void
setCommitMetadata(HoodieWriteMetadata> re
@Override
protected void commit(HoodieWriteMetadata> result) {
context.setJobStatus(this.getClass().getSimpleName(), "Commit write status
collect: " + config.getTableName());
-commit(result.getWriteStatuses(), result,
result.getWriteStats().isPresent()
-? result.getWriteStats().get() :
result.getWriteStatuses().map(WriteStatus::getStat).collectAsList());
+if (!config.getBoolean(WRITE_IGNORE_FAILED) &&
result.getWriteStatuses().filter(status -> status.getErrors().size() >
0).count() > 0) {
+ throw new HoodieException("Error writing record in the job");
Review Comment:
Please add some logs to show error info.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 closed pull request #10297: [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details URL: https://github.com/apache/hudi/pull/10297 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1870909616 ## CI report: * 8e87b6af48f9d7597c6955532d5ee7e61125813b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21510) * 05bed31829f2362de479344215d29ccca99bd449 UNKNOWN * f4571180b13ac61c97491a6ec4a005582e5daf58 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21739) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 closed pull request #10297: [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details URL: https://github.com/apache/hudi/pull/10297 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1870904874 ## CI report: * 8e87b6af48f9d7597c6955532d5ee7e61125813b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21510) * 05bed31829f2362de479344215d29ccca99bd449 UNKNOWN * f4571180b13ac61c97491a6ec4a005582e5daf58 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1437448466
##
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##
@@ -292,6 +294,9 @@ protected void
setCommitMetadata(HoodieWriteMetadata> re
@Override
protected void commit(HoodieWriteMetadata> result) {
context.setJobStatus(this.getClass().getSimpleName(), "Commit write status
collect: " + config.getTableName());
+if (!config.getBoolean(IGNORE_ERROR_WRITE) &&
result.getWriteStatuses().filter(status -> status.getErrors().size() >
0).count() > 0) {
+ throw new HoodieException("Error writing record in the job");
+}
commit(result.getWriteStatuses(), result,
result.getWriteStats().isPresent()
Review Comment:
fix it with: ignore error writestaus by filter them in default，then continue
commit logic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1437446264
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##
@@ -765,6 +765,14 @@ public class HoodieWriteConfig extends HoodieConfig {
.withDocumentation("Avro schema of the partial updates. This is
automatically set by the "
+ "Hudi write client and user is not expected to manually change the
value.");
+ public static final ConfigProperty IGNORE_ERROR_WRITE =
ConfigProperty
+ .key("hoodie.write.ignore.failed")
+ .defaultValue(true)
+ .markAdvanced()
+ .sinceVersion("1.0.0")
+ .withDocumentation("Whether ignore error and not throw exception
when write occur errors."
Review Comment:
done
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##
@@ -765,6 +765,14 @@ public class HoodieWriteConfig extends HoodieConfig {
.withDocumentation("Avro schema of the partial updates. This is
automatically set by the "
+ "Hudi write client and user is not expected to manually change the
value.");
+ public static final ConfigProperty IGNORE_ERROR_WRITE =
ConfigProperty
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1870899907 ## CI report: * 8e87b6af48f9d7597c6955532d5ee7e61125813b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21510) * 05bed31829f2362de479344215d29ccca99bd449 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
danny0405 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1437434113
##
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##
@@ -292,6 +294,9 @@ protected void
setCommitMetadata(HoodieWriteMetadata> re
@Override
protected void commit(HoodieWriteMetadata> result) {
context.setJobStatus(this.getClass().getSimpleName(), "Commit write status
collect: " + config.getTableName());
+if (!config.getBoolean(IGNORE_ERROR_WRITE) &&
result.getWriteStatuses().filter(status -> status.getErrors().size() >
0).count() > 0) {
+ throw new HoodieException("Error writing record in the job");
+}
commit(result.getWriteStatuses(), result,
result.getWriteStats().isPresent()
Review Comment:
should fix this method instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
danny0405 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1437434021
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##
@@ -765,6 +765,14 @@ public class HoodieWriteConfig extends HoodieConfig {
.withDocumentation("Avro schema of the partial updates. This is
automatically set by the "
+ "Hudi write client and user is not expected to manually change the
value.");
+ public static final ConfigProperty IGNORE_ERROR_WRITE =
ConfigProperty
+ .key("hoodie.write.ignore.failed")
+ .defaultValue(true)
+ .markAdvanced()
+ .sinceVersion("1.0.0")
+ .withDocumentation("Whether ignore error and not throw exception
when write occur errors."
Review Comment:
Whether to ignore the errors instead of throwing exception when the commit
metadata encounters errors. if set as false job would error out, default is true
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
danny0405 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1437433114
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##
@@ -765,6 +765,14 @@ public class HoodieWriteConfig extends HoodieConfig {
.withDocumentation("Avro schema of the partial updates. This is
automatically set by the "
+ "Hudi write client and user is not expected to manually change the
value.");
+ public static final ConfigProperty IGNORE_ERROR_WRITE =
ConfigProperty
Review Comment:
`IGNORE_ERROR_WRITE` -> `WRITE_IGNORE_FAILED`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1870867883 > @xuzifu666 Thanks for contribution. This pr add throw exception in each write handlers. Is it possible to check all write statuses for failures during the commit phase? Good advice，I would change it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
beyond1920 commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1870865138 @xuzifu666 Thanks for contribution. Is it possible to check all write statuses for failures during the commit phase? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1437417679
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java:
##
@@ -294,6 +295,9 @@ private Option prepareRecord(HoodieRecord
hoodieRecord) {
hoodieRecord.deflate();
return finalRecordOpt;
} catch (Exception e) {
+ if (!ignoreWriteFailed) {
+throw new HoodieInsertException("Error writing record " +
hoodieRecord, e);
+ }
Review Comment:
> Should we fix it in the handle or when commiting the metadata ?
ok，seem better in commit stage to fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1437324192
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java:
##
@@ -294,6 +295,9 @@ private Option prepareRecord(HoodieRecord
hoodieRecord) {
hoodieRecord.deflate();
return finalRecordOpt;
} catch (Exception e) {
+ if (!ignoreWriteFailed) {
+throw new HoodieInsertException("Error writing record " +
hoodieRecord, e);
+ }
Review Comment:
This may not be fix in handle，because would report error when data write
error such as schema not consistent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1870747269 @bvaradar PTAL，this pr aim to fix when some condition such as write data error，but writing stage not error out，this would confused user such as write data schema is not consistent with table schema in sparksql. so add a config to control whether error out not affect origin condition in default -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1437324192
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java:
##
@@ -294,6 +295,9 @@ private Option prepareRecord(HoodieRecord
hoodieRecord) {
hoodieRecord.deflate();
return finalRecordOpt;
} catch (Exception e) {
+ if (!ignoreWriteFailed) {
+throw new HoodieInsertException("Error writing record " +
hoodieRecord, e);
+ }
Review Comment:
This may not be fix in handle，because would report error when data write
error such as schema not consistent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
danny0405 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1429433856
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java:
##
@@ -294,6 +295,9 @@ private Option prepareRecord(HoodieRecord
hoodieRecord) {
hoodieRecord.deflate();
return finalRecordOpt;
} catch (Exception e) {
+ if (!ignoreWriteFailed) {
+throw new HoodieInsertException("Error writing record " +
hoodieRecord, e);
+ }
Review Comment:
Should we fix it in the handle or when commiting the metadata ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1857132742 java ci error seem not relate to the changed code，and azure is success -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1855693331 ## CI report: * 8e87b6af48f9d7597c6955532d5ee7e61125813b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21510) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1855504705 ## CI report: * 094849fa900d47b238a7ec1430a3b07adaedb0e8 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21507) * 8e87b6af48f9d7597c6955532d5ee7e61125813b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21510) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1855425213 ## CI report: * df87204ec493d6b0a7ca613e2de1f4e4dc52049f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21486) * 094849fa900d47b238a7ec1430a3b07adaedb0e8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21507) * 8e87b6af48f9d7597c6955532d5ee7e61125813b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21510) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1855414123 ## CI report: * df87204ec493d6b0a7ca613e2de1f4e4dc52049f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21486) * 094849fa900d47b238a7ec1430a3b07adaedb0e8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21507) * 8e87b6af48f9d7597c6955532d5ee7e61125813b UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 closed pull request #10297: [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details URL: https://github.com/apache/hudi/pull/10297 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1426251104
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieCreateHandle.java:
##
@@ -165,10 +165,7 @@ protected void doWrite(HoodieRecord record, Schema schema,
TypedProperties props
// record successful.
record.deflate();
} catch (Throwable t) {
- // Not throwing exception from here, since we don't want to fail the
entire job
- // for a single record
- writeStatus.markFailure(record, t, recordMetadata);
Review Comment:
addressed，add all Handles using markFailure and add an internal variable of
HoodieWriteHandle @boneanxs
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieCreateHandle.java:
##
@@ -165,10 +165,7 @@ protected void doWrite(HoodieRecord record, Schema schema,
TypedProperties props
// record successful.
record.deflate();
} catch (Throwable t) {
- // Not throwing exception from here, since we don't want to fail the
entire job
- // for a single record
- writeStatus.markFailure(record, t, recordMetadata);
Review Comment:
addressed，add all Handles using markFailure and add an internal variable of
HoodieWriteHandle @boneanxs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1855249975 ## CI report: * df87204ec493d6b0a7ca613e2de1f4e4dc52049f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21486) * 094849fa900d47b238a7ec1430a3b07adaedb0e8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21507) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1855241772 ## CI report: * df87204ec493d6b0a7ca613e2de1f4e4dc52049f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21486) * 094849fa900d47b238a7ec1430a3b07adaedb0e8 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1426226108
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##
@@ -765,6 +765,14 @@ public class HoodieWriteConfig extends HoodieConfig {
.withDocumentation("Avro schema of the partial updates. This is
automatically set by the "
+ "Hudi write client and user is not expected to manually change the
value.");
+ public static final ConfigProperty IGNORE_ERROR_WRITE =
ConfigProperty
+ .key("hoodie.write.ignore.write.failed")
+ .defaultValue(true)
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
danny0405 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1426220116
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##
@@ -765,6 +765,14 @@ public class HoodieWriteConfig extends HoodieConfig {
.withDocumentation("Avro schema of the partial updates. This is
automatically set by the "
+ "Hudi write client and user is not expected to manually change the
value.");
+ public static final ConfigProperty IGNORE_ERROR_WRITE =
ConfigProperty
+ .key("hoodie.write.ignore.write.failed")
+ .defaultValue(true)
Review Comment:
`hoodie.write.ignore.failed`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1426136858
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieCreateHandle.java:
##
@@ -165,10 +165,7 @@ protected void doWrite(HoodieRecord record, Schema schema,
TypedProperties props
// record successful.
record.deflate();
} catch (Throwable t) {
- // Not throwing exception from here, since we don't want to fail the
entire job
- // for a single record
- writeStatus.markFailure(record, t, recordMetadata);
Review Comment:
ok，thanks for your advices @boneanxs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
boneanxs commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1426123665
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieCreateHandle.java:
##
@@ -165,10 +165,7 @@ protected void doWrite(HoodieRecord record, Schema schema,
TypedProperties props
// record successful.
record.deflate();
} catch (Throwable t) {
- // Not throwing exception from here, since we don't want to fail the
entire job
- // for a single record
- writeStatus.markFailure(record, t, recordMetadata);
Review Comment:
1. May also need to add all other Handles that MergeHandle and check other
usages using `markFailure`
2. Make `config.ignoreErrorWhenWriteFailed()` to be an internal variable of
`HoodieWriteHandle` is more suitable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1853751626 ## CI report: * df87204ec493d6b0a7ca613e2de1f4e4dc52049f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21486) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1424984022
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieCreateHandle.java:
##
@@ -165,10 +165,7 @@ protected void doWrite(HoodieRecord record, Schema schema,
TypedProperties props
// record successful.
record.deflate();
} catch (Throwable t) {
- // Not throwing exception from here, since we don't want to fail the
entire job
- // for a single record
- writeStatus.markFailure(record, t, recordMetadata);
Review Comment:
add config to judge whether error out when write failed for spark scene
@boneanxs @danny0405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1853498003 ## CI report: * 68f3125e06ebb01154d659c2b4452c7ac4c5aa25 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21461) * df87204ec493d6b0a7ca613e2de1f4e4dc52049f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21486) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1853487189 ## CI report: * 68f3125e06ebb01154d659c2b4452c7ac4c5aa25 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21461) * df87204ec493d6b0a7ca613e2de1f4e4dc52049f UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1424984022
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieCreateHandle.java:
##
@@ -165,10 +165,7 @@ protected void doWrite(HoodieRecord record, Schema schema,
TypedProperties props
// record successful.
record.deflate();
} catch (Throwable t) {
- // Not throwing exception from here, since we don't want to fail the
entire job
- // for a single record
- writeStatus.markFailure(record, t, recordMetadata);
Review Comment:
add config to judge whether error out when write failed @boneanxs @danny0405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1424914507
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieCreateHandle.java:
##
@@ -165,10 +165,7 @@ protected void doWrite(HoodieRecord record, Schema schema,
TypedProperties props
// record successful.
record.deflate();
} catch (Throwable t) {
- // Not throwing exception from here, since we don't want to fail the
entire job
- // for a single record
- writeStatus.markFailure(record, t, recordMetadata);
Review Comment:
After the process， job would stop，seem not need the stats any more @boneanxs
can find it from issue: https://issues.apache.org/jira/browse/HUDI-7208
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1424914507
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieCreateHandle.java:
##
@@ -165,10 +165,7 @@ protected void doWrite(HoodieRecord record, Schema schema,
TypedProperties props
// record successful.
record.deflate();
} catch (Throwable t) {
- // Not throwing exception from here, since we don't want to fail the
entire job
- // for a single record
- writeStatus.markFailure(record, t, recordMetadata);
Review Comment:
after the process， job would stop，seem not need the stats any more @boneanxs
can find it from issue: https://issues.apache.org/jira/browse/HUDI-7208
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1424914507
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieCreateHandle.java:
##
@@ -165,10 +165,7 @@ protected void doWrite(HoodieRecord record, Schema schema,
TypedProperties props
// record successful.
record.deflate();
} catch (Throwable t) {
- // Not throwing exception from here, since we don't want to fail the
entire job
- // for a single record
- writeStatus.markFailure(record, t, recordMetadata);
Review Comment:
after the process， job would stop，seem not need the stats any more @boneanxs
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieCreateHandle.java:
##
@@ -165,10 +165,7 @@ protected void doWrite(HoodieRecord record, Schema schema,
TypedProperties props
// record successful.
record.deflate();
} catch (Throwable t) {
- // Not throwing exception from here, since we don't want to fail the
entire job
- // for a single record
- writeStatus.markFailure(record, t, recordMetadata);
Review Comment:
after the process， job would stop，seem not need the stats any more @boneanxs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
boneanxs commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1424913288
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieCreateHandle.java:
##
@@ -165,10 +165,7 @@ protected void doWrite(HoodieRecord record, Schema schema,
TypedProperties props
// record successful.
record.deflate();
} catch (Throwable t) {
- // Not throwing exception from here, since we don't want to fail the
entire job
- // for a single record
- writeStatus.markFailure(record, t, recordMetadata);
Review Comment:
Does any other code pieces requires this stats?(`markFailure`) I'm also
confused why we only need to do a mark here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1423553849
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java:
##
@@ -508,10 +509,7 @@ protected void doWrite(HoodieRecord record, Schema schema,
TypedProperties props
flushToDiskIfRequired(record, false);
writeToBuffer(record);
} catch (Throwable t) {
- // Not throwing exception from here, since we don't want to fail the
entire job
- // for a single record
- writeStatus.markFailure(record, t, recordMetadata);
- LOG.error("Error writing record " + record, t);
+ throw new HoodieInsertException("Error writing record " + record, t);
Review Comment:
spark seem not have the config @danny0405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1423553849
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java:
##
@@ -508,10 +509,7 @@ protected void doWrite(HoodieRecord record, Schema schema,
TypedProperties props
flushToDiskIfRequired(record, false);
writeToBuffer(record);
} catch (Throwable t) {
- // Not throwing exception from here, since we don't want to fail the
entire job
- // for a single record
- writeStatus.markFailure(record, t, recordMetadata);
- LOG.error("Error writing record " + record, t);
+ throw new HoodieInsertException("Error writing record " + record, t);
Review Comment:
spark seem not the config
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
danny0405 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1423550203
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java:
##
@@ -508,10 +509,7 @@ protected void doWrite(HoodieRecord record, Schema schema,
TypedProperties props
flushToDiskIfRequired(record, false);
writeToBuffer(record);
} catch (Throwable t) {
- // Not throwing exception from here, since we don't want to fail the
entire job
- // for a single record
- writeStatus.markFailure(record, t, recordMetadata);
- LOG.error("Error writing record " + record, t);
+ throw new HoodieInsertException("Error writing record " + record, t);
Review Comment:
Does Spark have similiar option?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1851381361 ## CI report: * 68f3125e06ebb01154d659c2b4452c7ac4c5aa25 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21461) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1851216322 ## CI report: * fe0a9fb6f96859b5d2bc2254899f2f5c8624d841 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21438) * 68f3125e06ebb01154d659c2b4452c7ac4c5aa25 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21461) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1851211389 ## CI report: * fe0a9fb6f96859b5d2bc2254899f2f5c8624d841 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21438) * 68f3125e06ebb01154d659c2b4452c7ac4c5aa25 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1423332239
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java:
##
@@ -508,10 +509,7 @@ protected void doWrite(HoodieRecord record, Schema schema,
TypedProperties props
flushToDiskIfRequired(record, false);
writeToBuffer(record);
} catch (Throwable t) {
- // Not throwing exception from here, since we don't want to fail the
entire job
- // for a single record
- writeStatus.markFailure(record, t, recordMetadata);
- LOG.error("Error writing record " + record, t);
+ throw new HoodieInsertException("Error writing record " + record, t);
Review Comment:
revert changes in HoodieAppendHandle @danny0405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
xuzifu666 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1423332239
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java:
##
@@ -508,10 +509,7 @@ protected void doWrite(HoodieRecord record, Schema schema,
TypedProperties props
flushToDiskIfRequired(record, false);
writeToBuffer(record);
} catch (Throwable t) {
- // Not throwing exception from here, since we don't want to fail the
entire job
- // for a single record
- writeStatus.markFailure(record, t, recordMetadata);
- LOG.error("Error writing record " + record, t);
+ throw new HoodieInsertException("Error writing record " + record, t);
Review Comment:
revert changed in HoodieAppendHandle @danny0405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
danny0405 commented on code in PR #10297:
URL: https://github.com/apache/hudi/pull/10297#discussion_r1423319781
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java:
##
@@ -508,10 +509,7 @@ protected void doWrite(HoodieRecord record, Schema schema,
TypedProperties props
flushToDiskIfRequired(record, false);
writeToBuffer(record);
} catch (Throwable t) {
- // Not throwing exception from here, since we don't want to fail the
entire job
- // for a single record
- writeStatus.markFailure(record, t, recordMetadata);
- LOG.error("Error writing record " + record, t);
+ throw new HoodieInsertException("Error writing record " + record, t);
Review Comment:
In Flink SQL, option `write.ignore.failed` controls the throwing behavior of
errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1849777182 ## CI report: * fe0a9fb6f96859b5d2bc2254899f2f5c8624d841 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21438) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1849483572 ## CI report: * fe0a9fb6f96859b5d2bc2254899f2f5c8624d841 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21438) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]
hudi-bot commented on PR #10297: URL: https://github.com/apache/hudi/pull/10297#issuecomment-1849474557 ## CI report: * fe0a9fb6f96859b5d2bc2254899f2f5c8624d841 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
